Rclone copy to Proton Drive recreates existing directories

derekmahar · April 24, 2024, 11:16pm

What is the problem you are having with rclone?

Command rclone copy to Proton Drive recreates directories in Proton Drive that already exist.

Run the command 'rclone version' and share the full output of the command.

$ rclone version
rclone v1.66.0
- os/version: freebsd 13.1-release-p9 (64 bit)
- os/kernel: 13.1-release-p9 (amd64)
- os/type: freebsd
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Proton Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

$HOME/bin/rclone copy \
  --checkers=12 \
  --checksum \
  --copy-links \
  --metadata \
  --no-traverse \
  --refresh-times \
  --transfers=4 \
  --verbose=3 \
  /mnt/zdd/share/Archive/Documents \
  proton:Archive/Documents

The rclone config contents with secrets removed.

$ rclone config redacted proton
[proton]
type = protondrive
username = <My Proton Mail email address>
password = XXX
client_uid = XXX
client_access_token = XXX
client_refresh_token = XXX
client_salted_key_pass = XXX

Double check the config for sensitive info before posting publicly```

A log from the command with the `-vv` flag

Here is an excerpt of the relevant log file entries:

2024/04/24 19:16:12 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/mnt/zdd/home/derek/bin/rclone" "copy" "--checkers=12" "--checksum" "--copy-links" "--metadata" "--no-traverse" "--refresh-times" "--transfers=4" "--verbose=3" "/mnt/zdd/share/Archive/Documents" "proton:Archive/Documents"]
2024/04/24 19:16:12 DEBUG : Creating backend with remote "/mnt/zdd/share/Archive/Documents"
2024/04/24 19:16:12 DEBUG : Using config file from "/mnt/zdd/home/derek/.config/rclone/rclone.conf"
2024/04/24 19:16:12 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2024/04/24 19:16:12 DEBUG : fs cache: renaming cache item "/mnt/zdd/share/Archive/Documents" to be canonical "local{12rtk}:/mnt/zdd/share/Archive/Documents"
2024/04/24 19:16:12 DEBUG : Creating backend with remote "proton:Archive/Documents"
2024/04/24 19:16:12 DEBUG : proton drive root link ID 'Archive/Documents': Has cached credentials
2024/04/24 19:16:14 DEBUG : proton drive root link ID 'Archive/Documents': Used cached credential to initialize the ProtonDrive API
2024/04/24 19:16:16 DEBUG : Business: Making directory
2024/04/24 19:16:17 DEBUG : Communication: Making directory
2024/04/24 19:16:18 DEBUG : Education: Making directory
2024/04/24 19:16:19 DEBUG : Family: Making directory
2024/04/24 19:16:19 DEBUG : Government: Making directory
2024/04/24 19:16:20 DEBUG : Identification: Making directory
2024/04/24 19:16:21 DEBUG : Immigration: Making directory
2024/04/24 19:16:22 DEBUG : Legal: Making directory
2024/04/24 19:16:22 DEBUG : Notes: Making directory
2024/04/24 19:16:23 DEBUG : Playlists: Making directory
2024/04/24 19:16:24 DEBUG : Technology: Making directory
2024/04/24 19:16:25 DEBUG : Travel: Making directory
2024/04/24 19:16:26 DEBUG : Education/Institutions: Making directory
2024/04/24 19:16:26 DEBUG : Business/Accounting: Making directory
2024/04/24 19:16:26 DEBUG : Communication/Contacts: Making directory
2024/04/24 19:16:26 DEBUG : Notes/Evernote: Making directory
2024/04/24 19:16:26 DEBUG : Playlists/Software Architecture: Making directory
2024/04/24 19:16:26 DEBUG : Technology/Computer: Making directory
2024/04/24 19:16:28 DEBUG : Education/Training: Making directory

Note the repeated "Making directory" log messages. The directories in Proton Drive to which these messages refer already exist. rclone copy attempts to recreate them every time, or at least reports that it attempts to recreate them.

asdffdsa · April 24, 2024, 11:39pm

welcome to the forum,
i think i can explain what is going on.

can you confirm by trying this flag.
https://rclone.org/docs/#no-update-dir-modtime

and in this case, for testing, can use --dry-run

derekmahar · April 25, 2024, 12:14am

Unfortunately, after adding this option, rclone still attempts to recreate the directories in Proton Drive.

asdffdsa · April 25, 2024, 12:34am

ok, so modtime of the directory is not the issue, maybe metadata

might want to test the simplest command possible; without these flags.

  --checkers=12 \
  --checksum \
  --copy-links \
  --metadata \
  --no-traverse \
  --refresh-times \
  --transfers=4 \

derekmahar · April 25, 2024, 1:23am

The culprit was --no-traverse, but this option is supposed shorten the time that a copy takes. In this case, though, it increases the completion time. I still think this is a bug.

asdffdsa · April 25, 2024, 12:35pm

there are many bugs/issues with proton remote, that are not getting fixed.

derekmahar · April 25, 2024, 8:35pm

Yes, but at least this one is relatively minor and we managed to find a workaround.

asdffdsa · April 25, 2024, 8:40pm

yes, we did. good for that.

not understanding how that is the culprit or if there is a rclone bug.

to confirm this is a bug, would be helpful so it can be fixed.
can you post two debug logs, with identical commands except for --no-traverse ?

derekmahar · April 25, 2024, 8:50pm

I think it's a bug in rclone copy --no-traverse. In the earlier version of rclone that I was using (I don't recall which one, but not too old), rclone copy --no-traverse did not try to recreate directories.

Furthermore, the description of --no-traverse does not suggest that it should cause rclone copy to recreate directories:

      --no-traverse                                 Don't traverse destination file system on copy

The only difference is that without --no-traverse, rclone copy doesn't report log messages similar to

2024/04/24 19:16:16 DEBUG : Business: Making directory

asdffdsa · April 25, 2024, 8:58pm

is the directory empty?

the lastest version of rclone does a have bug, related to directories.
https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057
that is why i first suggested to try --no-update-dir-modtime

tho, not still not understanding what --no-traverse has to do with that?

asdffdsa · April 25, 2024, 9:46pm

are you sure that rclone is actually re-creating the dir?
perhaps rclone is simply printing incorrect debug output.

derekmahar · April 25, 2024, 10:12pm

No, it contains other directories.

I don't know if this issue is related to my issue. That one refers to Google Drive, not Proton Drive, and refers to empty directories, but the directories in my case are not empty.

I don't know, but I think it shouldn't try to recreate existing directories.

derekmahar · April 25, 2024, 10:31pm

No, I'm not sure. It is possible that rclone is simply printing incorrect debug log messages. However, I also noticed that rclone copy --no-traverse is much slower when it displays those log messages. Without --no-traverse, rclone takes about the same time to copy all of the files, including skipping files and directories which already exist on the remote. I think --no-traverse is not behaving the same ways as it did in the earlier version of rclone that I was using.

asdffdsa · April 25, 2024, 10:43pm

due to issues like yours, other forum posts and that other bug report about directories, i have yet to upgrade from v1.65.2 to v1.66.0
i try to stay one version behind.

derekmahar · April 25, 2024, 10:55pm

That's probably a good practise assuming that the version of the program is working correctly. I found out that the earlier version that I was using was v1.64.2-DEV. Using that version, rclone copy --no-traverse --max-age=1d completes in 5.4 s instead of 38 minutes. I think the Proton Drive driver in rclone 1.66.0 introduced a few regressions. I will try 1.65.2 and see if the behaviour is the same as 1.64.2-DEV.

asdffdsa · April 25, 2024, 10:57pm

yeah, proton is a problem on to itself.
i am a paid subsriber and try my best, could not get rclone to work with proton.
for now, i gave up trying.

derekmahar · April 25, 2024, 11:05pm

I can use rclone to copy and sync to Proton Drive. Using rclone 1.64.2 and 1.65.2, rclone copy --no-traverse --max-age=1d takes seconds rather than minutes to complete. I run rclone sync once a week and so far haven't encountered any issues. I will also continue to use 1.65.2 until the next release after 1.66.0.

asdffdsa · April 25, 2024, 11:14pm

well, sure, that greatly affects the result. and none of your output from up above, used that flag?

derekmahar · April 26, 2024, 1:58am

Yes, that's correct. I had removed --max-age=1d because the directory recreation that --no-traverse triggered in 1.66.0 made --max-age ineffective. Fortunately, both --no-traverse and --max-age=1d work correctly in rclone 1.65.2.

system · May 26, 2024, 1:58am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.