Rclone copy to Proton Drive recreates existing directories

Suspected Bug
1

What is the problem you are having with rclone?

Command rclone copy to Proton Drive recreates directories in Proton Drive that already exist.

Run the command 'rclone version' and share the full output of the command.

$ rclone version
rclone v1.66.0
- os/version: freebsd 13.1-release-p9 (64 bit)
- os/kernel: 13.1-release-p9 (amd64)
- os/type: freebsd
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Proton Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$HOME/bin/rclone copy \
  --checkers=12 \
  --checksum \
  --copy-links \
  --metadata \
  --no-traverse \
  --refresh-times \
  --transfers=4 \
  --verbose=3 \
  /mnt/zdd/share/Archive/Documents \
  proton:Archive/Documents

The rclone config contents with secrets removed.

$ rclone config redacted proton
[proton]
type = protondrive
username = <My Proton Mail email address>
password = XXX
client_uid = XXX
client_access_token = XXX
client_refresh_token = XXX
client_salted_key_pass = XXX

Double check the config for sensitive info before posting publicly```

A log from the command with the -vv flag

Here is an excerpt of the relevant log file entries:

2024/04/24 19:16:12 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/mnt/zdd/home/derek/bin/rclone" "copy" "--checkers=12" "--checksum" "--copy-links" "--metadata" "--no-traverse" "--refresh-times" "--transfers=4" "--verbose=3" "/mnt/zdd/share/Archive/Documents" "proton:Archive/Documents"]
2024/04/24 19:16:12 DEBUG : Creating backend with remote "/mnt/zdd/share/Archive/Documents"
2024/04/24 19:16:12 DEBUG : Using config file from "/mnt/zdd/home/derek/.config/rclone/rclone.conf"
2024/04/24 19:16:12 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2024/04/24 19:16:12 DEBUG : fs cache: renaming cache item "/mnt/zdd/share/Archive/Documents" to be canonical "local{12rtk}:/mnt/zdd/share/Archive/Documents"
2024/04/24 19:16:12 DEBUG : Creating backend with remote "proton:Archive/Documents"
2024/04/24 19:16:12 DEBUG : proton drive root link ID 'Archive/Documents': Has cached credentials
2024/04/24 19:16:14 DEBUG : proton drive root link ID 'Archive/Documents': Used cached credential to initialize the ProtonDrive API
2024/04/24 19:16:16 DEBUG : Business: Making directory
2024/04/24 19:16:17 DEBUG : Communication: Making directory
2024/04/24 19:16:18 DEBUG : Education: Making directory
2024/04/24 19:16:19 DEBUG : Family: Making directory
2024/04/24 19:16:19 DEBUG : Government: Making directory
2024/04/24 19:16:20 DEBUG : Identification: Making directory
2024/04/24 19:16:21 DEBUG : Immigration: Making directory
2024/04/24 19:16:22 DEBUG : Legal: Making directory
2024/04/24 19:16:22 DEBUG : Notes: Making directory
2024/04/24 19:16:23 DEBUG : Playlists: Making directory
2024/04/24 19:16:24 DEBUG : Technology: Making directory
2024/04/24 19:16:25 DEBUG : Travel: Making directory
2024/04/24 19:16:26 DEBUG : Education/Institutions: Making directory
2024/04/24 19:16:26 DEBUG : Business/Accounting: Making directory
2024/04/24 19:16:26 DEBUG : Communication/Contacts: Making directory
2024/04/24 19:16:26 DEBUG : Notes/Evernote: Making directory
2024/04/24 19:16:26 DEBUG : Playlists/Software Architecture: Making directory
2024/04/24 19:16:26 DEBUG : Technology/Computer: Making directory
2024/04/24 19:16:28 DEBUG : Education/Training: Making directory

Note the repeated "Making directory" log messages. The directories in Proton Drive to which these messages refer already exist. rclone copy attempts to recreate them every time, or at least reports that it attempts to recreate them.

2

welcome to the forum,
i think i can explain what is going on.

can you confirm by trying this flag.
https://rclone.org/docs/#no-update-dir-modtime

and in this case, for testing, can use --dry-run

3

Unfortunately, after adding this option, rclone still attempts to recreate the directories in Proton Drive.

4

ok, so modtime of the directory is not the issue, maybe metadata

might want to test the simplest command possible; without these flags.

  --checkers=12 \
  --checksum \
  --copy-links \
  --metadata \
  --no-traverse \
  --refresh-times \
  --transfers=4 \
5

The culprit was --no-traverse, but this option is supposed shorten the time that a copy takes. In this case, though, it increases the completion time. I still think this is a bug.

6

there are many bugs/issues with proton remote, that are not getting fixed.

7

Yes, but at least this one is relatively minor and we managed to find a workaround.

8

yes, we did. good for that.

not understanding how that is the culprit or if there is a rclone bug.

to confirm this is a bug, would be helpful so it can be fixed.
can you post two debug logs, with identical commands except for --no-traverse ?

9

I think it's a bug in rclone copy --no-traverse. In the earlier version of rclone that I was using (I don't recall which one, but not too old), rclone copy --no-traverse did not try to recreate directories.

Furthermore, the description of --no-traverse does not suggest that it should cause rclone copy to recreate directories:

      --no-traverse                                 Don't traverse destination file system on copy

The only difference is that without --no-traverse, rclone copy doesn't report log messages similar to

2024/04/24 19:16:16 DEBUG : Business: Making directory
10

is the directory empty?

the lastest version of rclone does a have bug, related to directories.
https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057
that is why i first suggested to try --no-update-dir-modtime

tho, not still not understanding what --no-traverse has to do with that?

11

are you sure that rclone is actually re-creating the dir?
perhaps rclone is simply printing incorrect debug output.

12

No, it contains other directories.

I don't know if this issue is related to my issue. That one refers to Google Drive, not Proton Drive, and refers to empty directories, but the directories in my case are not empty.

I don't know, but I think it shouldn't try to recreate existing directories.

13

No, I'm not sure. It is possible that rclone is simply printing incorrect debug log messages. However, I also noticed that rclone copy --no-traverse is much slower when it displays those log messages. Without --no-traverse, rclone takes about the same time to copy all of the files, including skipping files and directories which already exist on the remote. I think --no-traverse is not behaving the same ways as it did in the earlier version of rclone that I was using.

14

due to issues like yours, other forum posts and that other bug report about directories, i have yet to upgrade from v1.65.2 to v1.66.0
i try to stay one version behind.

15

That's probably a good practise assuming that the version of the program is working correctly. I found out that the earlier version that I was using was v1.64.2-DEV. Using that version, rclone copy --no-traverse --max-age=1d completes in 5.4 s instead of 38 minutes. I think the Proton Drive driver in rclone 1.66.0 introduced a few regressions. I will try 1.65.2 and see if the behaviour is the same as 1.64.2-DEV.

16

yeah, proton is a problem on to itself.
i am a paid subsriber and try my best, could not get rclone to work with proton.
for now, i gave up trying.

17

I can use rclone to copy and sync to Proton Drive. Using rclone 1.64.2 and 1.65.2, rclone copy --no-traverse --max-age=1d takes seconds rather than minutes to complete. I run rclone sync once a week and so far haven't encountered any issues. I will also continue to use 1.65.2 until the next release after 1.66.0.

18

well, sure, that greatly affects the result. and none of your output from up above, used that flag?

19

Yes, that's correct. I had removed --max-age=1d because the directory recreation that --no-traverse triggered in 1.66.0 made --max-age ineffective. Fortunately, both --no-traverse and --max-age=1d work correctly in rclone 1.65.2.