Rclone not copying just changed files to pcloud unless using slow connection

Help and Support
1

What is the problem you are having with rclone?

sync to pcloud via a pcloud.fs mount with encryption copies all files, not just changed files, but is fast while sync to pcloud via webdav copies just changed files but is very slow.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-139-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

I am not on the latest version, I am running the version that was installed via the Ubuntu 20.04 repos.

Which cloud storage system are you using? (eg Google Drive)

pcloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The following only copies changed files but is slow.

rclone sync /root/testing pcloud-webdav-crypt:/test -vv

The rclone config contents with secrets removed.

[pcloud-webdav]
type = pcloud
token = xxxxxxx

[pcloud-webdav-crypt]
type = crypt
remote = pcloud-webdav:/backups
filename_encryption = obfuscate
password = xxxx
password2 = xxxx

A log from the command with the -vv flag

2023/03/02 16:03:56 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "sync" "/root/testing" "pcloud-webdav-crypt:/test" "-vv"]
2023/03/02 16:03:56 DEBUG : Creating backend with remote "/root/testing"
2023/03/02 16:03:56 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/03/02 16:03:56 DEBUG : Creating backend with remote "pcloud-webdav-crypt:/test"
2023/03/02 16:03:56 DEBUG : Creating backend with remote "pcloud-webdav:/backups/192.yjxy"
2023/03/02 16:03:56 DEBUG : fs cache: renaming cache item "pcloud-webdav:/backups/192.yjxy" to be canonical "pcloud-webdav:backups/192.yjxy"
2023/03/02 16:03:56 DEBUG : fs cache: switching user supplied name "pcloud-webdav:/backups/192.yjxy" for canonical name "pcloud-webdav:backups/192.yjxy"
2023/03/02 16:03:56 DEBUG : Encrypted drive 'pcloud-webdav-crypt:/test': Waiting for checks to finish
2023/03/02 16:03:56 DEBUG : hosts.allow: Size and modification time the same (differ by -195.833215ms, within tolerance 1s)
2023/03/02 16:03:56 DEBUG : hosts.allow: Unchanged skipping
2023/03/02 16:03:56 DEBUG : hosts: Size and modification time the same (differ by -195.833215ms, within tolerance 1s)
2023/03/02 16:03:56 DEBUG : hosts: Unchanged skipping
2023/03/02 16:03:56 DEBUG : hosts.deny: Size and modification time the same (differ by -195.833215ms, within tolerance 1s)
2023/03/02 16:03:56 DEBUG : hosts.deny: Unchanged skipping
2023/03/02 16:03:56 DEBUG : Encrypted drive 'pcloud-webdav-crypt:/test': Waiting for transfers to finish
2023/03/02 16:03:56 DEBUG : Waiting for deletions to finish
2023/03/02 16:03:56 INFO  : There was nothing to transfer
2023/03/02 16:03:56 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 3 / 3, 100%
Elapsed time:         0.5s

The following always copies all files but is fast:

rclone sync /root/testing pcloud:/test -vv

Config:

[pcloud]
type = crypt
remote = /srv/pcloud/backups
filename_encryption = obfuscate
password = xxxx
password2 = xxxx

Root cronjob to mount the pcloud drive:

@reboot /usr/local/bin/pcloudcc -u user@email.com -m /srv/pcloud -d

2023/03/02 16:05:11 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "sync" "/root/testing" "pcloud:/test" "-vv"]
2023/03/02 16:05:11 DEBUG : Creating backend with remote "/root/testing"
2023/03/02 16:05:11 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/03/02 16:05:11 DEBUG : Creating backend with remote "pcloud:/test"
2023/03/02 16:05:11 DEBUG : Creating backend with remote "/srv/pcloud/backups/192.yjxy"
2023/03/02 16:05:11 DEBUG : hosts: Modification times differ by -195.833215ms: 2023-03-02 14:16:58.195833215 -0500 EST, 2023-03-02 14:16:58 -0500 EST
2023/03/02 16:05:11 DEBUG : hosts.allow: Modification times differ by -195.833215ms: 2023-03-02 14:16:58.195833215 -0500 EST, 2023-03-02 14:16:58 -0500 EST
2023/03/02 16:05:11 DEBUG : hosts.deny: Modification times differ by -195.833215ms: 2023-03-02 14:16:58.195833215 -0500 EST, 2023-03-02 14:16:58 -0500 EST
2023/03/02 16:05:11 DEBUG : Encrypted drive 'pcloud:/test': Waiting for checks to finish
2023/03/02 16:05:11 DEBUG : Encrypted drive 'pcloud:/test': Waiting for transfers to finish
2023/03/02 16:05:11 DEBUG : preAllocate: got error on fallocate, trying combination 1/2: operation not supported
2023/03/02 16:05:11 DEBUG : preAllocate: got error on fallocate, trying combination 2/2: operation not supported
2023/03/02 16:05:11 DEBUG : hosts.deny: md5 = e1e2df0f0ce8be3ce33d9c4ccf35b3b4 OK
2023/03/02 16:05:11 INFO  : hosts.deny: Copied (replaced existing)
2023/03/02 16:05:11 DEBUG : hosts.allow: md5 = 4b497b8f06d829091d3a47a78afc1da7 OK
2023/03/02 16:05:11 INFO  : hosts.allow: Copied (replaced existing)
2023/03/02 16:05:11 DEBUG : hosts: md5 = aec22076a5215493d67c1b2da54d441c OK
2023/03/02 16:05:11 INFO  : hosts: Copied (replaced existing)
2023/03/02 16:05:11 DEBUG : Waiting for deletions to finish
2023/03/02 16:05:11 INFO  : 
Transferred:   	    1.451 KiB / 1.451 KiB, 100%, 0 B/s, ETA -
Checks:                 3 / 3, 100%
Transferred:            3 / 3, 100%
Elapsed time:         0.0s

So, both syncs are using the same unencrypted source.
They are both using the same encrypted destination, but with a different connection mechanism.
What is going on here?

I would like to use the mounted drive method, to take advantage of the speed bump, with the ability to only transfer changed files.

Can someone explain what is going on here and if there is a way to achieve my goal?

Thank you for taking the time to read this!!!

Chris

2

hello and welcome to the forum,

this is about the destination, not the source.

i think the issue is about https://rclone.org/docs/#modify-window-time

let's take the file hosts.deny

in your first example, the dest remote is pcloud, which has a modtime window of 1 second.
in this case, the time difference -195.833215ms is less than 1 second, so rclone does not copy the file.
and this can be seen from within tolerance 1s

in your second example, the dest remote is local, which has a modtime window of 1 nanosecond.
in this case, the time difference is -195.833215ms is greater than 1 nanosecond, so rclone does copy the file.

latest stable rclone is v1.61.1
https://rclone.org/downloads/#script-download-and-install

add --local-no-preallocate

3

Thank you! I would never have figured that out. That fixed my test case. I will check how the full backup works tonight.

4

Full backup worked as expected with "--modify-window=1s"

5

ok, good to know, and you are welcome

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.