Rclone copy reuploaded

Help and Support
#1

What is the problem you are having with rclone?

Hello, rclone copy reuploaded files into Backblaze as log mentioned that modification times differ by -140.177125ms.
The strange issue is that 3 files had modification times differ but for the two files just mentioned that

Updated modification time in destination
Unchanged skipping
Where the one file which is the largest (4GB) mentioned:
Modification times differ by -134.151245ms
Starting upload of large file in 442 chunks

Can you please assist why the largest file was reuploaded where the other two files just updated the modification time and not reuploaded, thank you!

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Centos 7

Which cloud storage system are you using? (eg Google Drive)

BackBlaze

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy source  destination --b2-upload-cutoff=1000M --transfers 16 --b2-chunk-size=400M   --log-file /home/log/rclone.log --b2-disable-checksum  -vv  --progress --fast-list
#2

What is the source filing system that you are copying from? Are its timestamps only accurate to 1s maybe? If so use --modify-window 1s

1 Like
#3

Hi, thank uoi once again for your reply, it's ext4. The only difference is that the files where uploaded from one server and the other files from another, but same files existed within Backblaze. Strange is that Rclone re-uploaded only the large file.

#4

It will be something to do with time difference skew I think,

You don't say your rclone version, but I suspect it is an old version as newer ones would have just corrected the timestamp.

#5

one server has
rclone --version
rclone v1.53.3

  • os/arch: linux/amd64
  • go version: go1.15.5

another:
rclone --version
rclone v1.53.2

  • os/arch: linux/amd64
  • go version: go1.15.3
#6

Ah, I know why... I suspect your large file didn't have an sha1sum for some reason (maybe it wasn't uploaded with rclone), so rclone couldn't be sure whether it was identical or not.

#7

Hi as it was uploaded with Rclone, actually all uploads are handled by rclone, is the --b2-disable-checksum flag the cause of our issue?

#8

Yes it is. If you do that, then large files won't have SHA1 checksums, so rclone can't prove to itself that two files with different timestamps are the same, so it re-uploads it.

#9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.