Rclone generate outgoing traffic in remote destination

Help and Support
1

What is the problem you are having with rclone?

Rclone generates outgoing traffic (600 GB) to a remote destination when I use the copy or sync command. I'm not sure which one caused it, because I ran both commands before I checked the traffic on AWS.
I believe that in the checking stage, rclone not just compare md5 hash (with ETag), but download the file from destination.
I know that the version I am using is old (from 2022), but I just want to be sure: is this expected behavior or a bug in that version?

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.0

  • os/version: debian 10.13 (64 bit)
  • os/kernel: 4.19.0-26-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Digital Ocean to AWS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy spaces:bucket/ aws:bucket/ --transfers=38 --checkers=38 --drive-chunk-size=128M --bwlimit=0 --fast-list --buffer-size=64M --tpslimit=30 --progress --s3-upload-concurrency=12

or

rclone sync spaces:bucket/ aws:bucket/ --transfers=25 --checkers=25 --drive-chunk-size=256M --bwlimit=0 --fast-list --buffer-size=300M --tpslimit=30 --progress --s3-upload-concurrency=14

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

Command not found
2

welcome to the forum,

correct, rclone has to download the file from the source and then upload to the test.

best to rclone selfupdate

that is for gdrive, not s3.

3

It should only download if the source and remote not within same region in same provider, right?
So for example if source and dest is a bucket within the same region, then it can use the native copy, right?

4

in that case, should be a server-side copy.
and you can easily test that for yourself, just copy a small text file and look that the rclone log.

5

Another question:
If I understand correctly, by default, rclone sends a HEAD request to determine if the file should be synchronized. I noticed that there is a --download flag, which downloads the file, hashes it locally, and then compares it with the files from the source. So, the default rclone configuration does not generate egress in the destination bucket, right?

6

your example lack specfic details, hard to provide an answer.

as i mentioned, just copy a small text file, look at the rclone log, then you will know.
in the log, would see file.ext: Copied (server-side copy)

7

Assume there are two providers that support ETag. In that case, it's not necessary to download the file and hash it locally. rclone can use the ETag to compare and determine whether it should be copied or not. Right?

8

by default, rclone compares files by modtime and size.
if you want rclone to use hash or etag, then can do that.

in your case, the source and dest are both s3, so using checksums is a good idea.
with rclone, there are two types of checksums,

  1. rclone saves the md5 hash as metadata
  2. rclone can use the etag
9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.