Networking: --dry-run flag and how bytes are recorded in Rclone

Help and Support
1

Hello, I am looking to get feedback on how network connections are established and expected behavior when executing a dry-run for Rclone.

What is the problem you are having with rclone?

Previously we were using the --dry-run flag to determine approximate sizes of files that would be moved from Dropbox and AWS S3 and validate information. We executed such for a large directory in a GCP VM (approximately 380TB) and although --dry-run was active, it seems the GCP VM recorded egress data. I have already contacted GCP support on addressing such but wanted to see what ports and protocols are established in RClone.

TL;dr - Rclone --dry-run egress data appeared to be the same value as the actual dropbox directory, but subsequent testing after restarting the VM showed significantly less data. Trying to understand how networking works in a dry-run flag Rclone run. Documentation appreciated!

Are you on the latest version of rclone?

rclone v1.69.1
-os/version: ubuntu 20.04 (64 bit)
-os/type: linux
-os/arch: arm64

Which cloud storage system are you using? (eg Google Drive)
Dropbox (source) and AWS S3 (target)

The command you were trying to run (eg rclone copy /tmp remote:tmp) :

rclone sync --dry-run --disable-http2 --checkers=12 --tpslimit=12 --buffer-size=1G --s3-chunk-size=512M dropbox-remote:$filepath s3-remote:$filepath2

Edit: One more note: our original dry run was on Rclone version 1.53.1 which was the original curled version installed on the ubuntu instance for GCP but we mistakely did not verify it was updated first. I am not sure if --dry-run was any different at that time.

2

welcome to the forum,

well, rclone sync has to compare the source and dest using api calls.
can see that for yourself using --dump=headers

https, port 443

1 Like
3

Understood, so standard 80/443 ports and data used through dry-runs would only be comparing if the file existed already or not in the target (S3) destination and an outbound call to the source (dropbox) destination to query what files exist, their size and file location - is that a fair statement?

4

yes, that is a fair statement.

5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.