What is recommended when copying large volumes and sized files?

Howto Guides
1

Being relatively new to rclone, is the command below sufficient in achieving the following (with or without encryption)?

  • Resuming transfers if interrupted
  • Delta transfers
  • Displaying itemized changes
  • Preserving metadata such as permissions, ownership
  • Parallelization

rclone -v --transfers=10 --stats --progress copy /source/ dropbox:/destination

Alternatively, what is the equivalent rclone command to a similar rsync command?

rysnc -avhlAXEWP --itemize-changes --stats --info=progress2 /source/ /destination

The primary remote will be DropBox. The totality of the size of the files is approximately 4TB+ and there are in excess of 100,000 files.

Additionally, will the command below support verification of corruption and a successful write after the initial upload?

rclone -vc --transfers=10 --stats --progress copy /source/ dropbox:/destination

What is the recommendation to limit the rate of transfer so that bandwidth utilization isn't saturated? The current upload limit is approximately 40Mbps.

2

You command will do pretty well at that, though you don't want --stats and --progress (and --stats takes a time unit, eg --stats 10s.

Rclone doesn't do delta transfers like rsync (cloud storage doesn't support it in general) nor preserve permissions (same reason). You can quit and restart rclone and it will pick up where it left off. Rclone is very good at parallel transfers.

Rclone stores files 1:1 your disk -> cloud storage.

If you want a backup program which does do delta transfers and preserve permissions then you could use restic which can use rclone as a backend to store the actual objects.

Rclone checks checksums after each upload. You can also use rclone check to do a complete check of all the checksums.

Use --bwlimit. The parameter it takes is in bytes/s so 40Mbit/s = 8 MByte/s. Knock that down a bit to avoid saturation, say --bwlimit 7M (you'll probably need to experiment).

3

Is there a particular reason that they shouldn't be combined? Is there a default for --stats or a recommendation?

4

I thought that rclone only transfers changes. Can rclone only be resumed in the same session or can it be reinstated in a separate session? For example, if I stop rclone, shutdown the device and restart it?

How does parallelization affect --bwlimit? For example, if --transfers=10 and --bwlimit=7M, will it attempt to sequentially transmit these within the limits of 7M as a total?

What does 1:1 mean?

Is there an equivalent to --itemize-changes?

5

Should this be run after a successful upload and is there any value if rclone is already verifying the checksum?

6

If you are using --progress then --stats controls the rate at which the progress messages get printed. So you can use them together provided you are aware of that.

Rclone only transfers changes, yes, but on a file by file basis. So if you have 1000 files to sync and rclone gets through 300 you can quit it and restart and it will transfer the remaining 700.

However if you have one 100G file and you quit rclone and restart it, rclone will start from the beginning of the 100G file (this may change in the future!).

In that scenario rclone will transmit 10 files concurrently with a maximum total bandwidth of 7MByte/s, so each transfer will get roughly 0.7MByte/s

One file on your disk goes to one object on the cloud storage.

Probably the closest thing to that is running with the -v flag which will list every file transferred.

1 Like
7

@ncw, thanks for clarifying these. It's very helpful.

1 Like
8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.