Is there a way to force rclone move to send 1 file at a time?

On this particular box, I have a staging area of limited size. I need to get the files (large, 10GB+) uploaded to my GDrive remote as quickly as possible and deleted as quickly as possible. At present, rclone move uploads 4 at a time, so each file is getting only 25% of my bandwidth. Rather than uploading 4 files at 2.5 MB/s, I want to upload 1 at a time at 10 MB/s.

In this hypothetical scenario, it takes 100 minutes for the first file to finish uploading and to get deleted (the other 3 may finish within 10-15 minutes, but that's not what I want on this box). Instead, I want the first file to be uploaded over 25 minutes, deleted, next file uploaded over 25 minutes, deleted, and so on. It allows me to provision my staging area more efficiently.

Is there a way to force this?

1 Like

when you posted your question, there were a bunch of questions that you were asked.
can you please supply that info?

yes, just add this flag to force rclone to use 1 transfer at a time:
--transfers 1

This is fine for large files, but a single transfer will result in poor performance on small files (because there is a not insignificant delay in opening and closing files, and thus a single transfer at a time may result in significant dead-time between each file when the files are small).

So if you have a mix of both very large and very small files it may be more efficient to do it in 2 parts. One for the large files with a single transfer. And one for the remainder of smaller files with 4 transfers. That would look something like this:

rclone move /localpath GdriveRemote:/remotepath --transfers 1 --min-size 50M --fast-list

rclone move /localpath GdriveRemote:/remotepath --fast-list

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.