Parallel and concurrent copy

[Abdullah1 (Abdullah)]

What is the problem you are having with rclone?

I am working on a project which should copy data from one cloud provider to the other one and reverse. Like google-to-box and box-to-google.
Now there can be multiple users and TBs of data for each user for each remote.
I have 2 questions
1: How many concurrent sync/copy call should I make assuming there are 100 users data to transfer within a single source and destination account and data can be in TeraBytes (more then google 750Gs daily limit).
2: How many parallel cloud transfer should be running? Or how many of the above process should I run at the same time.

Basically I am asking about the load management on rclone application.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

• os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
• os/kernel: 10.0.19045.3324 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.20.6
• go/linking: static
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Box

The command you were trying to run (eg rclone copy /tmp remote:tmp)

http://127.0.0.1:5572/sync/copy

The rclone config contents with secrets removed.

Paste config here

A log from the command that you were trying to run with the -vv flag

Paste  log here

[kapitainsky]

@Abdullah1

1: How many concurrent sync/copy call should I make assuming there are 100 users data to transfer within a single source and destination account and data can be in TeraBytes (more then google 750Gs daily limit).
2: How many parallel cloud transfer should be running? Or how many of the above process should I run at the same time.

As many as needed to reach your goals and staying at the same time within constraints imposed by your system (network bandwidth and latency, RAM, CPU usage etc.) and used remotes limitations (APIs throttling limits, contractual quotas, fair usage policies etc.).

Given number of variables involved probably the best approach is to build test system and run extensive experiments to find optimal setup.

Good design should also allow ad hoc changes of system configuration as most likely you do not have any SLAs with remotes' providers - so expect them changing things any time (e.g. more aggressive throttling, new quota etc.).