Speed up the copy on Digital Ocean spaces (s3)


I've to transfert 19Gb of images from an OVH dedicated server to a Digital Ocean Space. I made a test on a small portion of images

Transferred: 44.164 MBytes (768.370 kBytes/s)
Errors: 0
Checks: 0
Transferred: 1223
Elapsed time: 58.8s

If I've to do that with 19Gb its will take 6-7 hours. There's any option to speed up ?

The comand I use :

rclone copy /home/xxxxxxx/public_html/public/up spaces:xxxxxx --transfers=14

Without the --transfers=14 its taking 5 min, I tester 40 but its doesn't change.

Thx for you time

Maybe the problem is the small size of the files.

Its possible, the image are 600*x px, JPG and compressed. So there's no solution?

It's not an rclone issue. Most of these providers are slow with small files. About the only thing you can do is turn up the transfers but in your case it looks like Spaces is limiting the number of simultaneous transfers.

Did you try changing the chunk size to 32 MB while running 16 transfers?

If they are tiny files a larger chunk size won't help as these files are likely less than a few Meg if they are 600x px...

They're tiny between 25~100Ko

At that size you're best bet is going to be too create archives of many together if that even fits your use-case

No I can't unzip on a space/s3. I'll have to putt my website in maintenance.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.