Hi I try to upload large file to Scaleway
It fails several times on 3 file
Two files are 6Gb and largest is 384 Gb
I learned about about multipart upload threshold of 4.656Gi and Scaleway limitation of 1000 chunks!
I solved problem for two smaller file (6Gb) with flag --s3-copy-cutoff 0
But this flag no go for 384 Gb file.
I read forum and find an suggestion to use flag --s3-chunk-size so that number of chunks be smaller of 1000 - that in my case wil be --s3-chunk-size 384M but that not worked either.
Rclone is just killed after a while.
Here is tek stuf:
I use Ubuntu server:
Distributor ID: Ubuntu
Description:|Ubuntu 20.04.3 LTS| - server version
Release:|20.04|
Codename:|focal|
Virtual machine:
1 vcpu
2 GB RAM
20 GB boot drive
600 Gb temp drive
What is your rclone version (output from rclone version
)
rclone v1.56.0
Which cloud storage system are you using? (eg Google Drive)
Scaleway
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy Wasabi:rst.arhiva Scaleway:rst.arhiva --retries 10 --retries-sleep 15s --s3-chunk-size 384 -P
A log from the command with the -vv
flag
Screen #1 (flag: --s3-copy-cutoff 0 )
-----------------------------------------
rclone copy Wasabi:rst.arhiva Scaleway:rst.arhiva --retries 10 --retries-sleep 15s --s3-copy-cutoff 0 -P
Enter configuration password:
password:
Transferred: 87.888Gi / 395.192 GiByte, 22%, 51.022 MiByte/s, ETA 1h42m47s
Checks: 97 / 97, 100%
Transferred: 2 / 3, 67%
Elapsed time: 30m41.0s
Transferring:
* v152ufugvdlri395fgi5p0…pcvqgrblogqki28mmsqv3g: 19% /383.277Gi, 51.419Mi/s, Killed
Screen #2 (flag: --s3-chunk-size 430M )
------------------------------------------------------
rclone copy Wasabi:rst.arhiva Scaleway:rst.arhiva --retries 10 --retries-sleep 15s --s3-chunk-size 430M -P
Enter configuration password:
password:
Transferred: 6.997Gi / 383.277 GiByte, 2%, 30.363 MiByte/s, ETA 3h31m30s
Checks: 99 / 99, 100%
Transferred: 0 / 1, 0%
Elapsed time: 2m26.9s
Transferring:
* v152ufugvdlri395fgi5p0…pcvqgrblogqki28mmsqv3g: 1% /383.277Gi, 29.081Mi/s, Killed
Any idea how to upload 384 Gb file to Scaleway
I use FreeNAS with Rclone implementation for "cloud sync tasks" and I upload all this files to Wasabi with no problem. FreeNAS use 1.50.2 version of Rclone.
I try to install that version but I have no success to upload to Scaleway.
Any help / idea is welcome!
Thanks