Rclone big 7zip files upload to wasabi cloud stuck at 0%

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I am trying to upload a folder containing big 7z encrypted archive files, one is 200gb and the other 80gb.
Rclone kee[ getting stuck at 0% transfer even after one hour.
If i try to upload a folder with some test files ( not 7z archives ) it works flawlesly.
I did successfullly upload theese files with rclone to a onedrive remote, from the same machine.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0

  • os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
  • os/kernel: 10.0.22621.2861 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.21.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

I am using Wasabi Cloud, still in demo.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy .\archivi_dati\ wasabi:archivi-freddi/ --s3-upload-concurrency 4 --s3-chunk-size 5G --progress

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[wasabi]
type = s3
provider = Wasabi
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.eu-central-2.wasabisys.com
location_constraint = s3.eu-central-2.wasabisys.com
acl = private
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2023/12/26 21:16:22 DEBUG : rclone: Version "v1.65.0" starting with parameters ["C:\\Users\\sasso\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Rclone.Rclone_Microsoft.Winget.Source_8wekyb3d8bbwe\\rclone-v1.65.0-windows-amd64\\rclone.exe" "copy" ".\\archivi_dati\\" "wasabi:archivi-freddi/" "--s3-upload-concurrency" "4" "--s3-chunk-size" "5G" "--progress" "-vv"]
2023/12/26 21:16:22 DEBUG : Creating backend with remote ".\\archivi_dati\\"
2023/12/26 21:16:22 DEBUG : Using config file from "C:\\Users\\sasso\\AppData\\Roaming\\rclone\\rclone.conf"
2023/12/26 21:16:22 DEBUG : fs cache: renaming cache item ".\\archivi_dati\\" to be canonical "//?/E:/archivi_dati"
2023/12/26 21:16:22 DEBUG : Creating backend with remote "wasabi:archivi-freddi/"
2023/12/26 21:16:22 DEBUG : wasabi: detected overridden config - adding "{wmfIr}" suffix to name
2023/12/26 21:16:22 DEBUG : Resolving service "s3" region "us-east-1"
2023/12/26 21:16:22 DEBUG : fs cache: renaming cache item "wasabi:archivi-freddi/" to be canonical "wasabi{wmfIr}:archivi-freddi"
2023/12/26 21:16:22 DEBUG : archive1.enc.7z: Need to transfer - File not found at Destination
2023/12/26 21:16:22 DEBUG : archive2.enc.7z: Need to transfer - File not found at Destination
2023/12/26 21:16:22 DEBUG : archive1.enc.7z: multi-thread copy: disabling buffering because source is local disk
2023/12/26 21:16:22 DEBUG : archive2.enc.7z: multi-thread copy: disabling buffering because source is local disk
2023/12/26 21:16:22 DEBUG : S3 bucket archivi-freddi: Waiting for checks to finish
2023/12/26 21:16:22 DEBUG : S3 bucket archivi-freddi: Waiting for transfers to finish
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 2, 0%
Elapsed time:     15m29.0s
 *                     archive1.enc.7z: transferring
 *                  archive2.enc.7z: transferring

I do have a good connection, 1Gb fiber, upload is usually 300Mbs or more.
Thanks to anyone willing to help me

2

welcome to the forum,

based on what you posted, should not be the issue.

with S3 remotes, before rclone can upload a local file, rclone takes time to calculate its MD5 hash.
so the larger the source file, the longer it takes.
and by default, rclone works on multiple files at the same time.

fwiw, i have been uploading .7z to wasabi, on a daily basis for over five years.

if you want rclone to start the upload process sooner than later, maybe try something like
--checkers=1 --transfers=1 --order-by=size,ascending

3

Thanks man

I am trying what that options right now, are the 7z files you are uploading encrypted too? even if, i guess, that shouldn't make any difference

Are you waiting a lot too? i suppose the wait time implied to verify the hash is proportional to the file size

4

yes, the .7z are encrypted and good guess.

yes, and yes

one the windows servers, i manage, uses REFS, softraid, on three slow hard disks and slow cpu.

in fact, rclone performfance was so slow on larger veeam backup files, i started using VSS snapshots as the source.
and that is what prompted me to create my wiki
How-to-enable-VSS-for-rclone

5

Thanks for all theese details, the vss solution is very interesting too.

The upload has started in about 10 mins for the 80gb file.

Thanks again

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.