Slow upload times to Wasabi

Help and Support
1

I am averaging roughly 1 TB per week on an upload from an external NAS drive to Wasabi. I am trying to upload the entire drive (7 TB), which contains many folders and files above 100MB. I need to know if this is normal. If so, should I wait the full 7 weeks? If it is not normal, maybe I am doing something wrong. I also noticed the ETA is incorrect, but might have something to do with reading the system volume information.

Below is the command and status of the upload....

C:\Users\Joe King\Desktop\rclone-v1.66.0-windows-amd64>rclone copy G:\ wasabi:gdrivenewbusiness --progress --checksum
2024/04/03 19:32:40 ERROR : System Volume Information: failed to open directory "System Volume Information": open \?\G:\System Volume Information: Access is denied.
Transferred: 1.918 TiB / 2.389 TiB, 80%, 2.737 MiB/s, ETA 2d2h9m
Errors: 1 (no need to retry)
Checks: 12749 / 12749, 100%
Transferred: 26093 / 36105, 72%
Elapsed time: 1w1d14h6m24.0s
Transferring:

  • . . Anonymous/! …own & Brox 5-13-17.avi: 72% /9.069Gi, 917.962Ki/s, 46m38s
  • . . Anonymous/! … 20.33.17 qwqwqwqw.avi: 62% /9.069Gi, 926.604Ki/s, 1h4m57s
  • . . Anonymous/! …BLD_S012_S002_T001.MOV: 12% /6.675Gi, 947.245Ki/s, 1h48m14s
  • . . Anonymous/! … Brown 5-13-17 2.avi: transferring
2

welcome to the forum,

with s3 remotes, uploads of a new file is a two step process.
first, rclone has to take time to calculate the checksum of the entire source file.
second, after the checksum has been calculated, rclone can start the upload.

as for the upload speed, i have no issues saturating my 1Gbps internet connection.

and as a separate issue, are you sure you need, --checksum?
that is going to really slow down rclone for files that have not changed.

when you posted, there was a template of questions for you to answer???

3

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

Slow Upload times

Run the command 'rclone version' and share the full output of the command.

I can't, as it is uploading data, and I don’t want to interrupt it.\

> rclone copy G:\ wasabi:gdrivenewbusiness --progress –checksum

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

I can't, as it uploads data, and I don’t want to interrupt it.

A log from the command that you were trying to run with the -vv flag

Is there a log file I can pull without interrupting the upload?
is this what you are looking for?
"rclone/v1.66.0" -
3EF24F999510AFF2765324C958EB6BC5B0F73831243780F6B98EB38EC54B1FC7 gdrivenewbusiness [11/Apr/2024:09:24:48 +0000] 184.184.12.43 3EF24F999510AFF2765324C958EB6BC5B0F73831243780F6B98EB38EC54B1FC7 1E74F19C59D1340B:B REST.PUT.PART .+++++++++++++++.In+House%2F%21++++++++++++++++++++Projects%2F%21+++++++++++++++++Premium+Consulting+And+Law+MEDIA++%28PCALM%29%2F%21++++++++++++Initial+Movie+Script%2FScenes+Or+Events%2FRuth+NV+2022-11-15%2F2022-11-15++Trip+to+Ruth%2FAndrew+Jackson%2FVideos%2FPXL_20221117_195620752-005.mp4 "PUT /.%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20.In%20House/%21%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Projects/%21%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20Premium%20Consulting%20And%20Law%20MEDIA%20%20%28PCALM%29/%21%20%20%20%20%20%20%20%20%20%20%20%20Initial%20Movie%20Script/Scenes%20Or%20Events/Ruth%20NV%202022-11-15/2022-11-15%20%20Trip%20to%20Ruth/Andrew%20Jackson/Videos/PXL_20221117_195620752-005.mp4?partNumber=697&uploadId=_RFszGj_Go0PbX46fMwjVq3xfAeOl50O3RTiDDsQxx-SxiD5VKjfrwzHWeWtYf6ods7BXs6pc9kimoU6QraFDUbxGiXu4mVQ58732jxI1l5DJ0CQXMIk-peDgGbDFz4z" 200 - - 5242880 22057 0 ""
4

hard to help you unless you provide the requested information.

and what is the specific reason to use --checksum?
that is going to really slow down rclone.

then open a new terminal window

no, and next time, for debug output, use -vv

5

I am using Checksum because the files and folders have funky long names with multiple characters like !!, and I was under the impression this would check the file and folder formats before copying to ensure they are copied correctly. Please let me know if this helps or not.

6

sorry, not understanding what funky long names has to with checksums?

when rclone transfers a file, on s3 remote, it uses checksums for file verification.
use debug output to see that.

--checksum is used for a different purpose. check the rclone docs?

7

Thank you so much for your feedback! I stopped the upload, removed the "--checksum" from the run command, and added "-vv" to see the log. Below are the answers to your questions. Now Rclone is acting like it is starting over. Before, it said, "Transferred: 1.918 TiB / 2.389 TiB, 80%, 2.737 MiB/s, ETA 2d2h9m," and now it is saying, "Transferred: 32.867 GiB / 749.250 GiB, 4%, 2.740 MiB/s, ETA 3d2h21m. I don't know what to make of this. Keep in mind that I am transferring 7 TB of data from an external NAS drive, so the ETA is incorrect and the total Data is incorrect. and I am unclear as to what it has copied to date.

What is the problem you are having with rclone?

Slow Upload times

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.4170 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy G:\ wasabi:gdrivenewbusiness --progress -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[wasabi]
type = s3
provider = Wasabi
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.us-west-1.wasabisys.com

A log from the command that you were trying to run with the -vv flag

2024/04/12 14:30:59 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line - Copy/!          2023-01-21 NOTES/Files added after 2023-09-03/1  Files added after 2023-09-03.docx: Copied (new)
2024/04/12 14:30:59 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line - Copy/!          2023-01-21 NOTES/Files added after 2023-09-03/VD's Notes.docx: Copied (new)
2024/04/12 14:30:59 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line - Copy/!          2023-01-21 NOTES/Files added after 2023-09-03/desktop.ini: Copied (new)
2024/04/12 14:30:59 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/1   Basic.doc: Copied (new)
2024/04/12 14:31:00 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/1   Mexonline.com.doc: Copied (new)
2024/04/12 14:31:00 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/1   SAPI   Basic.doc: Copied (new)
2024/04/12 14:31:01 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/1  URL's.doc: Copied (new)
2024/04/12 14:31:03 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/2 Legal Dictioniary.doc: Copied (new)
2024/04/12 14:31:04 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/DLA-PIPER-MX-AMLL-Webinar_eng.pdf: Copied (new)
2024/04/12 14:31:04 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/Mexican Anti-Money Laundering Law.pdf: Copied (new)
2024/04/12 14:31:06 INFO  : .               .In House/!                    Projects/1   Mexico Legal/!                       NOW/!   2023-06-11 Time Line/!      Holding/MX Constitution & Law/en_mex-int-text-const.pdf: Copied (new)
Transferred:       32.300 GiB / 748.787 GiB, 4%, 2.758 MiB/s, ETA 3d1h53m
Errors:                 3 (no need to retry)
Checks:             23734 / 23734, 100%
Transferred:          977 / 10989, 9%
Elapsed time:   3h22m51.0s
Transferring:
 * .               .In Ho…on/video1770658469.mp4: 81% /1.928Gi, 1.393Mi/s, 4m27s
 * .   .   Anonymous/Mexi…85682369007/zoom_0.mp4:  6% /274.156Mi, 1.129Mi/s, 3m47s
 * .   .   Anonymous/Mexi…85682369007/zoom_1.mp4:  3% /156.588Mi, 224.592Ki/s, 11m31s
 * .   .   Anonymous/Mexi…o Photos/At CBX(1).JPG: 75% /1.475Mi, 339.716Ki/s, 1s
8

welcome

well, not really.
from the rclone docs, Copy files from source to dest, skipping identical files.

check the rclone debug log.

can search the forum about that and the rclone docs about --check-first
to make things more complex, with s3 remotes, before rclone uploads a file, rclone has to take time to compute the md5 hash.

9

Before, when it started it found it had to transfer 2.389 TB of files, and when you stopped it, 1.918 TB had already been transferred out of that.
Now, it's saying it's found 749GB of files and so far 32.867 GB have been transferred.