What is rclone doing after s3 upload reaches 100%?

Help and Support
1

What is the problem you are having with rclone?

for copying of files to a s3 remote, I notice there are 2 distinct delay at the start and end of the upload especially when copying large files.

The start of the upload is known to be rclone doing a checksum of the file being uploaded,
however, after the file is fully uploaded (AKA the console shows 100%), it also stays at 100% for a significant amount of time.

what is rclone doing during this 2nd pause at the end of the upload?

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -P "C:/source" "s3:bucket/dest" --fast-list --cache-dir ".\cache" --transfers 32 --checkers 64 --check-first
2

I can use my eight ball to guess it's running a checksum.

You deleted the line for your rclone version.
You didn't include a log file either.
You didn't include the rclone.conf.

Any reason to just ignore those in the template? We have to back and forth and I waste my time guessing an answer without all the information.

3

Reason being I'm not asking for debugging, just curious.

Unless I should have post curiosity type questions on another thread? If so, which one thread I post it in?

4

The reason for the template is for that very thing you've asked.

If you have a question, answering that question requires a set of information.

Without knowing what version, you are running, the answer may or may not be right.

Without knowing your rclone.conf, i.e. are you using a crypt remote when it changes checksums every upload.

A debug log validates what version you are running, specifically what commands you are running as many folks post a command, but what they actually execute is different (happens more than you think).

So the reason the template came into existence is to help you faster and more accurately the first time so be avoiding/not doing it, it wastes time for folks volunteering their time to answer your curious question.

Majority of questions tend to be answered themselves when a debug log is posted.

In my example, you can see the checksum at the end:

rclone copy jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv DB: -vvv
2022/01/26 08:43:59 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/01/26 08:43:59 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv" "DB:" "-vvv"]
2022/01/26 08:43:59 DEBUG : Creating backend with remote "jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv"
2022/01/26 08:43:59 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/01/26 08:43:59 DEBUG : fs cache: adding new entry for parent of "jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv", "/data"
2022/01/26 08:43:59 DEBUG : Creating backend with remote "DB:"
2022/01/26 08:43:59 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Sizes differ (src 404291584 vs dst 1504953150)
2022/01/26 08:44:00 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 1/9
2022/01/26 08:44:03 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 2/9
2022/01/26 08:44:05 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 3/9
2022/01/26 08:44:07 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 4/9
2022/01/26 08:44:10 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 5/9
2022/01/26 08:44:11 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 6/9
2022/01/26 08:44:14 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 7/9
2022/01/26 08:44:16 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 8/9
2022/01/26 08:44:18 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Uploading chunk 9/9
2022/01/26 08:44:19 DEBUG : Dropbox root '': Adding "/jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv" to batch
2022/01/26 08:44:19 DEBUG : Dropbox root '': Batch idle for 500ms so committing
2022/01/26 08:44:19 DEBUG : Dropbox root '': Committing sync batch length 1 starting with: /jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
2022/01/26 08:44:21 DEBUG : Dropbox root '': Upload batch completed in 68.160257ms
2022/01/26 08:44:21 DEBUG : Dropbox root '': Committed sync batch length 1 starting with: /jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
2022/01/26 08:44:21 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: dropbox = 203d9deeb61e36fb7a62a227e9c5ad5fc7f86b1bf1278a7ca4c7368dc0e0f739 OK
2022/01/26 08:44:21 INFO  : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Copied (replaced existing)
2022/01/26 08:44:21 INFO  :
Transferred:   	  385.562 MiB / 385.562 MiB, 100%, 17.620 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        21.8s

2022/01/26 08:44:21 DEBUG : 10 go routines active
2022/01/26 08:44:21 INFO  : Dropbox root '': Commiting uploads - please wait...

So for larger files/slower sources, it has to checksum the file to validate the upload and that takes time depending on your source system/disk speed/size of the file.

felix@gemini:/data$ time md5sum jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
99a4778625f050d24ecf4e75e1512365  jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv

real	0m1.708s
user	0m0.527s
sys	0m0.125s
felix@gemini:/data$ du -ms jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
386	jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv

So why not post what's asked and if you think something needs to be changed, ask about it as over time, we've configured the template to capture all the needed information for the majority of questions that come in. Is it perfect? Not a chance. Is is very good? Yes, it is.

This case, would have been helpful the first time as we'd have zero back and forth and you'd have a very accurate answer first time around.

Win-win for both of us.

5

hi,
--cache-dir does nothing with rclone copy, so that can be removed.

[/quote]
best to test using the simplest test command copy a single flie.
rclone copy -P "C:/source/singlefile" "s3:bucket/dest"

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.