Rclone copy seems to have restarted?

Help and Support
1

What is the problem you are having with rclone?

rclone copy seems to have restarted on its own and doing something weird, a couple of min ago the stats looked like this:
Transferred: 346.464 GiB / 386.297 GiB, 90%, 6.825 MiB/s, ETA 1h39m36s

Then I came back to check and got these errors:

2024/06/06 16:00:12 ERROR : sftp://bucket-prod@storage.bunnycdn.com:22/videos/: Discarding closed SSH connection: read tcp 66.29.138.103:47296->109.61.89.53:22: i/o timeout

I didnt give it much thought as I got that error before, and it just resumed without any issues, however now the stats shows:
Transferred: 399.408 GiB / 773.759 GiB, 52%, 10.773 KiB/s, ETA 1y8w16h
Checks: 2396 / 2396, 100%

so it checked/uploaded all the files yet now it doubled the amount of GB to upload as you can see if you compare the first one with the second one even though the checks show it has uploaded all files, also there are 5 concurrent uploads that are like: 4k.mp4: 97% /3.003Gi, 1.578Mi/s, 52s all at constant at 1.5mb/s (showing the total as above), yet now it says combined its 10kb/s and the ETA is in 1 year !!

(this is all on a "screen" window)

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.66.0

  • os/version: almalinux 8.9 (64 bit)
  • os/kernel: 4.18.0-513.24.1.el8_9.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

bunny.net uploading through sftp

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -P --disable PartialUploads /root/videos/ bunnycdn:videos/

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[bunnycdn]
type = sftp
host = XXX
user = XXX
port = 22
pass = XXX
shell_type = unix
md5sum_command = none
sha1sum_command = none

Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

rclone copy -P --disable PartialUploads /root/videos/ bunnycdn:videos/ -vv
2024/06/06 16:31:36 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "-P" "--disable" "PartialUploads" "/root/videos/" "bunnycdn:videos/" "-vv"]
2024/06/06 16:31:36 DEBUG : Creating backend with remote "/root/videos/"
2024/06/06 16:31:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/06/06 16:31:36 DEBUG : Reset feature "PartialUploads"
2024/06/06 16:31:36 DEBUG : fs cache: renaming cache item "/root/videos/" to be canonical "/root/videos"
2024/06/06 16:31:36 DEBUG : Creating backend with remote "bunnycdn:videos/"
2024/06/06 16:31:36 DEBUG : Reset feature "PartialUploads"
2024/06/06 16:31:37 DEBUG : sftp://bucket-prod@storage.bunnycdn.com:22/videos/: New connection 66.29.138.103:50512->109.61.89.55:22 to "SSH-2.0-RebexSSH_5.0.8348.0"
2024/06/06 16:31:38 DEBUG : sftp://bucket-prod@storage.bunnycdn.com:22/videos/: Shell type "unix" from config
2024/06/06 16:31:38 DEBUG : sftp://bucket-prod@storage.bunnycdn.com:22/videos/: Relative path resolved to "/videos"
2024/06/06 16:31:38 DEBUG : sftp://bucket-prod@storage.bunnycdn.com:22/videos/: Using root directory "/videos"
2024/06/06 16:31:41 INFO : 116: Set directory modification time (using DirSetModTime)
2024/06/06 16:31:41 DEBUG : Added delayed dir = "116", newDst=116
2024/06/06 16:31:41 INFO : 1160: Set directory modification time (using DirSetModTime)
2024/06/06 16:31:41 DEBUG : Added delayed dir = "1160", newDst=1160
2024/06/06 16:31:42 INFO : 1161: Set directory modification time (using DirSetModTime)
2024/06/06 16:31:42 DEBUG : Added delayed dir = "1161", newDst=1161
2024/06/06 16:31:42 INFO : 1162: Set directory modification time (using DirSetModTime)
2024/06/06 16:31:42 DEBUG : Added delayed dir = "1162", newDst=1162
2024/06/06 16:31:42 INFO : 1163: Set directory modification time (using DirSetModTime)
2024/06/06 16:31:42 DEBUG : Added delayed dir = "1163", newDst=1163
2024/06/06 16:31:43 INFO : 1164: Set directory modification time (using DirSetModTime)
2024/06/06 16:31:43 DEBUG : Added delayed dir = "1164", newDst=1164
2024/06/06 16:31:43 INFO : 1165: Set directory modification time (using DirSetModTime)
2024/06/06 16:31:43 DEBUG : Added delayed dir = "1165", newDst=1165

2

welcome to the forum,

the statistics displayed by rclone is not always accurate in real-time.
can read about --check-first and much discussed in the forum.

in addition, rclone retries and errors can affect the statistics.

3

thanks for the reply, do I stop it then ? the thing is that it says 700GB to upload yet the HD of the server is 405GB usuable, so its weird ... the destination folder HAD a lot more folders than origin as there was a previous upload done .. then what was already uploaded deleted from the origin, and then place new files to upload ... could this be the source of the issue ?

4

stop what?
if there are errors and failed file uploads, rclone will retry and that affects the statistics.

if you want to know exactly what rclone will do, you have at least two options.

  • --dry-run
  • rclone check
5

well I think its restarted the upload, I mean .. it completed the upload, and now it restarted it all over again, if I do --dry-run:

rclone copy -P --disable PartialUploads /root/videos/ bunnycdn:videos/ --dry-run

it gives:

Transferred: 387.462 GiB / 387.462 GiB, 100%, 6.662 GiB/s, ETA 0s
Checks: 2396 / 2396, 100%
Transferred: 2396 / 2396, 100%
Elapsed time: 58.8s
2024/06/06 17:43:04 NOTICE:
Transferred: 387.462 GiB / 387.462 GiB, 100%, 6.662 GiB/s, ETA 0s
Checks: 2396 / 2396, 100%
Transferred: 2396 / 2396, 100%
Elapsed time: 58.8s

If I do check:

rclone check --one-way /root/videos/ bunnycdn:videos/

it gives:

2024/06/06 17:45:04 ERROR : No common hash found - not using a hash for checks
2024/06/06 17:45:08 ERROR : 1166/1166_540.mp4: sizes differ
2024/06/06 17:45:08 ERROR : 1166/1166_360.mp4: sizes differ
2024/06/06 17:45:08 ERROR : 1166/1166_1080.mp4: sizes differ
2024/06/06 17:45:40 ERROR : 1216/1216_4k.mp4: sizes differ
2024/06/06 17:46:02 NOTICE: sftp://bucket-prod@storage.bunnycdn.com:22/videos/: 4 differences found
2024/06/06 17:46:02 NOTICE: sftp://bucket-prod@storage.bunnycdn.com:22/videos/: 4 errors while checking
2024/06/06 17:46:02 NOTICE: sftp://bucket-prod@storage.bunnycdn.com:22/videos/: 2392 hashes could not be checked
2024/06/06 17:46:02 NOTICE: sftp://bucket-prod@storage.bunnycdn.com:22/videos/: 2392 matching files
2024/06/06 17:46:04 Failed to check with 4 errors: last error was: 4 differences found

if i check the current transfer under "screen -r" I see:
Transferred: 433.863 GiB / 773.759 GiB, 56%, 10.773 KiB/s, ETA 1y2w3d
Checks: 2396 / 2396, 100%
Transferred: 2643 / 4791, 55%
Elapsed time: 18h59m39.6s

Transferring:

  • 1169/1169_4k.mp4: 82% /2.672Gi, 1.580Mi/s, 5m3s
  • 1171/1171_1080.mp4: 40% /1.141Gi, 1.556Mi/s, 7m26s
  • 1171/1171_4k.mp4: 6% /3.708Gi, 1.619Mi/s, 36m25s
  • 1171/1171_540.mp4: 40% /495.099Mi, 1.588Mi/s, 3m4s

this is the one I mean if i should stop it ?

and if I run in a separate screen:

rclone copy -P --disable PartialUploads /root/videos/ bunnycdn:videos/ --check-first

it just uploads it all over again even though it checks 100%:

rclone copy -P --disable PartialUploads /root/videos/ bunnycdn:videos/ --check-first
Transferred: 40.066 MiB / 387.462 GiB, 0%, 2.922 MiB/s, ETA 1d13h42m
Checks: 2396 / 2396, 100%
Transferred: 15 / 2396, 1%
Elapsed time: 2m7.7s
Transferring:

  • 1166/1166_1080.mp4: 1% /1.037Gi, 1.410Mi/s, 12m25s
  • 1166/1166_360.mp4: 8% /114.552Mi, 1.330Mi/s, 1m18s
  • 1166/1166_4k.mp4: 0% /3.371Gi, 1.446Mi/s, 39m40s
  • 1166/1166_540.mp4: 1% /449.093Mi, 1.387Mi/s, 5m18s
6

The statistics make sense to me, after a failure, rclone will finish and then retry the whole transfer to clear any failures. So this means the checks/data size doubled. Now, based on your statistics from first post, it shows it's doing checks, so just confirming the file is already there, not transferring the data again. rate is showing slow at 10kb/s as it's not really transferring anything so averages down. It could be it was just taking a long time to do the checksum or whatever check.

7

ahhh so its doing a check, ok makes sense, thanks, i will let it finish, thanks a lot :slight_smile:

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.