Very long running rclone tasks

Help and Support
1

What is the problem you are having with rclone?

Some tasks that have been running correctly for months are now suddenly fail to "finish".

What is your rclone version (output from rclone version)

rclone v1.54.0

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --config=tmp.conf --max-age 7d --buffer-size 50M -v remote:/ local-s3:folder/subfolder/ --progress

The rclone config contents with secrets removed.

[local-s3]
type = s3
env_auth = true
region = 
location_constraint = 
server_side_encryption = AES256
storage_class = STANDARD

[markit-deltaone]
type = sftp
host = remote.com
user = username
pass = password
port = 22
md5sum_command = none
sha1sum_command = none

A log from the command with the -vv flag

Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m22.1sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m22.6sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m23.1sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m23.6sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m24.1sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m24.6sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m25.1sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m25.6sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m26.1sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m26.6sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m27.1sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
Checks:             96454 / 96454, 100%
Transferred:         5978 / 5978, 100%
Elapsed time:   6h24m27.6sTransferred:   	    4.826G / 4.826 GBytes, 100%, 25.102 MBytes/s, ETA 0s
2

The tasks used to finish in abut 4 hours but since about 2 weeks, we don't get a clean exit.
The job keep running for 14 hours with no set ETA and we end up killing the process.

I did a size and I see this

# rclone size --config=tmp.conf remote:/
Total objects: 527475
Total size: 203.873 GBytes (218906747372 Bytes)

If I do a copy with --dry-run and "--checkers=25" I get,

2021/10/26 12:24:31 NOTICE:
Transferred:        4.178G / 4.178 GBytes, 100%, 2.426 GBytes/s, ETA 0s
Checks:             61394 / 61394, 100%
Transferred:         4162 / 4162, 100%
Elapsed time:   4h43m42.9s

I tried to increase "--checkers" to 25 and the job stil ran for 9 hrs and we ended up killing that too.

The logs keep showing me copied new files every one a while, telling me that nothing is wrong as such.
But no errors and the tasks doesn't complete.

What else to look for hints on ETA or speedup or any guidance at all.

Thanks,
Shirish

3

hi,

  • best to update rclone to v1.65.2 and test again.
  • need to see a full debug log.
  • the remotes in your command do not match the remotes in the config file?
  • local-s3 is that a local s3 backend, such as minio, what is it and where is it located?
  • if the same exact command suddenly fails, it might be a network issue, did you reboot internet equipment, routers and so on?
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.