Rclone copy stuck even after the file is already uploaded

What is the problem you are having with rclone?

rclone stuck/hang after a file has finish its transfer. The file is already uploaded but rclone was still running, as you can see in the log, the INFO part keep repeating the same information.

For your information, rclone is run using docker in Github Actions runner during our CI process. At first, it happened on the latest version of rclone docker image, and so then I decided to do the following:

  1. enable verbose mode (-vv)
  2. enable headers, requests and responses dump
  3. enable ignore-checksum
  4. pin docker image version to 1.70.0

However, it still did not resolve the problem that I am facing.

Run the command 'rclone version' and share the full output of the command.

Using docker rclone 1.70.0
docker.io/rclone/rclone:1.70.0@sha256:223fbd5db2554214b1da148db589f300c193901c74b2a55336f486cf3af4ffae

  #0 building with "default" instance using docker driver
  #1 [internal] load build definition from Dockerfile
  #1 transferring dockerfile: 115B done
  #1 DONE 0.0s
  #2 [internal] load metadata for docker.io/rclone/rclone:1.70.0
  #2 ...
  #3 [auth] rclone/rclone:pull token for registry-1.docker.io
  #3 DONE 0.0s
  #2 [internal] load metadata for docker.io/rclone/rclone:1.70.0
  #2 DONE 1.7s
  #4 [internal] load .dockerignore
  #4 transferring context: 2B done
  #4 DONE 0.0s
  #5 [internal] load build context
  #5 transferring context: 260B done
  #5 DONE 0.0s
  #6 [1/2] FROM docker.io/rclone/rclone:1.70.0@sha256:223fbd5db2554214b1da148db589f300c193901c74b2a55336f486cf3af4ffae
  #6 resolve docker.io/rclone/rclone:1.70.0@sha256:223fbd5db2554214b1da148db589f300c193901c74b2a55336f486cf3af4ffae done
.....

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "$CDM_DIST" "s3-remote:${CDM_STORAGE_BUCKET}" --ignore-existing --no-traverse --header-upload='Cache-Control: max-age=31536000' -vvvv --ignore-checksum --checkers=32 --s3-no-check-bucket --dump headers --dump requests --dump responses

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[s3-remote]
type = s3
provider = AWS
env_auth = true
region = eu-west-1
endpoint = https://s3.amazonaws.com

A log from the command that you were trying to run with the -vv flag

Log file

welcome to the forum,

for testing, simplify as much as possible.
run rclone on command line with as few flags as possible to reproduce the issue.

Update here. I found out that the copy process failed from v1.70.0 and above. After reverting to docker image version v1.69.3 (sha256:1f497a86a6466395e62a5886613a14b7b18809543566ef9fa35fa1371a7ecc0f) it works fine again.

Another update: it is also failing when running rclone after installing the binary in the machine.

This is being tracked in This is being tracked in deadlock in --no-traverse when all transfers skipped · Issue #8656 · rclone/rclone · GitHub

Can anyone having this problem please try

v1.71.0-beta.8831.f41ac3342.fix-8656-march-deadlock on branch fix-8656-march-deadlock

1 Like

Yes, the fix works. Will wait for official release then. Thank you for fixing it :folded_hands:

1 Like

Thanks for testing @mqayyuum - once I'm sure of it, I'll put this fix in v1.70.3

I found another deadlock in that code - here is an update

v1.71.0-beta.8837.f7b6572b2.fix-8656-march-deadlock on branch fix-8656-march-deadlock (uploaded in 15-30 mins)

I've merged this fix to master now which means it will be in the latest beta in 15-30 minutes and released in v1.71 and v1.70.3

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.