Rclone copy from S3 stuck at "100%"

Help and Support
1

What is the problem you are having with rclone?

An rclone copy operation from AWS S3 to a local disk has been stuck at 100% for over 12 hours with no visible progress, even through I estimate it only transferred <2% of the data. The S3 bucket in question should have about 750GB and 830k objects (according to the AWS console).

The local path to which the copy is being attempted already contained some of the data from a previously aborted attempt to copy in the data.

  • There is plenty of local disk space available.
  • The local disk is a spinning-platter drive; I'm not sure if the rclone process is stuck because it's trying to checksum files on the local drive and it's slow.
  • This is difficult to verify, but I believe that most files are located in the same "directory" in S3. Consequently, there may be a performance problem with just listing a directory with such a large number of files, either locally or remotely.

The rclone command did copy some of the data initially, but progress stopped after about 2 hours.

I'm not sure if I should just abort and retry with -vv, --fast-list, or other flags. IOW: sunk cost bias versus persistence.

What is your rclone version (output from rclone version)

rclone v1.55.1-DEV
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.4
- go/linking: dynamic
- go/tags: cmount

rclone was installed from nixpkgs.

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS 10.14.6, 64-bit

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ rclone copy --progress s3:my-bucket ./local-path
Transferred:       10.142G / 10.142 GBytes, 100%, 1.626 MBytes/s, ETA 0s
Checks:            114662 / 114662, 100%
Transferred:       107427 / 107427, 100%
Elapsed time:  15h41m40.5s

The rclone config contents with secrets removed.

[s3]
type = s3
provider = AWS
region = us-east-1
env_auth = true

A log from the command with the -vv flag

No log provided, because I did not initially run the command with -vv, and have left the command running in case it makes progress.

2

Are you seeing disk IO? CPU IO?

1 Like
3

Yes. Activity Monitor shows 3-7% CPU activity on the rclone process, and "Bytes Read" slowly ticks up on the Disk tab (it moved from around 304.50GB to 304.90GB) in the time it took to write this reply.

So I guess it's not stuck? There's about 425GB of data on the destination that had previously been copied from the S3 bucket, and maybe it's being checksummed very very slowly?

4

Seems to be moving, but with no logs, it's a bit of a guessing game.

1 Like
5

It completed eventually (like 40 hours later). Thanks for your help!

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.