rclone copy operation from AWS S3 to a local disk has been stuck at 100% for over 12 hours with no visible progress, even through I estimate it only transferred <2% of the data. The S3 bucket in question should have about 750GB and 830k objects (according to the AWS console).
The local path to which the copy is being attempted already contained some of the data from a previously aborted attempt to copy in the data.
- There is plenty of local disk space available.
- The local disk is a spinning-platter drive; I'm not sure if the rclone process is stuck because it's trying to checksum files on the local drive and it's slow.
- This is difficult to verify, but I believe that most files are located in the same "directory" in S3. Consequently, there may be a performance problem with just listing a directory with such a large number of files, either locally or remotely.
rclone command did copy some of the data initially, but progress stopped after about 2 hours.
I'm not sure if I should just abort and retry with
--fast-list, or other flags. IOW: sunk cost bias versus persistence.
rclone v1.55.1-DEV - os/type: darwin - os/arch: amd64 - go/version: go1.16.4 - go/linking: dynamic - go/tags: cmount
rclone was installed from nixpkgs.
macOS 10.14.6, 64-bit
$ rclone copy --progress s3:my-bucket ./local-path Transferred: 10.142G / 10.142 GBytes, 100%, 1.626 MBytes/s, ETA 0s Checks: 114662 / 114662, 100% Transferred: 107427 / 107427, 100% Elapsed time: 15h41m40.5s
[s3] type = s3 provider = AWS region = us-east-1 env_auth = true
No log provided, because I did not initially run the command with
-vv, and have left the command running in case it makes progress.