Trouble migrating with rclone

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I'm having trouble migrating with rclone, currently I'm using rclone to migrate minio's data to a distributed storage system that supports the S3 protocol, minio's data is about 10TB, there is only a first-level directory, and the number of files is as high as 5 million, even if you use rclone to join --fast-list, it can't be completed, it will keep prompting Elapsed time: 15.0sTransferred: 0 B / 0 B, - , 0 B/s, ETA -, no data is passed

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: oracle 8.9 (64 bit)
  • os/kernel: 5.4.17-2136.330.7.1.el8uek.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

miniO

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -P -vv --fast-list --no-check-certificate --transfers=10 --checkers=10 --buffer-size=300M --log-file /appl/rclone_copy.log --files-from /appl/file_src/xaa minio:files HUAWEIOBS:cloudfiles

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

![image|649x500](upload://qbsx4qlFhOjnQMpgeYGQnvWXgb0.png)

A log from the command that you were trying to run with the -vv flag

Elapsed time:      3m59.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      3m59.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m0.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m0.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m1.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m1.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m2.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m2.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m3.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m3.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m4.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m4.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m5.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m5.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m6.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m6.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m7.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m7.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m8.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m8.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m9.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m9.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m10.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m10.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m11.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m11.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m12.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m12.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m13.0sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      4m13.5sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -

Give it time. It will be completed (given you have enough RAM to store listing). How long it takes to list and start transferring 5m objects depends on your setup - can be hours easy.

See similar subject discussions:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.