Rclone stuck at Large Folders

Help and Support
1

Dear Community, i have a strange Situation with rclone, we currently migrate Folders mit more than 150 Million Files in it from One S3 Provider to Another. Most of them worked like a charm but one of them is driving me cracy.

What is the problem you are having with rclone?

The Command is running form more than 15 hours, Listing all the 148863722 files but only checks 2386797 of them. After that the log just print out the same status an the transfer is stuck.

I've already tried the Workaround Big syncs with millions of files - Howto Guides - rclone forum but this will end with the same result. At some time rclone just stop transferring files. RAM is not maxed out. No OOM Kills or any thing else.

Hardware:

  • Debian 13
  • 12 oder 24 CPU Cores (no differcence)
  • 24 / 32 or 64GB of RAM (no difference)

Run the command 'rclone version' and share the full output of the command.

rclone v1.74.3

  • os/version: debian 13.5 (64 bit)
  • os/kernel: 6.12.90+deb13.1-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.26.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3 Services form two different Cloud Provider.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy src:src-bucket dst:dst-bucket --transfers 64 --checkers 128 --fast-list=false --s3-list-chunk 1000 --no-traverse --buffer-size 0 --max-backlog 200 --stats=2m --stats-log-level NOTICE --log-file=migration-cloud.log --include "/1*/**"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[src]
type = s3
provider = Other
env_auth = false
access_key_id = XXX
secret_access_key = XXX
region = other-v2-signature
endpoint = xxx
[dst]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = xxx
acl = private

A log from the command that you were trying to run with the -vv flag

2026/06/26 04:58:07 NOTICE:
Transferred:        1.374 MiB / 1.374 MiB, 100%, 25.147 KiB/s, ETA 0s
Checks:           2386797 / 2386797, 100%, Listed 148863722
Transferred:         1561 / 1561, 100%
Elapsed time:   16h26m0.0s

2026/06/26 05:00:07 NOTICE:
Transferred:        1.374 MiB / 1.374 MiB, 100%, 25.147 KiB/s, ETA 0s
Checks:           2386797 / 2386797, 100%, Listed 148863722
Transferred:         1561 / 1561, 100%
Elapsed time:   16h28m0.0s

2026/06/26 05:02:07 NOTICE:
Transferred:        1.374 MiB / 1.374 MiB, 100%, 25.147 KiB/s, ETA 0s
Checks:           2386797 / 2386797, 100%, Listed 148863722
Transferred:         1561 / 1561, 100%
Elapsed time:   16h30m0.0s

2026/06/26 05:04:07 NOTICE:
Transferred:        1.374 MiB / 1.374 MiB, 100%, 25.147 KiB/s, ETA 0s
Checks:           2386797 / 2386797, 100%, Listed 148863722
Transferred:         1561 / 1561, 100%
Elapsed time:   16h32m0.0s

2026/06/26 05:06:07 NOTICE:
Transferred:        1.374 MiB / 1.374 MiB, 100%, 25.147 KiB/s, ETA 0s
Checks:           2386797 / 2386797, 100%, Listed 148863722
Transferred:         1561 / 1561, 100%
Elapsed time:   16h34m0.0s

2026/06/26 05:08:07 NOTICE:
Transferred:        1.374 MiB / 1.374 MiB, 100%, 25.147 KiB/s, ETA 0s
Checks:           2386797 / 2386797, 100%, Listed 148863722
Transferred:         1561 / 1561, 100%
Elapsed time:   16h36m0.0s


When running rclone with the paramet -vv it just stops during the transfer. listing continues until 14 Millions Files but Checks and Transfers are not executed:

2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffcb264c0593ca69cfea7495bde70a71de103c: size = 232 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffcb264c0593ca69cfea7495bde70a71de103c: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffcb264c0593ca69cfea7495bde70a71de103c: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff3278f23f7b54c076dc9ded2903d1810e64e5: size = 97 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff3278f23f7b54c076dc9ded2903d1810e64e5: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffd5ea0884d7f14d6d0aaa2426203bd91db045: size = 452 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffd5ea0884d7f14d6d0aaa2426203bd91db045: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffdcec72ce866389bc8a227226db3c75b3bacc: size = 2419 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff7d87a0972fbf3d831e6cec2182d7c93ae21b: size = 7849 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffdcec72ce866389bc8a227226db3c75b3bacc: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff7d87a0972fbf3d831e6cec2182d7c93ae21b: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff7d87a0972fbf3d831e6cec2182d7c93ae21b: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff3278f23f7b54c076dc9ded2903d1810e64e5: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffdcec72ce866389bc8a227226db3c75b3bacc: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff416654d71173f706c305aef2344d685578e1: size = 10436 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff416654d71173f706c305aef2344d685578e1: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff416654d71173f706c305aef2344d685578e1: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/fffefead9ea7eacefba143a46b410332e20f6c0d: size = 12345 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/fffefead9ea7eacefba143a46b410332e20f6c0d: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/fffefead9ea7eacefba143a46b410332e20f6c0d: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff832725dca57dab167c0210f55e8c1c89f972: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff832725dca57dab167c0210f55e8c1c89f972: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff858fc6084b40c05300213348a48986831f10: size = 471 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff858fc6084b40c05300213348a48986831f10: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff858fc6084b40c05300213348a48986831f10: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8a7e505912d7acc103f578eb59e7b6ebc6f5: size = 98 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8a7e505912d7acc103f578eb59e7b6ebc6f5: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8a7e505912d7acc103f578eb59e7b6ebc6f5: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8d6f864a716f78b858af75334ae0aba38df1: size = 463 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8d6f864a716f78b858af75334ae0aba38df1: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8d6f864a716f78b858af75334ae0aba38df1: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8ed9b94fc3c1a03a99b33ff8aebfa4380eb7: size = 12960 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8ed9b94fc3c1a03a99b33ff8aebfa4380eb7: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff8ed9b94fc3c1a03a99b33ff8aebfa4380eb7: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/fffee271d418fa4127523892d1c547103e2959fd: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff9ffbb40cabaffeada620376070b1917817e6: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff9ffbb40cabaffeada620376070b1917817e6: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff0bd940627c43b6eb66251a644f9912c41910: size = 8281 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff0bd940627c43b6eb66251a644f9912c41910: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff0bd940627c43b6eb66251a644f9912c41910: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/fffeecfd6a2e09bed950fd95ab73d964db49e2ee: size = 240 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/fffeecfd6a2e09bed950fd95ab73d964db49e2ee: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/fffeecfd6a2e09bed950fd95ab73d964db49e2ee: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffc2738ace7abe4d521ba265ea9c8eb55f8470: size = 291 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffa838b7a08f77758b935c9fd4da39ff0e94e0: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffc2738ace7abe4d521ba265ea9c8eb55f8470: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffc2738ace7abe4d521ba265ea9c8eb55f8470: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff74d99fec3b9f896264c91dd0b7a2df07ca5a: size = 472 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff74d99fec3b9f896264c91dd0b7a2df07ca5a: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff74d99fec3b9f896264c91dd0b7a2df07ca5a: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff4576b383a0832b55a3ca3211cad466b9b112: size = 475 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff4576b383a0832b55a3ca3211cad466b9b112: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffc36354e6ccd4ccc0025bb433a92d7b620a23: size = 258 OK
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffd5ea0884d7f14d6d0aaa2426203bd91db045: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffc36354e6ccd4ccc0025bb433a92d7b620a23: Size and modification time the same (differ by 0s, within tolerance 1ns)
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffffc36354e6ccd4ccc0025bb433a92d7b620a23: Unchanged skipping
2026/06/26 07:18:36 DEBUG : 11e14966-5e8b-4ed5-9a9c-dda8583e3520/ffff4576b383a0832b55a3ca3211cad466b9b112: Unchanged skipping
2026/06/26 07:19:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 93957485
Transferred:           64 / 64, 100%
Elapsed time:     2h8m0.0s

2026/06/26 07:20:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 94854485
Transferred:           64 / 64, 100%
Elapsed time:     2h9m0.0s

2026/06/26 07:21:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 95797885
Transferred:           64 / 64, 100%
Elapsed time:    2h10m0.0s

2026/06/26 07:22:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 96762485
Transferred:           64 / 64, 100%
Elapsed time:    2h11m0.0s

2026/06/26 07:23:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 97675485
Transferred:           64 / 64, 100%
Elapsed time:    2h12m0.0s

2026/06/26 07:24:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 98569485
Transferred:           64 / 64, 100%
Elapsed time:    2h13m0.0s

2026/06/26 07:25:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 99482985
Transferred:           64 / 64, 100%
Elapsed time:    2h14m0.0s

2026/06/26 07:26:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 100361585
Transferred:           64 / 64, 100%
Elapsed time:    2h15m0.0s

2026/06/26 07:27:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 101228485
Transferred:           64 / 64, 100%
Elapsed time:    2h16m0.0s

2026/06/26 07:28:35 INFO  :
Transferred:       56.947 KiB / 56.947 KiB, 100%, 0 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:           2388356 / 2388356, 100%, Listed 102104485
Transferred:           64 / 64, 100%
Elapsed time:    2h17m0.0s
2

welcome to the forum,

sorry, not clear, is --include "/1*/**" that crazy folder or what?
to workaround that and finish the migration.

  • --exclude=/path/to/crazy/folder/** that single crazy folder
  • finish the rest of the migration
  • --include=/path/to/crazy/folder/** that single crazy folder with a debug log, see what the issue is.

that is not a debug log, just progress output.
need to use a debug log and see what rclone is doing at the time of the slowdown/stopage.
--log-level=DEBUG

which providers?

test using default values

fwiw, the odds are it is something to do with the s3 providers.
might contact them for support

as this is a one-time migration, it might be a bit of a hassle but could be a workaround.

  1. split the need-to-transfer file into smaller parts
  2. feed each part to rclone, one at a time.
3

One small thing I'd isolate before changing hardware: try one run without --no-traverse and with much lower --checkers/--transfers, just on that include prefix. With 150M listed objects, --no-traverse plus a huge checker pool can make it look idle while the destination checks are the bottleneck. If the counter still freezes, the next useful log is DEBUG around the first Errors: 1 line, not only the progress stats; that should show whether one S3 side is throttling, retrying, or returning a bad object listing.

4

Possibly the provider is looping the directory listing - we've seen that before.

Which provider is it?

Did an rclone ls remote get stuck too? Or rclone lsf -R --disable ListR remote:

If so adding -vv -dump responses then seeing if rclone gets stuck in a loop requesting the same thing. That will make a really long log but the answer should be in the last 2 transactions.

If rclone appears to be doing nothing at all in the log the try kill -QUIT rclone and post the back trace somewhere and I'll take a look.