Missing files on copy

After transferring files from a Swift remote to an S3-backed crypt remote using rclone copy, many files remain missing in the destination. The list of missing files appears to be random, because there’s no obvious pattern as to which files will be missing. But it’s also consistent: repeated runs of rclone copy will exit with no error messages displayed but with the same files still missing.

This issue sounds similar to Rclone copy is missing files? but the final recommendation in that thread, which is to run copy again so that missing files are transferred, is not working for me. rclone size clearly displays a much higher number of files in the source compared to the destination, before and after repeated runs of rclone copy. I’ve also tried rclone sync (even though that’s not the command I want to run) and it suffers from the same problem.

Meanwhile, rclone ls does display the files from the source remote, and it also shows that the files did not get copied to the destination.

I’d be grateful for any ideas as to what might be wrong or where to begin investigating this if this is indeed a bug.

Output of rclone -V:

rclone v1.39
- os/arch: linux/amd64
- go version: go1.9.2

The issue was present from version 1.37, and I tried upgrading to 1.39 with the same results but haven’t yet tried this with the beta. The command being run is:

rclone -v copy swiftremote:swiftcontainer s3cryptremote:

My hunch is that this issue is related to there being a large number of files in the source remote.

The output of rclone -vv is far too large to paste here but it says there are no errors. This is a sample of a copy in progress:

Transferred:   26.991 MBytes (5.545 kBytes/s)
Errors:                 0
Checks:            762716
Transferred:          249
Elapsed time:   1h23m3.9s

Thank you.

When i userclone -v copy, it’s normal. I have copy about 40000+ files. All is ok. You can double check your configerration or others. I copy files from VPS to Google Drive.

Thank you for the response.

I’ve verified that the files are missing in the target S3 bucket with the following methods:

  • Manually in the AWS control panel
  • rclone ls
  • rclone size
  • rclone copy trying to copy an individual file out of the target S3 bucket. This works for the files that did get copied and fails for the missing files, as would be expected.

So I don’t believe rclone copy is really working correctly.

For reference, my source Swift container contains over 2 million files.

Hmm, unusual!

rclone copy and rclone size both recurse the file system in essentially the same way so both should come up with the same files.

I’d be suprised if there wasn’t some sort of log telling you what is going on, but with 2million files it could be difficult to find it!

rclone -v copy swiftremote:swiftcontainer s3cryptremote:

If you’ve got enough memory, try using --fast-list which may speed things up. It will certainly use less transactions. Also --size-only or --checksum will speed up the transfer as both S3 and Swift need an extra transaction to read the modification time. Increasing --checkers and --transfers will speed things up too. So I’d use

To help you with this, I really need to see a log made with -vv --log-file rclone.log. Can you compress it and upload it somewhere? Either post the link here or email it to me at nick@craig-wood.com (put a link to this page in the email so I know what it is about!).

If you could also put up the output of rclone ls on the source and the destination that would be really helpful too.

Thank you. I tried your suggestion of --fast-list and it’s indeed about 8 times faster. But it also doesn’t transfer new files. To expand: the source bucket is continually having new files created, and repeated runs of rclone copy do transfer over these new files (while skipping the aforementioned files that never get transferred), but with --fast-list none are transferred at all.

I’ll email you with a link to an uploaded rclone log shortly.

This is pure speculation, but is it possible that this is somehow related to a combination of pagination + new files being created all the time in the source bucket? If rclone is looking at a file list page by page, it’s possible that by the time the “last page” is consulted the files in it are no longer the same as in the beginning because new files have come in and pushed them away from that page. I tried digging a bit into the rclone source to see if this is the case but I’m not so familiar with Go yet.

Quite possibly, yes.

There have been other bugs with pagination in the past!, eg this one which went into v1.37

If you want to investigate, what you could do is run

rclone -vv --dump bodies --fast-list swiftremote:swiftcontainer 2>debug.log >ls.log (or whichever is the source).

And look in the actual data returned in debug.log compared to the output of ls in ls.log.

If you know the name of a missing file check the two log files to see if it is in either - that would be a really good test.

While trying your suggestion of using --dump bodies, I just realized that rclone has been getting killed by the OOM killer all this time :man_facepalming:

This would explain why there’s no error message in its output.

Will try running this on a machine with more memory, but I imagine it will turn out to work correctly as long as it doesn’t run out.

Sorry for the invalid bug report and thank you for the help!

Errm, yes, rclone buffers the input and output in memory so if you are uploading and downloading big items it can run you out of memory. You should be OK with rclone ls though.

Note there is also --dump requests and --dump responses which means you can not dump the uploaded / downloaded files if you want.