After transferring files from a Swift remote to an S3-backed crypt remote using rclone copy, many files remain missing in the destination. The list of missing files appears to be random, because there’s no obvious pattern as to which files will be missing. But it’s also consistent: repeated runs of rclone copy will exit with no error messages displayed but with the same files still missing.
This issue sounds similar to Rclone copy is missing files? but the final recommendation in that thread, which is to run copy again so that missing files are transferred, is not working for me. rclone size clearly displays a much higher number of files in the source compared to the destination, before and after repeated runs of rclone copy. I’ve also tried rclone sync (even though that’s not the command I want to run) and it suffers from the same problem.
Meanwhile, rclone ls does display the files from the source remote, and it also shows that the files did not get copied to the destination.
I’d be grateful for any ideas as to what might be wrong or where to begin investigating this if this is indeed a bug.
Output of rclone -V:
rclone v1.39
- os/arch: linux/amd64
- go version: go1.9.2
The issue was present from version 1.37, and I tried upgrading to 1.39 with the same results but haven’t yet tried this with the beta. The command being run is:
When i userclone -v copy, it’s normal. I have copy about 40000+ files. All is ok. You can double check your configerration or others. I copy files from VPS to Google Drive.
I’ve verified that the files are missing in the target S3 bucket with the following methods:
Manually in the AWS control panel
rclone ls
rclone size
rclone copy trying to copy an individual file out of the target S3 bucket. This works for the files that did get copied and fails for the missing files, as would be expected.
So I don’t believe rclone copy is really working correctly.
For reference, my source Swift container contains over 2 million files.
If you’ve got enough memory, try using --fast-list which may speed things up. It will certainly use less transactions. Also --size-only or --checksum will speed up the transfer as both S3 and Swift need an extra transaction to read the modification time. Increasing --checkers and --transfers will speed things up too. So I’d use
To help you with this, I really need to see a log made with -vv --log-file rclone.log. Can you compress it and upload it somewhere? Either post the link here or email it to me at nick@craig-wood.com (put a link to this page in the email so I know what it is about!).
If you could also put up the output of rclone ls on the source and the destination that would be really helpful too.
Thank you. I tried your suggestion of --fast-list and it’s indeed about 8 times faster. But it also doesn’t transfer new files. To expand: the source bucket is continually having new files created, and repeated runs of rclone copy do transfer over these new files (while skipping the aforementioned files that never get transferred), but with --fast-list none are transferred at all.
I’ll email you with a link to an uploaded rclone log shortly.
This is pure speculation, but is it possible that this is somehow related to a combination of pagination + new files being created all the time in the source bucket? If rclone is looking at a file list page by page, it’s possible that by the time the “last page” is consulted the files in it are no longer the same as in the beginning because new files have come in and pushed them away from that page. I tried digging a bit into the rclone source to see if this is the case but I’m not so familiar with Go yet.
Errm, yes, rclone buffers the input and output in memory so if you are uploading and downloading big items it can run you out of memory. You should be OK with rclone ls though.
Note there is also --dump requests and --dump responses which means you can not dump the uploaded / downloaded files if you want.
