Previously I have used rclone sync to successfully sync the files from here https://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/ currently I have the issue that when running rclone sync based on the logs everything is working and rclone ls is showing the new files.
On the other hand when looking at gsutil ls the newer files aren't showing up. I was wondering what can I do to debug this issue?
Is one using the fast-list functionality while the other is not? rclone's sync can do either. gsutil might use this by default? (I haven't checked).
I think I've read that when fast-list is used some of that data is fetched from cache in Googles backend and it may sometimes be a couple of minutes out of date and miss very recently added files when listing. I see this happening sometimes if I sync some files to a Gdrive and then immediately --fast-list is back. It's not unusual for a handful of files or a recently folder or two missing in that list.
This shouldn't be a problem if not using --fast-list , but for very large collections of files in complex hierarchies that will also be way slower when listing the whole drive. Assuming that this is your issue the files should be there and they will show up on their own in a bit, so it may actually not be a problem depending on your needs.
If you have the drive mounted then it will be able to using polling to keep updated and this should reduce the issue a lot. The polling interval can also be lowered if needed.
Do the newer files show up eventually? All cloud storage systems are "eventually consistent" some more eventually than others!
As far as I can tell no the files never show up.
I'd guess that rclone and gsutil aren't looking in the same place. Are you using the same account / service account with both?
It sounds like it's probably something fairly simple like what NCW suggests. If rclone transfers the files they are somewhere. They got verified on the upload so they can't just go poof into the ether.
It would probably help if you could share the full commands you are using on both programs so we can compare.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.