Local cache causes rclone to see remote files that aren't really there?

Help and Support
1

Hi, I'm a long-time user of rclone. I was using it to sync a large number of photos with Google Drive in a Windows 11 virtual machine. It was working great. Then I shut the VM down without first using Ctrl-C to abort the rclone job. Rclone had transferred 600 out of about 1,830 files.

When I restarted the job today, rclone said there was nothing to transfer — apparently believing that the 1,200 or so files that it hadn't copied to Google Drive were in fact there already. I used the check feature (checking filesizes only to save tiem) and it "saw" them, miraculously, on Google Drive. I used the selfupdate feature to update rclone but had the same problem.

There must be some kind of local caching going on, anyone know if that's a feature? I was able to solve my problem by renaming folders and starting the transfer again.

Thanks

Richard

2

welcome to the forum,

when you posted, there was a basic template of questions for you to answer?

it depends, that is why we ask you for the answers...

rclone does not have a --magic flag.
the answer to your questions would be in the rclone debug log.

how did you shutdown the vm?
from inside the vm, using standard windows shutdown
or actually shutdown the vm itself?
or pause/save the vm?

3

Thanks for the reply. No template that I noticed in this category. My question is more of a "how does rclone handle caching, which is of academic interest to me" rather than "help help please fix my problem" (for which I do have a workaround, as noted in my original post.)

Updated to add: I just shutdown the host computer. The VM kept going (if that makes sense).

4

yes, total sense, that is why i asked, that is what i thought happened.

by default, hyperv will re-start the vm automatically, after the host boots.
automatic start action is set to automatically start if it was running when the service stopped

so when you re-started the host machine, hyperv resumed the vm, and then rclone finished its work.

does all that make sense?

1 Like
5

Thanks. Yes it makes sense, especially because the VM was running for a little while before I connected to it today, long enough for rclone to transfer over the rest of the files. BUT according to Google Drive's web interface, they never arrived. If this is expected behavior based on whatever caching system rclone uses then OK; if not, please let me know, so I can attempt to reproduce with logs and file a bug report.

6

ok, see, no need for --magic ;wink

i mentioned that most commands do not use cache, it can depend on the remote and command you ran?
please post the exact command you ran?

7

Sure yes here it is: (this is the one that was running when the VM was abruptly shutdown).

\rclone.exe sync -v Z:\Pictures Google_Drive:Pictures

Today I tried naming in the command the specific subfolder that had the discrepancy in the number of files, without success. Having renamed folders, I am now running this above command again, and it is working.

Thank you!

1 Like
8

welcome,

so, that command and remote will not use local cache.

really, use a debug log, then you can see what happened.
and can post the debug log in the forum.

9

Thanks! OK I will post a log in the forum if it happens again, thank you!