What is the problem you are having with rclone?
rclone mount on macOS (with
--vfs-cache-mode full) is repeatedly uploading the same files to Google Drive, despite the fact that the files have not changed. When I compare the old and new versions of the files, the only difference I can find is the modtime; the checksums and all other metadata match exactly.
This happens only with a few specific files (and not many others in the same directory). There is nothing special about the problematic files, so far as I can tell. I have checked for potential duplicates and possible illegal characters in the filenames, but there are none that I can find. It happens a few times per day, which I can see by reviewing the Activity History in Google Drive (via the https://drive.google.com/ UI). It is a bit unsettling to see random files being "edited" randomly when I haven't touched them, even though no data seems to be changing.
I've tried debugging a bit and have found a sequence of steps to reliably reproduce it (see below.) Note that this is just one way to produce it, and not the ONLY way (for example, I have also seen it happen while the mount is running and has not been recently mounted or unmounted.)
One theory I had (which could be totally wrong) is that some other process on my computer (perhaps system processes related to Finder or Quick Look) is accessing the files with read/write access but proceeds to only read (and not write) them, and rclone interprets this as a write, and accordingly re-uploads them with an updated modtime. But if that's the case, I can't explain why it would only affect a few specific files and not others. (Presumably the same process would be reading those files too?)
Also worth noting that I do not see this behavior in a normal (non-mounted) local folder with the same files in it. In other words: it seems to be rclone (or FUSE?), not Finder or something else, that is responsible for updating the modtime. It is not happening when we take rclone/FUSE out of the equation.
Repro steps (and corresponding lsl output):
To make the simplest possible test, I created a new folder in Google Drive containing only 3 files: the 2 "problematic" files (
PXL_20220511_194921851.MP.jpg) and 1 normal file (
PXL_20220717_044546991.MP.jpg). I will set this folder as the
root_folder_idfor my drive remote.
Start the mount. At this point, if I run
rclone lslon my cache-dir, I see as follows (only the
.DS_Storethat Finder puts there by default. So far so good.)
6148 2023-02-21 05:36:12.476326737 .DS_Store
- Open the
/path/to/local/mountdirectory in Finder. Now,
rclone lslreveals that the 2 problematic files (but not the 1 normal file) have been downloaded.
6148 2023-02-21 05:36:12.476326737 .DS_Store 276 2023-02-21 06:01:59.936642041 vfsMeta/gdrive_testfolder/PICT_20211210_130955.JPG 278 2023-02-21 06:02:01.980363484 vfsMeta/gdrive_testfolder/PXL_20220511_194921851.MP.jpg 529165 2023-02-21 05:36:35.938000000 vfs/gdrive_testfolder/PICT_20211210_130955.JPG 6455244 2023-02-21 05:36:34.259000000 vfs/gdrive_testfolder/PXL_20220511_194921851.MP.jpg
- Wait for the cache to expire, so that rclone automatically removes the cached files. (I've set
0h1m0sto make testing easier.) Now,
rclone lslconfirms the cache is cleaned:
6148 2023-02-21 05:36:12.476326737 .DS_Store
- Unmount the mount. Immediately upon doing so (and before the terminal process fully terminates), rclone re-downloads the 2 problematic files (but not the 1 normal file) and updates their modtimes, as if they had been edited. Note the changed modtimes and the absence of the third file:
6148 2023-02-21 05:36:12.476326737 .DS_Store 275 2023-02-21 06:04:03.952351995 vfsMeta/gdrive_testfolder/PICT_20211210_130955.JPG 278 2023-02-21 06:04:03.633825357 vfsMeta/gdrive_testfolder/PXL_20220511_194921851.MP.jpg 529165 2023-02-21 06:04:03.634066000 vfs/gdrive_testfolder/PICT_20211210_130955.JPG 6455244 2023-02-21 06:04:01.935404000 vfs/gdrive_testfolder/PXL_20220511_194921851.MP.jpg
- Start the mount again. Now, rclone will replace the remote files with the cache files, because they are newer.
6148 2023-02-21 05:36:12.476326737 .DS_Store 276 2023-02-21 06:05:48.024043094 vfsMeta/gdrive_testfolder/PICT_20211210_130955.JPG 279 2023-02-21 06:05:49.105709685 vfsMeta/gdrive_testfolder/PXL_20220511_194921851.MP.jpg 529165 2023-02-21 06:04:03.634066000 vfs/gdrive_testfolder/PICT_20211210_130955.JPG 6455244 2023-02-21 06:04:01.935404000 vfs/gdrive_testfolder/PXL_20220511_194921851.MP.jpg
- For completeness's sake, here's what I get when I run
rclone lslon the Google Drive remote:
529165 2023-02-21 06:04:03.634000000 PICT_20211210_130955.JPG 6455244 2023-02-21 06:04:01.935000000 PXL_20220511_194921851.MP.jpg 5566678 2022-07-17 00:55:06.000000000 PXL_20220717_044546991.MP.jpg
(The absence of
.DS_Store is expected due to
--noappledouble and I believe the rounded modtime is also expected per https://rclone.org/drive/#modified-time )
Run the command 'rclone version' and share the full output of the command.
rclone v1.61.1 - os/version: darwin 13.2.1 (64 bit) - os/kernel: 22.3.0 (arm64) - os/type: darwin - os/arch: arm64 - go/version: go1.19.4 - go/linking: dynamic - go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
rclone mount gdrive_testfolder: /Users/redacted/rclone/gdrive_testfolder --vfs-cache-mode full --stats 1m --rc -vv --cache-dir /Users/redacted/rclone/cache/gdrive_testfolder --vfs-cache-max-age 0h1m0s
The rclone config contents with secrets removed.
[gdrive_testfolder] type = drive client_id = redacted client_secret = redacted scope = drive token = redacted team_drive = root_folder_id = redacted export_formats = url
A log from the command with the
Log corresponding to Steps 1-5 above: rclone log 2023-02-21 Steps 1-5 - Pastebin.com
Log corresponding to Step 6 above: rclone log 2023-02-21 Step 6 - Pastebin.com