I'm trying to understand the semantics of rclone mount when it comes to shutting down the mount and making sure changes are synced. I'm experiencing problems where changes don't sync, and I'm wondering how I can tell when things are fully synced / whether I can force them to be before exiting.
For example, consider the following usage:
Start an rclone mount to mount an S3 bucket locally, with --vfs-cache-mode full
Copy a large file (like 3GB) into the mount folder. Wait a few seconds for rclone to start uploading.
Ctrl+C the mount process, and it exits silently and immediately (!)
Check my S3 bucket and find the file hasn't been written.
So it seems the mount can still be terminated when uploads are still in progress.
To try to fix this, I added my own retry logic to use the remote API, fetching the "core/stats" endpoint and checking that the "transfers" field is an empty list, and then terminating the mount. This seems to help somewhat, but if I create a file quickly and then exit the mount before rclone picks it up, I can still get missing files.
Is there a way to make this more safe? What I'd really like is for rclone to respond to a SIGINT or "mount/unmount" remote call by doing something like the following:
a) scan over the whole mounted dir to check for any new files
b) wait to exit/return until all such files have been synced (or exit/return with an error if this isn't possible)
(To be extra clear, the thing I'm concerned about is when a file "appears" to be written but is not, i.e. the local filesystem writes have completed successfully. I guess this concern only exists when VFS write caching is enabled, because otherwise the writes are being synchronously forwarded to the remote.)
Please let me know if there's something simple I'm missing!
Run the command 'rclone version' and share the full output of the command.
rclone 1.56.0
os/version: ubuntu 21.04 (64 bit)
os/kernel: 5.11.0-44-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.16.7
go/linking: dynamic
go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Sure, I can try with the latest version but the Changelog doesn't seem indicate any changes that would affect this.
I think it's working as designed as you said. Stopping the mount seems to abort writing files immediately, which isn't the behavior I'd prefer.
My use case is just to work as part of a system that uses ephemeral mounts. For example, maybe I want to mount a folder, run a script that does some processing, and then close the mount (safely). Maybe this happens in a Docker container that disappears as soon as the job is done, for example. I know this isn't how mounts are typically used, but it would be nice if it were possible.
If it will write the files when it starts back up--where does it keep that state? In the cache directory perhaps? Wondering if I can programmatically check such state before shutting down...
I tried uploading a large .iso file. I can see that it begins writing. When I interrupt it, it actually has a FUSE error trying to unmount, which leaves an errored FUSE mount on my system.
This is unfortunate because a subsequent mount then fails:
<7>DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "s3:tom-rclone-test/" "./test2" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "s3:tom-rclone-test/"
<7>DEBUG : Using config file from "/home/tom/.config/rclone/rclone.conf"
<7>DEBUG : fs cache: renaming cache item "s3:tom-rclone-test/" to be canonical "s3:tom-rclone-test"
Fatal error: Directory already mounted, use --allow-non-empty to mount anyway: /home/tom/mounts/test2
If I pass --allow-non-empty and try to start the mount process again then I get
<7>DEBUG : S3 bucket tom-rclone-test: Mounting on "./test2"
mount helper error: fusermount: failed to access mountpoint /home/tom/mounts/test2: Permission denied
Fatal error: failed to mount FUSE fs: fusermount: exit status 1
By interrupt I mean SIGINT. I just did a Ctrl+C on the rclone mount process.
I believe rclone is running fusermount under the hood.
It has to complete or you have to forcibly kill it. It's like any other Linux file system as you can't unmount something with active IO. File systems in general don't like to have the rug pulled out from under them with IO going on.
I agree, this is why I sort of think rclone should block or something until the IO is done
@Ole the fusermount docs aren't very clear about what "lazy unmount" means. I suspect it might clean up the mount eventually once it becomes non-busy, which would solve one problem. But if the rclone process is already dead then it doesn't solve the main problem of getting my files uploaded...
So if you a non service and you are running something in the foreground and you hit control C, that's your SIGINT and the process will terminate the upload and remain in the cache area until it's started again as seen here:
felix@gemini:~$ rclone mount GD: /home/felix/test --vfs-cache-mode full -vvv
2022/01/11 10:47:12 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/01/11 10:47:12 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "GD:" "/home/felix/test" "--vfs-cache-mode" "full" "-vvv"]
2022/01/11 10:47:12 DEBUG : Creating backend with remote "GD:"
2022/01/11 10:47:12 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/01/11 10:47:12 DEBUG : vfs cache: root is "/home/felix/.cache/rclone"
2022/01/11 10:47:12 DEBUG : vfs cache: data root is "/home/felix/.cache/rclone/vfs/GD"
2022/01/11 10:47:12 DEBUG : vfs cache: metadata root is "/home/felix/.cache/rclone/vfsMeta/GD"
2022/01/11 10:47:12 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfs/GD/"
2022/01/11 10:47:12 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfsMeta/GD/"
2022/01/11 10:47:12 DEBUG : jellyfish-30-mbps-hd-h264.mkv: vfs cache: truncate to size=112143943
2022/01/11 10:47:12 DEBUG : jellyfish-30-mbps-hd-h264.mkv: vfs cache: setting modification time to 2022-01-11 10:46:53.754388856 -0500 EST
2022/01/11 10:47:12 INFO : jellyfish-30-mbps-hd-h264.mkv: vfs cache: queuing for upload in 5s
2022/01/11 10:47:12 DEBUG : : Added virtual directory entry vAddFile: "jellyfish-30-mbps-hd-h264.mkv"
2022/01/11 10:47:12 DEBUG : Google drive root '': Mounting on "/home/felix/test"
2022/01/11 10:47:12 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item jellyfish-30-mbps-hd-h264.mkv not removed, freed 0 bytes
2022/01/11 10:47:12 INFO : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 106.949Mi (was 106.949Mi)
2022/01/11 10:47:12 DEBUG : : Root:
2022/01/11 10:47:12 DEBUG : : >Root: node=/, err=<nil>
2022/01/11 10:47:17 DEBUG : jellyfish-30-mbps-hd-h264.mkv: vfs cache: starting upload
2022/01/11 10:47:17 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 0 length 8388608
2022/01/11 10:47:18 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 8388608 length 8388608
2022/01/11 10:47:19 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 16777216 length 8388608
^C2022/01/11 10:47:19 INFO : Signal received: interrupt
2022/01/11 10:47:19 DEBUG : vfs cache: cleaner exiting
2022/01/11 10:47:19 ERROR : /home/felix/test: Unmounted rclone mount
2022/01/11 10:47:19 INFO : Exiting...
and if I start it back up.
felix@gemini:~$ rclone mount GD: /home/felix/test --vfs-cache-mode full -vvv
2022/01/11 10:47:55 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/01/11 10:47:55 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "GD:" "/home/felix/test" "--vfs-cache-mode" "full" "-vvv"]
2022/01/11 10:47:55 DEBUG : Creating backend with remote "GD:"
2022/01/11 10:47:55 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/01/11 10:47:55 DEBUG : vfs cache: root is "/home/felix/.cache/rclone"
2022/01/11 10:47:55 DEBUG : vfs cache: data root is "/home/felix/.cache/rclone/vfs/GD"
2022/01/11 10:47:55 DEBUG : vfs cache: metadata root is "/home/felix/.cache/rclone/vfsMeta/GD"
2022/01/11 10:47:55 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfs/GD/"
2022/01/11 10:47:55 DEBUG : Creating backend with remote "/home/felix/.cache/rclone/vfsMeta/GD/"
2022/01/11 10:47:55 DEBUG : jellyfish-30-mbps-hd-h264.mkv: vfs cache: truncate to size=112143943
2022/01/11 10:47:55 DEBUG : jellyfish-30-mbps-hd-h264.mkv: vfs cache: setting modification time to 2022-01-11 10:46:53.754388856 -0500 EST
2022/01/11 10:47:55 INFO : jellyfish-30-mbps-hd-h264.mkv: vfs cache: queuing for upload in 5s
2022/01/11 10:47:55 DEBUG : : Added virtual directory entry vAddFile: "jellyfish-30-mbps-hd-h264.mkv"
2022/01/11 10:47:55 DEBUG : Google drive root '': Mounting on "/home/felix/test"
2022/01/11 10:47:55 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item jellyfish-30-mbps-hd-h264.mkv not removed, freed 0 bytes
2022/01/11 10:47:55 INFO : vfs cache: cleaned: objects 1 (was 1) in use 1, to upload 1, uploading 0, total size 106.949Mi (was 106.949Mi)
2022/01/11 10:47:55 DEBUG : : Root:
2022/01/11 10:47:55 DEBUG : : >Root: node=/, err=<nil>
2022/01/11 10:48:00 DEBUG : jellyfish-30-mbps-hd-h264.mkv: vfs cache: starting upload
2022/01/11 10:48:00 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 0 length 8388608
2022/01/11 10:48:01 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 8388608 length 8388608
2022/01/11 10:48:01 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 16777216 length 8388608
2022/01/11 10:48:02 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 25165824 length 8388608
2022/01/11 10:48:02 DEBUG : jellyfish-30-mbps-hd-h264.mkv: Sending chunk 33554432 length 8388608
^C2022/01/11 10:48:02 INFO : Signal received: interrupt
2022/01/11 10:48:02 DEBUG : vfs cache: cleaner exiting
2022/01/11 10:48:02 ERROR : /home/felix/test: Unmounted rclone mount
2022/01/11 10:48:02 INFO : Exiting...
A clean shutdown is handled with a SIGINT as that does work without any issues and uploads would resume on the next start.
I think your case is writing a file to slow disk and you are aborting the write midstream which case it has to complete the write and then exit. That's not really a rclone thing as it's fuse and the OS complaining about having the mount removed.
This works great but you have to wait for the IO to complete before the umount is 'done' and remove any processes that might have the mountpoint locked on IO which tends to be tricky for folks.
I do it a little different and all my systemd services that require the rclone/fuse mounts have "Required" in them so if the mergerfs or rclone (both fuse mounts) go away, all the processes that use them stop as well and can only be started when the fuse mounts are back up.
Back in the day, we used to get stuck NFS servers all the time if they had IO blocked and their remote server was dead/down/not working which is somewhat similar here.
I can't figure out what your goal is here as to why the mount is being shut down and started up. You are already writing to disk locally with your options. If the mount is the problem/challenge, why not rclone copy/move directly to the remote.
Oh I think you're right, I must have failed to wait for the copy to complete in that attempt. If I open another terminal and cp a large file into the mount, and make sure the cp finishes before interrupting the mount, then I do get a clean exit when I interrupt the mount like you did.
What's the reason for the mount?
Imagine the following happening in a cronjob or cloud service.
Ephemeral docker container starts up and mounts a remote.
Container starts up a script to do some processing (this script expects to read and write files with normal filesystem operations, hence the mount)
Script finishes processing and completes its mutations to the folder.
Now I want to tear down the mount and the container, so all state disappears now and all changes are successfully persisted to the remote.
It seems like rclone is aware upon receiving the interrupt signal that it still has work left to do. I guess what I'd love to happen here is for there to be an rclone flag that says "when you get an interrupt, don't close down the mounts immediately, but instead finish syncing and then close down". This is what I expected the default behavior to be actually.
I looked in ~/.cache/rclone/vfs/[mount name], and I can see that my big file still lives there. Could a possible workaround here be to double-check that this cache directory is empty before trying to cancel the process? (Maybe I need to invalidate the read cache first to ensure nothing is cached for reading in there?)
Why not write to a local file area? Why does it have to be a mount? Just add another step to upload after this. The mount complicates the process imo and makes for a big failure point.
You don't seem like you need data from the mount for your process.
You could try that but I'm not sure I'd personally rely on that but I could be mistaken on that aspect.
I definitely do, sorry if that wasn't clear. You can imagine the processing script is doing some data analysis on the current contents of the remote, and writing some result files into the remote.
(However, it doesn't necessarily use every file in the remote, so it would be wasteful to e.g. copy the entire remote down, work on it locally, and then sync it back up.)
Hmm, that's a big more complex than so in my case, I use mergerfs to merge a local mount and my rclone mount and I just upload every hour as that works for my use case.
You want to tear things down as well so not sure that's a great fit for your scenario, but it might be.
Basically, I use mergerfs to combine a local area and a rclone mount and writes always go to the local disk based on policy and I cron an upload job every hour. My use case requires hard linking which mergerfs provides me.
Huh, thanks for the suggestion. I'm going to have to read up on mergerfs to see if I could use it somehow. I suspect what I want can't be expressed as a simple mergerfs policy, since I want rclone's VFS write cache to work normally until the time comes to tear down the mount...
I'm going to go with the working hypothesis that you can tell when rclone is "done syncing" a mount if both of the following are true:
a) The cache directory is empty, and
b) The core/stats endpoint returns empty results for the "transferring" and "checking" fields.
Not sure about the implications of read caching for part a). I'd appreciate if anyone who knows about this could let me know if I'm on the right track. In the meantime I'll do some experimentation!
Also: I have a feeling that rclone knows internally whether a mount is done syncing, and a flag could be added relatively straightforwardly to get the blocking behavior I'm after. Would love to hear from the devs if such a thing might be possible.
Hi Thomas
The next rclone release will provide enough remote control statistics to answer the "did the mount finished all uploads?" question. You can wait for it or do rclone self-update --beta and play with it right now. See explanations below:
