How to detect and handle failed uploads when using rclone mount with VFS cache

[docwm (Stefano)]

What is the problem you are having with rclone?

I’m working on a Go client that integrates with rclone (mount) + WinFsp to mount WebDAV volumes.

My main concern: what happens when an upload fails (network issue, permission error, quota exceeded, etc.)?

• Does rclone keep the file in the --cache-dir until it can retry?

• Can I rely only on rclone’s stdout/stderr logs (I’m already parsing JSON logs) to know if a file was successfully uploaded or if it’s still “dirty”?

• Are there recommended best practices for monitoring pending/failed uploads?

• Is checking the cache-dir directly enough, or is there a more “official” way (maybe via rc commands like vfs/status)?

The idea is that my client should be able to notify the user:
“File is still pending in cache / failed to upload”

Because if the user unplugs my go client that runs the rclone process, it "loses files" in the remote webdav server.

What’s the best approach here to ensure users don’t lose data if rclone can’t push the file upstream immediately?

Thanks in advance!

Run the command 'rclone version' and share the full output of the command.

v1.71.0

Which cloud storage system are you using? (eg Google Drive)

Webdav

The command you were trying to run (eg rclone copy /tmp remote:tmp)`

```
RCLONE MOUNT of:		
args := []string{
			"mount",
			":webdav:",
			"--webdav-url", link,
			"--webdav-user", username,
			"--webdav-pass", obscuredPassword,
			driveLetter,
			"--vfs-cache-mode", "full",
			"--volname", fmt.Sprintf("\"%s\"", volumeName),
			"--no-console",
			"--use-cookies",
			"--use-json-log", //Used to better handle logs
			"--log-level", "INFO",
			"--vfs-write-back", "0",
			"--dir-cache-time", "5s",
			"--poll-interval", "10s",
			"--cache-dir", "tmp",
		}
```

[ncw (Nick Craig-Wood)]

@docwm

Does rclone keep the file in the --cache-dir until it can retry?

Yes it does (with --vfs-cache-mode writes or full). Without the upload is synchronous so the uploading application will get an error.

This will persist over restarts.

@docwm

Can I rely only on rclone’s stdout/stderr logs (I’m already parsing JSON logs) to know if a file was successfully uploaded or if it’s still “dirty”?

You will get an INFO log when the file is uploaded

No log if it is still dirty.

@docwm

Are there recommended best practices for monitoring pending/failed uploads?

You can do this via the RC

Remote Control / API

There is no such thing as a failed upload, rclone will keep trying at doubling intervals

@docwm

Is checking the cache-dir directly enough, or is there a more “official” way (maybe via rc commands like vfs/status)?

I recommend not doing that. Implementation may change.

@docwm

What’s the best approach here to ensure users don’t lose data if rclone can’t push the file upstream immediately?

I would like at some point when you unmount the drive rclone makes sure it is synced.

We could probably do an RC call for sync too.

[docwm (Stefano)]

Thanks a lot Nick, that clears up many things!

I just wanted to ask for a bit more clarification on this part:

@ncw

I would like at some point when you unmount the drive rclone makes sure it is synced.

We could probably do an RC call for sync too.

• Do you mean this is something planned but not yet implemented?

• Right now, is there any way to “force flush” all pending dirty files before unmounting (maybe with an existing rc command I’ve missed)?

• If not, would the idea be to introduce something like rc vfs/sync to explicitly trigger the upload of everything in the cache?

This would be super useful for my case, since I’d like to ensure that when the user disconnects the drive, all files are guaranteed to be uploaded and no data remains pending.

Thank you again for this amazing work.

[ncw (Nick Craig-Wood)]

@docwm

Do you mean this is something planned but not yet implemented?

Well there is an issue about it (too lazy to search for it!)

@docwm

Right now, is there any way to “force flush” all pending dirty files before unmounting (maybe with an existing rc command I’ve missed)?

No. Rclone Will flush them as fast as it can anyway, but sticking to --transfers

@docwm

If not, would the idea be to introduce something like rc vfs/sync to explicitly trigger the upload of everything in the cache?

That is pretty much what the issue is about. It's not quite as simple as you would hope though...

Rclone would have to do this for vfs/sync

• pause all writers
• do the upload
• unpause writers

Or alternatively if unmounting

• wait for all writers to finish (or pause them)
• set mount read only (or unmount it)
• sync dirty files
• exit

The OS gets upset if you unmount with files open.

Other file systems do this little dance too!