Is there a way to disable writeback?

Help and Support
1

Hello!

I'm considering using RClone as a backup destination and have a question about the VFS Caching layer.

I get that if I want full file operation support I need to use write caching, and it's not a problem while some file is being worked with.
However, the asynchronous nature of file writeback (i.e. - file could be uploaded long after the app that created it already closed) means that any error is not propagated out to an app, that is working with a file. This is not very convenient in my case, because even if my backup software/script reports that all is done, there might be some upload failures later. Which is a data loss, that could not be caught immediately.

Any way to make RClone initiate upload right when file is closed and don't report success back until upload is complete, similar to what happens on delete?
That way I could (at the cost of some performance) ensure data consistency, and an error could propagate out to an app, so that it could decide what to do with it (log, do its own retry, alert or whatever).
Actually, this is something that RClone kinda does with --vfs-cache-mode off - if there is an error, app gets to know that there was one. But it's not an option for me.

--vfs-write-back 0s doesn't work for me either (and might even have a bug, still trying to pin it down before reporting).

Any suggestions? If there is no way of doing this, should I make feature request on GitHub?

2

So since we have no idea what backend/version/etc you are using since you deleted the template, it's tough to answer your questions.

That's generally what happens but not sure what flags you are using which can modify that behavior.

Depends on what you are running and what your application is doing and how it's writing. All in the debug logs...

felix@gemini:~$ rclone mount DB: /home/felix/test -v

and

felix@gemini:/data$ time cp jellyfish-30-mbps-hd-h264.mkv /home/felix/test

real	0m12.460s
user	0m0.007s
sys	0m0.083s

Didn't return until it was uploaded.

Share the info in the template as we will know what version and command you are using.

3

Sorry for ignoring template - I thought that since I have not a problem, but general configuration question you won't need any logs. :sweat:

I'm using 1.57.0 Windows version of RClone.

rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19043.1348 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Backend that I'm planning to use is yandex, but question is not tied to any specific backend, all further commands are demonstrating my point using memory backend, limited to 10M/s.

Command that I'm using right now for testing:
./rclone mount :memory::/ Y:\ -vv --vfs-cache-mode=full --bwlimit=10M --cache-db-purge

This way ~1 GiB file copy exits after 1-2 second, way too earlier than you would expect with a 10M limit, suggesting that there is delayed cache writeback.

(sorry, I'm on Windows, no time command available, so here is PowerShell way of measuring a copy):

PS C:\Users\igord\Downloads> Measure-Command { cp .\StackOverflow2010.7z Y:\ }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 294
Ticks             : 12941532
TotalDays         : 1,4978625E-05
TotalHours        : 0,000359487
TotalMinutes      : 0,02156922
TotalSeconds      : 1,2941532
TotalMilliseconds : 1294,1532

This is something that I would prefer to disable. Read caching is fine, write caching is also fine for the purpose of seeking a read+write files. But closing a file should end only after it was committed to remote, at least in my use-case.

Since some software I'm planning to use does open files to read+write and does seeking - using default --vfs-cache-mode=off is not an option for me, I need writes level or higher. But it looks like any other than off (including minimum) uses writeback cache with a delay.

So my question is - can this delay be disabled on writes+ levels so that copying file (or closing it in software) initiated upload and didn't return until it ended?
Example that you showed me uses default off level, isn't it?

Minified log: RClone log (minified) · GitHub
Same full log: RClone log (full) · GitHub

4

That's a flag for the cache backend and does nothing.

With full mode, you get caching as that's what it is designed for.

Do you have a log that shows that? You can read/seek without any cache mode.

Depending on how your app writes, you may or may not need a cache mode.

No that's why the cache mode exists. There isn't a flag to turn it off.

5

It writes, than resets pointer to the beginning, and tries to read.

Can't read and write to file without --vfs-cache-mode >= minimal

Which is expected, as per doc.

Okay, I hoped there were a way to commit cached changes on close.

Thank you for your help! :smiley:

6

If you have control of the app, can you adjust that so it just does a normal write?

7

I could do that for my own software - yes. However, borg, that is supposed to access same mount does require caching, if I get that right.
I'm not comfortable letting borg think that all is good and commited when in reality it's somewhere in a cache, waiting for upload (that could never come).

8

Yeah, you didn't mention Borg until now as I've familiar with that issue already.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.