Clarification on vfs-cache modes / O_APPEND O_TRUNC

My understanding of the limitations of the VFS cache modes is a bit spotty.
I am looking to understand better what the practical implications are.

For example if doing a simple bulk-move to mount with no cache, rclone fills up with truncate errors:

2019/10/29 01:41:53 ERROR : checksums.sha256: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes

yet the files seem to make it through safe and whole anyway (hash verified). So is rclone complaining about nothing in that case? Why is it trying to truncate in this case, and what exactly is the consequence of it not being able to do so?

  • Could anything go horribly wrong and mangle files? What's the worst that can happen?
  • What are some practical examples of things that would fail to work using no cache or minimal cache?
  • The lines regarding O_TRUNC and O_APPEND are the most confusing to me, so some examples involving those would be great to have.

I've pasted the relevant reference information below:

--vfs-cache-mode off

In this mode the cache will read directly from the remote and write directly to the remote without caching anything on disk.

This will mean some operations are not possible

Files can’t be opened for both read AND write
Files opened for write can’t be seeked
Existing files opened for write must have O_TRUNC set
Files open for read with O_TRUNC will be opened write only
Files open for write only will behave as if O_TRUNC was supplied
Open modes O_APPEND, O_TRUNC are ignored
If an upload fails it can’t be retried

--vfs-cache-mode minimal

This is very similar to “off” except that files opened for read AND write will be buffered to disks. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

These operations are not possible

Files opened for write only can’t be seeked
Existing files opened for write must have O_TRUNC set
Files opened for write only will ignore O_APPEND, O_TRUNC
If an upload fails it can’t be retried

How are you doing the move? With what command?

If the files come through OK then that message might be a bug... Truncate is used to set a file to a given size. You can't do this on a cloud storage system - hence the need for the cache. However rclone keeps track of the files size, and if truncate() is called with the same size that the file already is, then rclone is supposed to skip it.

It would be interesting to see the log with -vv for a single move.

With truncate() the file might be the wrong size, so have extra rubbish data on the end.

Random access writes on a file, trying to set the size of a file with truncate(), trying to open the file for both read and write at the same time.

So running a database won't work, nor will tagging your mp3s.

Those are both ways of opening files. O_TRUNC means open this existing file but set its size to 0 when you do that and O_APPEND means open this existing file but only write data on the end.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.