Clarification on vfs-cache modes / O_APPEND O_TRUNC

[thestigma]

My understanding of the limitations of the VFS cache modes is a bit spotty.
I am looking to understand better what the practical implications are.

For example if doing a simple bulk-move to mount with no cache, rclone fills up with truncate errors:

2019/10/29 01:41:53 ERROR : checksums.sha256: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes

yet the files seem to make it through safe and whole anyway (hash verified). So is rclone complaining about nothing in that case? Why is it trying to truncate in this case, and what exactly is the consequence of it not being able to do so?

• Could anything go horribly wrong and mangle files? What's the worst that can happen?
• What are some practical examples of things that would fail to work using no cache or minimal cache?
• The lines regarding O_TRUNC and O_APPEND are the most confusing to me, so some examples involving those would be great to have.

I've pasted the relevant reference information below:

---

--vfs-cache-mode off

In this mode the cache will read directly from the remote and write directly to the remote without caching anything on disk.

This will mean some operations are not possible

Files can’t be opened for both read AND write
Files opened for write can’t be seeked
Existing files opened for write must have O_TRUNC set
Files open for read with O_TRUNC will be opened write only
Files open for write only will behave as if O_TRUNC was supplied
Open modes O_APPEND, O_TRUNC are ignored
If an upload fails it can’t be retried

--vfs-cache-mode minimal

This is very similar to “off” except that files opened for read AND write will be buffered to disks. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.

These operations are not possible

Files opened for write only can’t be seeked
Existing files opened for write must have O_TRUNC set
Files opened for write only will ignore O_APPEND, O_TRUNC
If an upload fails it can’t be retried

On another topic

These are both flags to the open system call.

       O_TRUNC
              If the file already exists and is a regular file and the
              access mode allows writing (i.e., is O_RDWR or O_WRONLY) it
              will be truncated to length 0.

and

       O_APPEND
              The file is opened in append mode.  Before each write(2), the
              file offset is positioned at the end of the file, as if with
              lseek(2).  The modification of the file offset and the write
              operation are performed as a single atomic step.

[ncw (Nick Craig-Wood)]

For example if doing a simple bulk-move to mount with no cache

How are you doing the move? With what command?

rclone fills up with truncate errors:

2019/10/29 01:41:53 ERROR : checksums.sha256: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes

If the files come through OK then that message might be a bug... Truncate is used to set a file to a given size. You can't do this on a cloud storage system - hence the need for the cache. However rclone keeps track of the files size, and if truncate() is called with the same size that the file already is, then rclone is supposed to skip it.

It would be interesting to see the log with -vv for a single move.

Could anything go horribly wrong and mangle files? What's the worst that can happen?

With truncate() the file might be the wrong size, so have extra rubbish data on the end.

What are some practical examples of things that would fail to work using no cache or minimal cache?

Random access writes on a file, trying to set the size of a file with truncate(), trying to open the file for both read and write at the same time.

So running a database won't work, nor will tagging your mp3s.

• The lines regarding O_TRUNC and O_APPEND are the most confusing to me, so some examples involving those would be great to have.

Those are both ways of opening files. O_TRUNC means open this existing file but set its size to 0 when you do that and O_APPEND means open this existing file but only write data on the end.