When copying to a CIFS mounted drive, if the mounted drive runs out of disk space in the middle of a copy, the copy will error out (expected).
However, after that, the destination file will not be able to be accessed through rclone or through the filesystem (cannot touch, rm, etc. the file). Attempting to access the file returns "no such file or directory". The only way to interact with the file is to remount the mounted drive.
Previously added a similar forum post found here: Input/Output Error Copying to CIFS mount
Using the environment variable suggested by ncw (GODEBUG=asyncpreemptoff=1) fixed the occurrence of the original issue as far as we can tell (copied thousands of files since then).
However, it appears that the environment variable does not resolve the issue when the mounted drive runs out of space.
Copy of relevant logs are below. The "close ... input/output error" occurs on the original error (drive filling up during the copy). The "no such file or directory" occurs when rclone attempts to access the logs again.
2021/02/07 10:49:06 NOTICE: Serving remote control on http://127.0.0.1:5815/
2021/02/07 10:49:08 NOTICE: Config file "/home/redacted/.config/rclone/rclone.conf" not found - using defaults
2021/02/07 10:49:38 NOTICE: __600-mb.tmp.txt: Removing partially written file on error: close /mnt/odin/layer-1/layer-2/TestFiles/__600-mb.tmp.txt: input/output error
2021/02/07 10:49:39 ERROR : 600-mb.txt: Failed to copy: close /mnt/odin/layer-1/layer-2/TestFiles/__600-mb.tmp.txt: input/output error
2021/02/07 10:50:13 ERROR : 600-mb.txt: Failed to copy: open /mnt/odin/layer-1/layer-2/TestFiles/__600-mb.tmp.txt: no such file or directory
That code was never actually written, it was only an idea that rclone could be doing something different here.
I had a go at implementing it here - this will abort the transfer before copying anything if preallocate returns a disk full error. Let me know if it fixes your problem!
Hello @ncw, we cherry-picked your changes into our copy of rclone (to ensure only one change from our current version) and still ran into the issue. Our setup is a little odd so let me know if there are questions on the logs.
To follow up on one thing: we have found that the issue generally occurs if two files start copying at the same time. For example, if there is 1GB free but we send copy an 800MB file and a 500MB file, we run into the error. If there is only a single file that is too large (for example, copying a 1.5GB file to a 1GB drive), Rclone provides a "no space available" response before attempting the copy.
I pushed another commit onto the branch which puts a mutex so rclone doesn't call preallocate simultaneously. This seemed to help in my tests. It still wasn't perfect and the fact that it is needed at all probably indicates concurrency problems with the accounting in the file system. However it did make an improvement with my tests.