Input/Output Error Copying to CIFS mount - Disk Space Filled

What is your rclone version (output from rclone version)

1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 7

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Start Rclone using the following command

rcd --log-level DEBUG --rc-no-auth --rc-addr=127.0.0.1:5816 --log-file="REDACTED" --copy-links --local-no-set-modtime

After that, perform simple copy commands using RC: https://rclone.org/rc/#operations-copyfile

The rclone config contents with secrets removed.

None.

Description of Issue

When copying to a CIFS mounted drive, if the mounted drive runs out of disk space in the middle of a copy, the copy will error out (expected).
However, after that, the destination file will not be able to be accessed through rclone or through the filesystem (cannot touch, rm, etc. the file). Attempting to access the file returns "no such file or directory". The only way to interact with the file is to remount the mounted drive.

Previously added a similar forum post found here: Input/Output Error Copying to CIFS mount
Using the environment variable suggested by ncw (GODEBUG=asyncpreemptoff=1) fixed the occurrence of the original issue as far as we can tell (copied thousands of files since then).
However, it appears that the environment variable does not resolve the issue when the mounted drive runs out of space.

Copy of relevant logs are below. The "close ... input/output error" occurs on the original error (drive filling up during the copy). The "no such file or directory" occurs when rclone attempts to access the logs again.

2021/02/07 10:49:06 NOTICE: Serving remote control on http://127.0.0.1:5815/
2021/02/07 10:49:08 NOTICE: Config file "/home/redacted/.config/rclone/rclone.conf" not found - using defaults
2021/02/07 10:49:38 NOTICE: __600-mb.tmp.txt: Removing partially written file on error: close /mnt/odin/layer-1/layer-2/TestFiles/__600-mb.tmp.txt: input/output error
2021/02/07 10:49:39 ERROR : 600-mb.txt: Failed to copy: close /mnt/odin/layer-1/layer-2/TestFiles/__600-mb.tmp.txt: input/output error
2021/02/07 10:50:13 ERROR : 600-mb.txt: Failed to copy: open /mnt/odin/layer-1/layer-2/TestFiles/__600-mb.tmp.txt: no such file or directory

This sounds like a CIFS bug to me. Nothing rclone can do should ever cause CIFS to lose track of its files like this.

Can you replicate the problem without rclone? Maybe with cp?

We were unable to replicate the issue with "cp" command. The "cp" command does not complete (expected) but it does not end up with the "zombie" file.

We were able to replicate the issue using .Net File.Move command

We are still investigating the issue on our end, currently looking into how the preallocate function fits into how disk full errors are being handled.

We noticed that the preallocate used to check for ENOSPC error but any handling was commented out: https://github.com/rclone/rclone/blob/master/lib/file/preallocate_unix.go#L47

OK

Interesting...

That code was never actually written, it was only an idea that rclone could be doing something different here.

I had a go at implementing it here - this will abort the transfer before copying anything if preallocate returns a disk full error. Let me know if it fixes your problem!

v1.55.0-beta.5166.639abfb52.fix-preallocate-disk-full on branch fix-preallocate-disk-full (uploaded in 15-30 mins)

Thanks for the update, ncw. We will try out the updated build and report back here.

1 Like

Hello @ncw, we cherry-picked your changes into our copy of rclone (to ensure only one change from our current version) and still ran into the issue. Our setup is a little odd so let me know if there are questions on the logs.

To follow up on one thing: we have found that the issue generally occurs if two files start copying at the same time. For example, if there is 1GB free but we send copy an 800MB file and a 500MB file, we run into the error. If there is only a single file that is too large (for example, copying a 1.5GB file to a 1GB drive), Rclone provides a "no space available" response before attempting the copy.

output.log (87.7 KB)

I pushed another commit onto the branch which puts a mutex so rclone doesn't call preallocate simultaneously. This seemed to help in my tests. It still wasn't perfect and the fact that it is needed at all probably indicates concurrency problems with the accounting in the file system. However it did make an improvement with my tests.

v1.55.0-beta.5167.8c6161fb2.fix-preallocate-disk-full on branch fix-preallocate-disk-full (uploaded in 15-30 mins)

Hello @ncw, it does not appear that the latest update resolved the issue.

As you've mentioned (similar to the other CIFS problem we reported), the issue likely resides with CIFS rather than Rclone itself.

output.log (80.6 KB)

I don't see any preallocate: file too big for remaining disk space errors in your log - does preallocate work on CIFS at all?

I'm gonig to merge those fixes as I think they are useful anyway.

That appears to be the issue - attempting fallocate from the command prompt returns "fallocate: fallocate failed: Operation not supported"

Ah. That is not going to work then!

I fixed another bug in the preallocate code but you probably aren't interested in that now since CIFS doesn't support preallocate.

v1.55.0-beta.5215.ffda2aede.fix-preallocate-disk-full on branch fix-preallocate-disk-full (uploaded in 15-30 mins)

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.