Why does a file transfer continue when there is no available storage?

[RyanH]

The screenshot below indicates that there is no longer any available storage.

However when attempting to upload a file rather displaying a message that no space is available, rclone appears to continue transferring the file.

[ncw (Nick Craig-Wood)]

What is the destination? Rclone relies on the destination telling it that it is full to know when to stop uploading.

[RyanH]

The remote in this example is DropBox.

Is there the option to first calculate the total size of the transfer followed by validating the available storage? If it exceeds, rclone spits out a warning and quits.

Assuming, no and rclone commences the push, how will it fail? For example, if a 10GB file is being copied and there is only 2GB of remote storage.

[ncw (Nick Craig-Wood)]

Is there the option to first calculate the total size of the transfer followed by validating the available storage? If it exceeds, rclone spits out a warning and quits.

Not at the moment, no, though dropbox does have a free space call. Note that those calls can lag behind the actual space...

Assuming, no and rclone commences the push, how will it fail? For example, if a 10GB file is being copied and there is only 2GB of remote storage.

I'm not sure with dropbox. It might die at 2GB or it might die at 10GB or it might die right at the start when rclone annouces it is starting a 10GB upload. You'll have to experiment.

[RyanH]

I'm not sure with dropbox. It might die at 2GB or it might die at 10GB or it might die right at the start when rclone annouces it is starting a 10GB upload. You'll have to experiment.

It does neither. It continues to upload until the process is interrupted. Is there the option to output the results of failed transfers when there is insufficient storage?

[ncw (Nick Craig-Wood)]

If you run with -vv --dump responses you'll see the HTTP responses that dropbox is returning - can you post them here?

[RyanH]

Here are the screenshots.

Attached are the responses. The test was run with an attempt to upload 3 ISO images. It was subsequently interrupted once it surpassed the available storage.

[ncw (Nick Craig-Wood)]

Thanks for that, very useful..

What is happening is that it gets this error

73
{"error_summary": "path/insufficient_space/...", "error": {".tag": "path", "path": {".tag": "insufficient_space"}}}
0

Then rclone does a low-level retry to try to do the upload again, so rclone essentially tries 10 times.

What I can do is make the insufficient_space a non retriable error which should stop this.

Can you give this a go?

https://beta.rclone.org/branch/v1.50.2-116-g9dc5942f-fix-dropbox-full-beta/ (uploaded in 15-30 mins)

[RyanH]

Then rclone does a low-level retry to try to do the upload again, so rclone essentially tries 10 times.

What I can do is make the insufficient_space a non retriable error which should stop this.

Can you give this a go?

Thanks @ncw. Is it possible that rather than simply end the process that it (1) retries with an explicit message that there is insufficient storage x times (2) it returns a list of files that failed to copy or synchronize.

[ncw (Nick Craig-Wood)]

(1) retries with an explicit message that there is insufficient storage x times

I could make it so the error is retried in high level retries (3 by default controlled by --retries flag) but I'm not sure that is particularly helpful.

(2) it returns a list of files that failed to copy or synchronize.

Rclone will give ERROR messages for files which failed to copy and you can use rclone check to get a list of them.

[RyanH]

@ncw, in my personal opinion a very useful feature would be to perform a check before commencing a transfers. For example, if the size of the source exceeds the available space of the remote, a warning is generated. A choice is given as whether rclone should progress and if yes it fails with as with ERROR messages which can be piped with rclone check.

[calisro (Rob)]

The issue with that is that rclone doesn't know upfront what the size would be. It calculates that on the fly and limits how far it descends to conserve memory.

[RyanH]

It could however I imagine perform a preliminary check similar to rclone about remote:. Secondly, I assume that it can continue to perform a similar check each time a transfer is successfully completed.

[calisro (Rob)]

Yes I'm sure it could but it would need to do more than just an 'about'. It would need to determine what would need to be changed which would take significantly longer to start the sync and it would have to recurse the entire tree on both source and dest to calculate that space checking all files. That would be a pretty big overhead just to detect a situation that wouldn't occur very frequently. If the source and destinations were very large that would be even a larger impact. Just my two cents but i'm sure @ncw will chime in.

EDIT: Honestly in my opinion, it would be better to just write a job that watches your space usage on your destination and alerts you if you get 'close' (subjective % free).

[RyanH]

I am suggesting it start with a basic check on both source and destination. For instance, it perform a similar validation as du -sch /source followed by rclone about remote: .If the collective source size exceeds that of the remote, it presents a warning before commencing.