Add flag to delete incomplete files on exit

Feature
1

Creating this feature request based on my experience with Google Photos as per Detecting incomplete file download from Google Photos

I think it makes sense to add a new flag along the lines of --delete-incomplete-files-on-exit which should be triggered when a SIGTERM is received. What this flag would do: delete all files from the destination that are in the process of being transferred/copied to by rclone.

This would be tremendously useful for Google Photos since there is no easy way to detect incompletely downloaded files on next rclone run. This flag would ensure that at least no incomplete files are left in place by rclone.

2

I've discovered some version of this already exists as I encountered it when running this:

timeout -v --preserve-status -k 1m 5280s rclone sync --transfers=10 --max-duration=58m --retries 1 --fast-list --gphotos-include-archived --log-level INFO --log-file=REDACTED.txt gphotos:album album

~25 minutes after --max-duration was reached (elapsed time was 1h23m), rclone gave up on a connection that disappeared and deleted the partially downloaded file.

2021/03/09 01:52:55 NOTICE: REDACTED/VID_20190828_054220.mp4: Removing partially written file on error: context deadline exceeded
2021/03/09 01:52:55 INFO  : can't close account: file already closed
github.com/rclone/rclone/fs/operations.init
	github.com/rclone/rclone/fs/operations/reopen.go:28
runtime.doInit
	runtime/proc.go:5652
runtime.doInit
	runtime/proc.go:5647
runtime.doInit
	runtime/proc.go:5647
runtime.doInit
	runtime/proc.go:5647
runtime.main
	runtime/proc.go:191
runtime.goexit
	runtime/asm_amd64.s:1374
2021/03/09 01:53:01 INFO  : 
Transferred:   	   15.177G / 15.177 GBytes, 100%, 3.275 MBytes/s, ETA 0s
Errors:                 6 (retrying may help)
Checks:             81283 / 81283, 100%
Transferred:          572 / 1440, 40%
Elapsed time:    1h23m0.0s
Transferring:
 * REDACTED 1…ID_20190825_153918.mp4:  0% /off, 0/s, -
 * REDACTED 1…ID_20190826_083814.mp4:  0% /off, 0/s, -
 * REDACTED 1…ID_20190826_165940.mp4:  0% /off, 0/s, -
 * REDACTED 1…ID_20190827_073318.mp4:  0% /off, 0/s, -
 * REDACTED 1…ID_20190829_041405.mp4: transferring

This helps my original issue if I can figure out how to speed up up the "context deadline exceeded".

But I think this doesn't address the reason I've requested this feature. I think it still makes sense to have the option that if SIGTERM is received by rclone this code should be called before exit (system could be shutting down or process could be getting evicted).

3

Rclone will delete partially downloaded files if it exits properly. Not if you just kill it though.

Rclone could add an atexit handler to do that...

I'd quite like to address this in a different way though - download to a temporary file name then rename the file to be the actual file at the end

This would have several advantages

  • it wouldn't overwrite an existing file until the new file was complete
  • partial transfers that rclone didn't clean up would be easy to spot

Rclone should probably do this for all backends capable of a partial transfer, eg local, sftp and ftp.

4

This sounds like the best solution. No need to worry about deletion.

Existence of a partial file at next run could also be used a strong signal to attempt to resume the partial transfer from where it was left interrupted.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.