Temporary file support on copy/sync

What is the problem you are having with rclone?

Is it possible that a failed rclone copy results in a 0 byte output file?

Referring to https://github.com/cjnaz/rclonesync-V2/issues/56, there appear to be cases where if the transfer is terminated during a file copy that a 0 byte (or perhaps incomplete) file gets created at the destination, which is probably data loss for a bi-directional sync.

The transfer may fail for several reasons, including.. the internet connection drops, the client machine is hibernated, crashes or is rebooted.

Questions...

  1. When a copy (or a sync) is in process, is a temporary file created at the destination, and renamed/moved to the final output name at the end of the successful transfer?
  2. If so, are all remotes as destinations supported for this behavior?
  3. Is a transfer to a local filesystem supported for this behavior?
  4. Are there any rclone switches related to this behavior?
  5. If this behavior is not supported, what are the blocking reasons?

What is your rclone version (output from rclone version)

rclone v1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Centos 7.8 64b and Windows 10 64b

Which cloud storage system are you using? (eg Google Drive)

Google Drive, Local, Owncloud, ...

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy GDrive:fileXYZ <localdrivepath>

or 

rclone copy <localdrivepath> GDrive:fileXYZ

The rclone config contents with secrets removed.

na

A log from the command with the -vv flag

na

There's a feature request out there for resume uploads:

Not on any of the cloud platforms.

However on local, sftp and ftp I think this probably is possible.

Rclone tries to tidy up failed copies by deleting partially transferred files. This doesn't always work if the user quits rclone before the end.

Normally this doesn't matter as rclone will just see the file is wrong the next time the transfer is re-run and retry however it looks like it does matter for your application.

From the issue it looks like it was a multipart copy that failed to the local disk.

Hmm when copying to local disk, ftp, etc then yes the destination file is overwritten at the start of the copy so if the copy fails the destination file will be corrupted.

Rsync has a mode where it transfers new files with a name like file.txt.xxxrandomstringxxx and renames it into place at the end.

This probably needs to be implemented on the local backend, so files are transferred with a temporary name (if there is an existing file of that name) then renamed into place at the end.

This could be a local backend option or it could be a global option for rclone which is implemented by the core copy routine.

I don't know whether it should be the default.

For now I'll look into implementing temp file naming on local/sftp/ftp on Path1. I use an rclone sync Path1 Path2 as the final step, so would need rclone built in support if a local/sftp/ftp target is on Path2. I'll put in a feature request on github.

Great! If you could put your use case as in why you need the feature. I think exactly what to implement needs some discussion :slight_smile:

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.