What happens when a new upload happens during a download?

What is the problem you are having with rclone?

I'm trying to understand how atomic rclone's operations are. Suppose user A is doing an rclone sync of a folder from the cloud (e.g. a google drive folder) to a local folder. While the transfer is happening, user B does an rclone copy of a file X from his local folder to the same folder that user A is copying. What are the possible outcomes? e.g.

  • A will definitely end up with the old version of X (or nothing if it didn't previously exist)
  • A will definitely end up with the new version of X
  • A might end up with either the old or the new version of X, but definitely one or the other
  • A might get a corrupt (partly written) version of X
  • A will (or might) get an error message (but can try again successfully?)
  • B will (or might) get an error message (but can try again successfully?)
  • It depends which cloud service is being used

Could A's download fail (timeout?) or be long-delayed if B's network connection drops during the upload?

Run the command 'rclone version' and share the full output of the command.

- os/version: Microsoft Windows 11 Pro 21H2 (64 bit)
- os/kernel: 10.0.22000.1219 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

We're still making plans. Initially Google Drive. We'd like to be able to support others too.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

We're still in the planning stage trying to figure out whether rclone will work for us. Most likely user A is using rclone sync and user B is using rclone copy.

The rclone config contents with secrets removed.

I don't have a specific cloud service configured yet.

A log from the command with the -vv flag

...and so this is not possible, sorry. It would also probably not help. This situation will be rare, and probably any problems will be even rarer. I'm just trying to figure out whether they are possible, so I know whether we have to allow for them in our code that will be using rclone.

Generally there isn't any locking so if you doing things at the same time, it really depends.

If I'm uploading a file and a user is downloading it, depends on when the file gets finished, the download if it was in progress would probably fail / error out.

On Google Drive, there isn't a thing as a partial file so you won't ever get that to the best of my knowledge.

The outcome is entirely dependent on the cloud backend.

For most backends (except local, ftp, sftp, smb) you either have a file or don't. You can't have a partial file. So you might get As file or you might get Bs file depending on who finished first (but even that isn't guaranteed). You will get one complete file though.

For the other backends the file will get corrupted, rclone will notice and do a retry of the upload.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.