Is it possible to tell rclone to always sync but retry only failed? (--ignore-times but retry only failed)

Help and Support
1

What is the problem you are having with rclone?

This is trying to fix an edge case on syncrclone. Though this question is more related to general rclone usage.

The thing is, my tool decides what to transfer based on its own logic and then writes it out for --files-from. I then want to always push those files. No need for rclone do to anything but copy. And I want it to copy unconditionally!

Using --ignore-times gives me what I want but if a copy fails, it will recopy everything! (right? I know that is the case for --no-check-dest. It doesn't say in the docs for but I think it's a safe assumption).

I don't want that. I just want rclone to retry failed copies.

So I can tell it to just copy without the flag but then it has to do its own determination of whether or not to copy. This is both wasteful and possible to get a false negative. (e.g. I decide to copy files based on hashes. Then if there are two files with the same size and modtime but different hashes, rclone will either not copy it (default) or re-hash it (--checksum).)

Another solution is to do one rclone instance per copy but I've found some race conditions on this. And I think it is less efficient.

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.0
- os/version: darwin 12.6 (64 bit)
- os/kernel: 21.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.19.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Any to Any. For now, local to local but really not remote specific

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Something like

rclone sync source: dest: --ignore-times --files-from files.txt

but where it will not retry all files upon a single failure. Just the

The rclone config contents with secrets removed.

N/A -- Local to local for now.

A log from the command with the -vv flag

N/A -- Theoretical. I am not even sure how to make it cause an error. I am trying to think ahead.

Thanks!

2

I don't think you can a) make rclone ignore all checks in the first pass then b) use them in the second pass.

You could use --checksum which won't copy the file if the size and checksum are identical, but in that case you can be pretty sure the files are identical anyway. It will then set the dest timestamp to be the same as the source.

A retry only failed files would be a useful addition to rclone to save rclone grinding though the entire sync again but there are ways that syncs can fail that you need that.

3

Ok. Thanks.

Am I correct that doing many single-file transfer calls can cause some race conditions? Any ways to avoid it or reduce it?

I could probably track the status with logs myself to do the logic. Is there a way to test this and cause failures?

4

It shouldnt do, no, but it is inefficient.

If you want to do single transfers more efficiently then run an rclone rcd and use rclone rc to start the transfers.

Or use librclone directly.

I think you'd be better off working out how to get rclone to do this.

Perhaps it needs a new flag.

5

Perhaps we need a new level of retries, today we have:

  • Request level retries (--low-level-retries with progressive sleep controlled by the pacer of the backend)
  • Entire copy/sync level retries/attempts (--retries with --retries-sleep)

I guess Justin is looking for an intermediate file level retry to make it look something like this:

  • Request level retries (--low-level-retries with progressive sleep controlled by the pacer of the backend)
  • File level retries (--file-level-retries with --file-level-retries-sleep)
  • Entire copy/sync level retries/attempts (--retries with --retries-sleep)

The big question is what type of recoverable errors do we see falling into this category?

I can see temporarily locked files and files that where updated during the first copy attempt. The first typically leads to the second. Now the interesting question is how rclone, bisync and syncrclone would handle a situation where a file level retry copies a file that is different from the conditions causing the file to be copied initially. Not to mention a situation where the copy failed due to a lock or similar on the target file. Now it starts to get very complicated in my head, time for a break...

Idea: Perhaps file level retries can be implemented a bit like a copy/sync attempt with --files-from --no-traverse to just build a newly updated backlog. Which would be much like starting with --retries=1 scanning for errors and then restarting with --files-from=failedfiles.txt --no-traverse.

6

I lost you a bit there but your solution does make sense. In syncrclone, I try to catch edge cases but I can’t hit them all unfortunately. It is the risk you take using a wrapper. But I try to anticipate and think about these things, hence the question.

I did come up with a mediocre solution for syncrclone.

Split the sync into two: One when file sizes also mismatch then I can efficiently tell rclone size-only. Honestly, this will capture the vast majority in real life. Then the second is the rest when I will let it use mod time or checksum. This will be less efficient for those but I can handle that.

This all came about because I am working on a new tool for backups. The end result is nearly identical to using backup-dir but I don’t want to have to list the destination. It is too slow on many non-bucket remotes. And since it’s backup from one machine, I can be stateful. It’ll borrow a lot of code from syncrclone so it will hopefully be easy enough to write. The real challenge is finding time with a full-time job and two kids. I keep telling them the importance of backup but to a four year old, park time with dad is more important apparently!

Thanks for the help @Ole and @ncw!

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.