--delete-after and errors

Help and Support
1

Hi. I’m using rclone to backup (sync) my entire home directory to pcloud. Invariably, I get some I/O errors during the process. Things like files being updated (including the rclone log file, oops!). My understanding is that the default --delete-after will not delete any files if a single error occurs. Wouldn’t it make more sense for only files that have errors associated with them not to be deleted? In 99.9% of my cases, they can be deleted safely, it seems. Since it sounds like the behavior of --delete-after is to build a list of files that can be moved/deleted at the end of the sync, couldn’t problem files just be excluded from this list? I guess I could use --delete-during, but that doesn’t seem as safe as my proposed change.

Thanks,
Reid

2

The point of not doing the deletes if there is an error is that some errors can be catastrophic... Let's say you were copying from remote:A to otherremote:B and for some reason you got an error reading A. No files in A implies delete all the files in B - hence the check.

Note that the default is --delete-during to avoid building that massive delete list which isn't quite as save as --delete-after but it is much better than --delete-before.

Note also

  --ignore-errors                         delete even if there are I/O errors

Rclone counts any sort of error as an IO error so they aren't attached to specific files normally. I see what you are getting at though.

3

I see… If an error isn’t associated with a specific file, then my approach isn’t workable. Unfortunately, I get roughly 10 errors every time I do a sync, so I’ll have to sort them out to see if ignore-errors would be safe. Though, I suppose delete-during is probably fine in my circumstance. (The docs say that delete-after is the default?)

Thanks for the info.

4

What errors are they? Maybe you can exclude those problem files?

It used to be a long time ago... Can you point out where?

5

Yes, as I mentioned, one of them is because of the rclone log file itself. I don’t need to back those up. The others are mostly similar errors about files being updated.

This page indicates that --delete-after is the default:

https://rclone.org/docs/

I am confused about one thing, though. If --delete-during is the default, then I shouldn’t be seeing this at the end of my sync job, right?

not deleting files as there were IO errors

Or is it only referring to those files that had the IO errors?

Thanks,
Reid

6

Here’s a better link:

https://rclone.org/docs/#delete-before-during-after

7

I checked the source - that page is correct, but the help for the command line flags are wrong which state --delete-during is the default!

Sorry for the confusion - that flag has changed values over the lifetime of rclone!

I'll fix the help which is wrong to be this

      --delete-after                          When synchronizing, delete files on destination after transfering (default)
      --delete-before                         When synchronizing, delete files on destination before transfering
      --delete-during                         When synchronizing, delete files during transfer
8

Heh, no problem, I know this software stuff is hard! It’s a very impressive package, by the way.

Just one last clarification… --delete-during will cause deletes to happen even if there’s any kind of IO error? Functionally, it’d be identical to --delete-after in conjunction with --ignore-errors?

Ideally, it would be possible to detect an IO error associated with a single file so you’d know whether it’s safe to delete it, but I guess that might be a feature request at this point.

I’m not sure why anyone would want to use --delete-before, hm.

Thanks.

9

Yes that is correct (I just checked the source!). --delete-during will delete stuff regardless of the error state.

It is if you are short of space on the destination.

10

I’m finally giving it a go with --delete-during. I think that’ll work pretty well for me. Thanks for your assistance!

Reid

1 Like