Understand Reason for Deletion

Help and Support
1

What is the problem you are having with rclone?

Hi:. I rclone sync a data directory from nfs to AWS S3. The other day, looking at the logs, i realized that rclone decided to delete files, and i am trying to understand why. The files are in the source, so they should not be deleted.

What is your rclone version (output from rclone version)

rclone v1.44

  • os/arch: linux/amd64
  • go version: go1.11.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Redhat 6.9

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -vv /nfs/directory some-s3-bucket/ --log-file rclone-20.log &

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

This is an example for one specific file and dry run

The following is a dry run example that deletes filea.csv, even though this file is in the source. I used this example to limit the size of the logs, but this is happening with all the files.

rclone sync --dry-run -vvv --include /nfs/data/filea.csv /nfs/data/ some-s3-bucket:some-s3-bucket-backup/
2019/09/09 19:05:52 DEBUG : rclone: Version "v1.44" starting with parameters ["rclone" "sync" "--dry-run" "-vvv" "--include" "/nfs/data/filea.csv"
"/nfs/data/" "some-s3-bucket:some-s3-bucket-backup/"]
2019/09/09 19:05:52 DEBUG : Using config file from "/data/user/.config/rclone/rclone.conf"
2019/09/09 19:05:53 DEBUG : pacer: Reducing sleep to 0s
2019/09/09 19:05:53 DEBUG : dir1: Excluded
2019/09/09 19:05:53 DEBUG : dir2: Excluded
2019/09/09 19:05:53 DEBUG : dir3: Excluded
2019/09/09 19:05:53 DEBUG : dir4: Excluded
2019/09/09 19:05:53 DEBUG : dir5: Excluded
2019/09/09 19:05:53 DEBUG : dir6: Excluded
2019/09/09 19:05:53 DEBUG : dir7: Excluded
2019/09/09 19:05:53 DEBUG : dir8: Excluded
2019/09/09 19:05:53 DEBUG : dir9: Excluded
2019/09/09 19:05:53 DEBUG : dir10: Excluded
2019/09/09 19:05:53 DEBUG : dir11: Excluded
2019/09/09 19:05:53 DEBUG : dir12: Excluded
2019/09/09 19:05:53 DEBUG : dir13: Excluded
2019/09/09 19:05:53 DEBUG : dir14: Excluded
2019/09/09 19:05:53 DEBUG : dir15: Excluded
2019/09/09 19:05:53 DEBUG : dir16: Excluded
2019/09/09 19:05:53 DEBUG : dir17: Excluded
2019/09/09 19:05:53 DEBUG : dir18: Excluded
2019/09/09 19:05:53 DEBUG : dir19: Excluded
2019/09/09 19:05:53 DEBUG : dir20: Excluded
2019/09/09 19:05:53 DEBUG : dir21: Excluded
2019/09/09 19:05:53 DEBUG : dir22: Excluded
2019/09/09 19:05:53 DEBUG : dir23: Excluded
2019/09/09 19:05:53 DEBUG : dir24: Excluded
2019/09/09 19:05:53 DEBUG : dir25: Excluded
2019/09/09 19:05:53 DEBUG : dir26: Excluded
2019/09/09 19:05:53 DEBUG : dir27: Excluded
2019/09/09 19:05:53 DEBUG : dir28: Excluded
2019/09/09 19:05:53 DEBUG : dir29: Excluded
2019/09/09 19:05:53 DEBUG : dir30: Excluded
2019/09/09 19:05:53 DEBUG : dir31: Excluded
2019/09/09 19:05:53 DEBUG : dir32/fileb.csv: Excluded
2019/09/09 19:05:53 DEBUG : dir32/filec.csv: Excluded
2019/09/09 19:05:53 INFO : S3 bucket some-s3-bucket: Waiting for checks to finish
2019/09/09 19:05:53 INFO : S3 bucket some-s3-bucket: Waiting for transfers to finish
2019/09/09 19:05:53 INFO : Waiting for deletions to finish
2019/09/09 19:05:53 NOTICE: dir32/filea.csv: Not deleting as --dry-run

Thanks!
Regards

2

That's a pretty old version. Can you update and re-share the log out?

3

Note that rclone didn't actually delete anything, that message is just a note about which phase the sync is in.

4

Hey Nick:
That is weird, because i did got a delete marker in my buket when i ran that.

How can i find out how rclone decided the file should be deleted?

5

If you run with -vv you'll see the files being deleted in the log. I didn't see any in the log you posted though!

They look like this

2019/09/10 21:46:16 INFO  : Waiting for deletions to finish
2019/09/10 21:46:16 INFO  : file: Deleted
6

Hey Nick:
Thanks for the answer.
As mentioned in the thread, this was an example with dry-run enabled, tho show the point.
That is why it shows:
2019/09/09 19:05:53 NOTICE: dir32/filea.csv: Not deleting as --dry-run
If i remove the --dry-run, that would change to DELETE
thanks.
Regards

7

Sorry, I missed that!

The problem is that the --include line you have needs to be relative to the root of the source, so you need

rclone sync --dry-run -vvv --include /filea.csv /nfs/data/ some-s3-bucket:some-s3-bucket-backup/

I'm just wondering why you are using sync here though... sync with filters is dangerous in my opinion! I'd use copy instead.

8

Well, the filter is only to show the issue in one file in order to show a short log.. I actually sync the whole directory, which is where the delete happens.

9

Can you post a full log of the problem? I don't understand what the problem is. Rclone will never delete files from the source, only in the destination.

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.