Rclone sync not delete files on destination

Help and Support
1

What is the problem you are having with rclone?

rclone sync - just dumping files and destination folder size is 3 times more the source folder size

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.58.0

  • os/version: centos 7.9.2009 (64 bit)
  • os/kernel: 3.10.0-1160.59.1.el7.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.8
  • go/linking: static
  • go/tags: none

yes on latest version

Which cloud storage system are you using? (eg Google Drive)

SwiftStack

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here
```/usr/bin/rclone sync source folder destination folder --transfers 80 --checkers 80 --checksum --ignore-existing --ignore-errors --max-age 3h --max-delete 50 --log-file /var/log/rclone/sync.log_`date +\%H\%M` --log-level INFO

#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

Paste config here

[source]
type = swift
user = ****
key = ****
auth = http://****
domain =
tenant =
tenant_domain =
region =
storage_url =
auth_version = 1

[destination]
type = swift
user = ****
key = ****
auth = http://****
domain =
tenant =
tenant_domain =
region =
storage_url =
auth_version = 1


#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

Paste log here

2

Pl note - I just added the "--max-delete 50" flag today. Wanted to see if that will help.

3

hi,
if there is an error during the sync, then rclone will not delete files.
that behavior can be tweaked.

using a rclone debug log would show that.

4

Hi! Thanks for the reply.
Yes, I have used DEBUG option in the past and saw some errors - this sync is done long distance (between countries) so I am sure there will be errors at some point during the sync and over the years the files just built up at the destination.

So I have two options:

  1. Delete the contents of destination and restart a fresh sync job.
  2. Clean up the destination to match the source - can you pl suggest how I can achieve this? Does rclone dedupe or check work ? If so, what would be the command?

Obviously, I do not want to manually compare and delete stuff - the data at source is around 4 TB and destination is around 16 TB.

Appreciate your help.

5

Check out the docs here

And then use --delete-during and rclone sync should cleanup for you. You might need to use --delete-before if that doesn't work.

6

Hi! Nick,
Look that none of these options have helped - did not delete any data. So, I had to delete all the content on the destination manually and planning to do a fresh copy/sync again.
So, I need some advice here - Should I use copy or sync ?

What will the command be - can you pl modify the below command?

```/usr/bin/rclone sync source folder destination folder --transfers 80 --checkers 80 --checksum --ignore-existing --ignore-errors --max-age 3h --max-delete 50 --log-file /var/log/rclone/sync.log_date +\%H\%M --log-level INFO

What I want is to make sure is destination folder has to be identical to source folder.

7

Please try removing "--ignore-existing" from your command

8

I think this is correct - --ignore-existing has been the root of all your problems!

9

Thank you Sl-Gundam and Nick.
I will keep that in mind the next time. So now I have an empty destination - I would like to make sure I have the right command to start the copy sync again.
Can one of you pl review the below command and let me know of any changes ?

Should I do a sync even thought the destination folder is empty or should I do a copy?

/usr/bin/rclone sync source folder destination folder --transfers 80 --checkers 80 --checksum --ignore-errors  --log-file /var/log/rclone/sync.log_date +\%H\%M --log-level INFO
10

I would remove this, unless you have a good reason for it.

These are probably a bit high, unless you've tested they actually increase throughput.

You can do either. I usually start with rclone copy as it is safer (it never deletes stuff) then try rclone sync --dry-run then rclone sync.

11

Thank you very much Nick. Appreciate it.

1 Like
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.