Sync not removing deleted files

Help and Support
1

What is the problem you are having with rclone?

Sync job never seems to compete, it appears to get everything copied over, but doesn't delete any files.
Have been running it on and off for a year, but only realised today that its never removing files that no longer exist.
Have simply deleted the files on the source that it said were giving errors, but it still doesn't complete. Any ideas?
I'm not super-technical but can script enough to use recline (just about!)

Run the command 'rclone version' and share the full output of the command.

rclone v1.55.1

  • os/type: darwin

  • os/arch: amd64

  • go/version: go1.16.3

  • go/linking: dynamic

  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -v -PL --stats 2s --track-renames --track-renames-strategy modtime --backup-dir <previous_date> --filter-from /<path>/rcloneFilters.txt / DB_rclone_crypt:/<destPath>/

The rclone config contents with secrets removed.

to follow, once I find it....

A log from the command with the -vv flag

Too much to redact... some snippets..... at this point there's nothing new to sync, so should be cleaning up.
After several minutes I get this:

START
.
.

2022-05-19 21:37:22 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 300 seconds.
2022-05-19 21:37:22 DEBUG : pacer: low level retry 1/10 (error too_many_requests/...)
2022-05-19 21:37:22 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2022-05-19 21:37:22 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 300 seconds.
2022-05-19 21:37:22 DEBUG : pacer: low level retry 1/10 (error too_many_requests/)
2022-05-19 21:37:22 DEBUG : pacer: Reducing sleep to 3m45s
2022-05-19 21:37:22 NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 300 seconds.
2022-05-19 21:37:22 DEBUG : pacer: low level retry 2/10 (error too_many_requests/..)
2022-05-19 21:37:22 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2022-05-19 21:37:22 DEBUG : pacer: Reducing sleep to 3m45s
2022-05-19 21:37:22 DEBUG : pacer: Reducing sleep to 2m48.75s
2022-05-19 21:37:22 DEBUG : pacer: Reducing sleep to 2m6.5625s
.
.

2022-05-19 21:37:22 DEBUG : pacer: Reducing sleep to 1m34.921875s
2022-05-19 21:37:25 DEBUG : pacer: Reducing sleep to 1m11.19140625s
.
.
2022-05-19 21:46:16 DEBUG : pacer: Reducing sleep to 9.502905606s
.

** endless 'Size and modification time the same (differ by 0s, within tolerance 1s) type output **
.
.
.
Transferred:   	   56.903M / 56.903 MBytes, 100%, 2.356 MBytes/s, ETA 0s
Errors:                 1 (no need to retry)
Checks:             31768 / 31768, 100%
Transferred:            5 / 5, 100%
Elapsed time:      14m4.7s
.
.
** Then seems to just sit here without doing anything further **
** unless something's happening and not appearing in -vv **
.

Transferred:   	   56.903M / 56.903 MBytes, 100%, 2.356 MBytes/s, ETA 0s
Errors:                 1 (no need to retry)
Checks:             31768 / 31768, 100%
Transferred:            5 / 5, 100%
Elapsed time:     15m54.7s
.
.
Transferred:   	   56.903M / 56.903 MBytes, 100%, 2.356 MBytes/s, ETA 0s
Errors:                 1 (no need to retry)
Checks:             31768 / 31768, 100%
Transferred:            5 / 5, 100%
Elapsed time:     17m34.7s

2022-05-19 21:53:59 DEBUG : pacer: Reducing sleep to 1m11.19140625s
Transferred:   	   56.903M / 56.903 MBytes, 100%, 2.356 MBytes/s, ETA 0s
Errors:                 1 (no need to retry)
Checks:             31768 / 31768, 100%
Transferred:            5 / 5, 100%
Elapsed time:     20m26.7s
..
2022-05-19 22:00:55 INFO  : Encrypted drive 'DB_rclone_crypt:/laptop/latest/': Making map for --track-renames
2022-05-19 22:00:55 INFO  : Encrypted drive 'DB_rclone_crypt:/laptop/latest/': Finished making map for --track-renames
2022-05-19 22:00:55 DEBUG : Encrypted drive 'DB_rclone_crypt:/laptop/latest/': Waiting for checks to finish
2022-05-19 22:00:55 DEBUG : Encrypted drive 'DB_rclone_crypt:/laptop/latest/': Waiting for renames to finish
2022-05-19 22:00:55 DEBUG : Encrypted drive 'DB_rclone_crypt:/laptop/latest/': Waiting for transfers to finish
2022-05-19 22:00:55 ERROR : Encrypted drive 'DB_rclone_crypt:/laptop/latest/': not deleting files as there were IO errors
2022-05-19 22:00:55 ERROR : Encrypted drive 'DB_rclone_crypt:/laptop/latest/': not deleting directories as there were IO errors
2022-05-19 22:00:55 ERROR : Attempt 3/3 failed with 2 errors and: not deleting files as there were IO errors
..
Transferred:   	   56.903M / 56.903 MBytes, 100%, 2.222 MBytes/s, ETA 0s
Errors:                 2 (retrying may help)
Checks:             38512 / 38512, 100%
Transferred:            5 / 5, 100%
Elapsed time:     25m56.0s
2022/05/19 22:00:55 INFO  : 
Transferred:   	   56.903M / 56.903 MBytes, 100%, 2.222 MBytes/s, ETA 0s
Errors:                 2 (retrying may help)
Checks:             38512 / 38512, 100%
Transferred:            5 / 5, 100%
Elapsed time:     25m56.0s

2022/05/19 22:00:55 DEBUG : 5 go routines active
2022/05/19 22:00:55 Failed to sync with 2 errors: last error was: not deleting files as there were IO errors
2

hi,

based on forum posts, try
--tpslimit=10 --tpslimit-burst=10

best to update to latest stable v1.58.1

try rclone config show DB_rclone_crypt:
and redact as needed.

3

The big piece there is sync won't delete files if there are IO errors. Your logs have tons of IO errors so deletions won't happen.

For each app registration, the sweet for me has been 12 tps for each value, but on my mounts I was a little more cautious and I set it to 10 as I've never had any issues on performance.

4

Thanks for all the feedback, I’ll try the suggestions when I’m back at my laptop and add more information. I love shell type stuff and rclone, but in reality it’s probably just out of my comfort zone and should really try and find a decent gui style alternative!

To be honest I did download the latest (it was the data that copied in this sync) but I couldn’t actually work out how to install it! I didn’t want to reconfigure everything, just use my current config. I’ve done it in the past but my brain failed me!

Also it says mount no longer works and I do use that feature as I find it way easier to visually navigate in finder to check the backup.

5

The rclone config contents with secrets removed.

> rclone config show DB_rclone_crypt:
--------------------
[DB_rclone_crypt]
type = crypt
remote = DB_rclone:/_backups/rclone/Crypt/
filename_encryption = obfuscate
directory_name_encryption = true
password = *** ENCRYPTED ***
...
6

Thanks for all the help so far....

Ran the job again, using the suggested parameters above of --tpslimit=10 --tpslimit-burst=10

It definitely ran much smoother and none of the limiter thingies kicked in.
However, still the same error.

2022/05/20 11:12:05 INFO  : 
Transferred:   	   16.622M / 16.622 MBytes, 100%, 796.629 kBytes/s, ETA 0s
Errors:                 2 (retrying may help)
Checks:             38523 / 38523, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                4
Transferred:            4 / 4, 100%
Elapsed time:     17m17.8s

2022/05/20 11:12:05 Failed to sync with 2 errors: last error was: not deleting files as there were IO errors

As far as I can tell there is only one actual file error:

2022-05-20 11:00:39 ERROR : .VolumeIcon.icns: Listing error: symlink: stat /.VolumeIcon.icns: no such file or directory

But .VolumeIcon.icns is specifically excluded in my filter-from list.

- .DS_Store
- .VolumeIcon.icns

+ /<includePathA>/**
+ /<includePathB>/** 
+ (etc)

# exclude everything else
- *

Any other ideas?

Also I finally found the selfupdate command.
I ran it but it failed. Any ideas there?

/ % sudo rclone selfupdate --check
Password:
yours:  1.55.1       
latest: 1.58.1                                   (released 2022-04-29)
  upgrade: https://downloads.rclone.org/v1.58.1
beta:   1.59.0-beta.6157.3ec25f437               (released 2022-05-19)
  upgrade: https://beta.rclone.org/v1.59.0-beta.6157.3ec25f437

/ % sudo rclone selfupdate        
2022/05/20 11:16:33 Error: failed to update rclone: failed with 404 Not Found downloading https://downloads.rclone.org/v1.58.1/rclone-v1.58.1-darwin-amd64.zip
/ %
7

not sure, i would try again, or do the manual install/update

8

The release is actually called rclone-v1.58.1-macos-amd64.zip

I think this is an old bug we fixed already but you'll have to do a manual install - sorry

Unfortunately the excludes get run after the directory listings. I think this particular .VolumeIcon.icns is a dangling symlink - can you remove it?

9

Not a problem, I just couldn't seem to work out doing it.

Thanks
As far as I can tell, this is a Mac system file -at the root of the drive and well away from my data folders - don't think I want to be getting into deleting it to backup my stuff.....

I think in reality rclone is probably the wrong tool for me here. I love rsync in general and like the ability to have a sort of snapshot, but for what I'm using it for it's a bit too complex when I add all the steps and trouble-shooting.
I think I'll have to sacrifice the power of sync, and just locally encrypt and then copy it all the dropbox on a regular basis. It's not such a big dataset. What to do that with I'm not sure yet!

10

hi, there are a few GUI for rclone
--- rclone itself has as simple gui
--- RcloneBrowser
--- rcx on android.

and using rclone mount DB_rclone_crypt: /path/to/local/dir, can have dropbox appear as a local folder.
now, can use the macos file explorer, or most any other tool to view/copy /path/to/local/dir

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.