--backup-dir cannot be in root even when excluded

What is the problem you are having with rclone?

rclone 1.59 added logic to allow the --backup-dir to be in the root of the destination as long as it was excluded. Technically the release notes say 'filtered' ... sync: Overlap check is now filter-sensitive so --backup-dir can be in the root provided it is filtered (Nick) ... and tweaked in 1.59.1.

I brought this up a bit ago as a feature request, but got around it with a for loop. It's just painfully slow in this case and I know now that this is supposed to work. I'm hoping I'm miss-formatting the command but Β―\_(ツ)_/Β―

Nothing I've tried allows me to get around the "Failed to sync: destination and parameter to --backup-dir mustn't overlap" error. I've tried as many possible permutations as I can think of with --exclude, --exclude-from and --filter-from. I'm noting the --exlude command as it seems the most straight-forward

Thanks in advance for any help.

Run the command 'rclone version' and share the full output of the command.

PS C:\> rclone version
rclone v1.63.0
- os/version: Microsoft Windows Server 2012 R2 Standard (64 bit)
- os/kernel: 6.3.9600.20919 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.5
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

SharePoint Online

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync C:\SPO ROOT: --exclude zRecycle/** --backup-dir ROOT:zRecycle --progress

I've ensured that ...

rclone ls ROOT: --exclude zRecycle/**

... does not return the "zRecycle" folder or any of the test files in the folder

The rclone config contents with secrets removed.

[ROOT]
type = onedrive
token =
drive_id = 
drive_type = documentLibrary

A log from the command with the -vv flag

Here's the clone with -vvv ... even though "--exclude zRecycle" is noted, there's no "DEBUG : zRecycle: Excluded" line :

PS C:\> rclone sync C:\SPO ROOT: --exclude zRecycle/** --backup-dir ROOT:zRecycle --progress -vvv
2023/07/12 02:38:01 DEBUG : rclone: Version "v1.63.0" starting with parameters ["C:\\Windows\\system32\\rclone.exe" "sync" "C:\\SPO" "ROOT:" "--exclude" "zRecycle/**" "--backup-dir" "ROOT:zRecycle" "--progress" "-vvv"]
2023/07/12 02:38:01 DEBUG : Creating backend with remote "C:\\SPO"
2023/07/12 02:38:01 DEBUG : Using config file from "C:\\Users\\user\\AppData\\Roaming\\rclone\\rclone.conf"
2023/07/12 02:38:01 DEBUG : fs cache: renaming cache item "C:\\SPO" to be canonical "//?/C:/SPO"
2023/07/12 02:38:01 DEBUG : Creating backend with remote "ROOT:"
2023-07-12 02:38:02 DEBUG : Creating backend with remote "ROOT:zRecycle"
2023-07-12 02:38:02 ERROR : Fatal error received - not attempting retries
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (fatal error encountered)
Elapsed time:         4.4s
2023/07/12 02:38:02 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (fatal error encountered)

Here's the ls with -vvv ... here there's a "DEBUG : zRecycle: Excluded" and the "zRecycle" root directory is not returned:

PS C:\> rclone ls ROOT: --exclude zRecycle/** -vvv
2023/07/12 02:42:03 DEBUG : rclone: Version "v1.63.0" starting with parameters ["C:\\Windows\\system32\\rclone.exe" "ls" "ROOT:" "--exclude" "zRecycle/**" "-vvv"]
2023/07/12 02:42:03 DEBUG : Creating backend with remote "ROOT:"
2023/07/12 02:42:03 DEBUG : Using config file from "C:\\Users\\user\\AppData\\Roaming\\rclone\\rclone.conf"
2023/07/12 02:42:05 DEBUG : zRecycle: Excluded
       39 FileinRoot01.txt
       41 FileinRoot02.txt
2023/07/12 02:42:05 DEBUG : 5 go routines active

Just in case this was a regression, I downloaded and tried 1.59 and 1.59.1 and found the same behavior.

It's either a bug, or more likely I'm holding it wrongβ„’.

Fingers still crossed for an easy answer.

Thanks.Peet

Indeed since v1.59 it should work but does not.

How to reproduce:

$ mkdir src
$ mkdir dst

$ rclone sync src dst --backup-dir dst/Deleted
2023/07/12 10:42:32 ERROR : Fatal error received - not attempting retries
2023/07/12 10:42:32 Failed to sync: destination and parameter to --backup-dir mustn't overlap

$ rclone sync src dst --backup-dir dst/Deleted --exclude "/Deleted/"
2023/07/12 10:42:43 ERROR : Fatal error received - not attempting retries
2023/07/12 10:42:43 Failed to sync: destination and parameter to --backup-dir mustn't overlap

I suspect it has never worked...

And it should since this PR:

I am puzzled because this works fine for me

$ tree src/
src/
β”œβ”€β”€ file.txt
└── link.txt -> file.txt

0 directories, 2 files

$ tree dst
dst
β”œβ”€β”€ file.txt
└── link.txt -> file.txt

0 directories, 2 files

$ echo hellohellosdfsdf > src/file.txt 

$ rclone-v1.63.0 -l sync src dst --backup-dir dst/Deleted --exclude "Deleted/"
$ tree dst
dst
β”œβ”€β”€ Deleted
β”‚   └── file.txt
β”œβ”€β”€ file.txt
└── link.txt -> file.txt

hmmmm....

I have replicated your example to the letter and:

$ rclone sync src dst --backup-dir dst/Deleted --exclude "Deleted/"
2023/07/12 16:45:35 ERROR : Fatal error received - not attempting retries
2023/07/12 16:45:35 Failed to sync: destination and parameter to --backup-dir mustn't overlap

also using released v1.63

Only difference I see is that I am on macOS, you probably Linux, OP on Windows.

And indeed. I have tried on Debian and it works.

I have tried Windows - does not work.

So there is some OS related bug.

this did work for me locally on Ventura:

rclone sync src dst --exclude "backup/" --backup-dir "backup/" -vv

Here's output ...

 % rclone sync src dst --exclude "backup/" --backup-dir "backup/" -vv
2023/07/12 10:32:00 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "sync" "src" "dst" "--exclude" "backup/" "--backup-dir" "backup/" "-vv"]
2023/07/12 10:32:00 DEBUG : Creating backend with remote "src"
2023/07/12 10:32:00 NOTICE: Config file "/Users/peet/.config/rclone/rclone.conf" not found - using defaults
2023/07/12 10:32:00 DEBUG : fs cache: renaming cache item "src" to be canonical "/Users/Shared/TEST_Rclone/src"
2023/07/12 10:32:00 DEBUG : Creating backend with remote "dst"
2023/07/12 10:32:00 DEBUG : fs cache: renaming cache item "dst" to be canonical "/Users/Shared/TEST_Rclone/dst"
2023/07/12 10:32:00 DEBUG : Creating backend with remote "backup/"
2023/07/12 10:32:00 DEBUG : fs cache: renaming cache item "backup/" to be canonical "/Users/Shared/TEST_Rclone/backup"
2023/07/12 10:32:00 DEBUG : backup: Excluded
2023/07/12 10:32:00 DEBUG : Local file system at /Users/Shared/TEST_Rclone/dst: Waiting for checks to finish
2023/07/12 10:32:00 DEBUG : file1: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/07/12 10:32:00 DEBUG : file1: Unchanged skipping
2023/07/12 10:32:00 DEBUG : file2: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/07/12 10:32:00 DEBUG : file2: Unchanged skipping
2023/07/12 10:32:00 DEBUG : Local file system at /Users/Shared/TEST_Rclone/dst: Waiting for transfers to finish
2023/07/12 10:32:00 DEBUG : Waiting for deletions to finish
2023/07/12 10:32:00 INFO  : file2 copy: Deleted
2023/07/12 10:32:00 INFO  : file2 copy: Moved (server-side)
2023/07/12 10:32:00 INFO  : file2 copy: Moved into backup dir
2023/07/12 10:32:00 INFO  : file1 copy: Deleted
2023/07/12 10:32:00 INFO  : file1 copy: Moved (server-side)
2023/07/12 10:32:00 INFO  : file1 copy: Moved into backup dir
2023/07/12 10:32:00 INFO  : There was nothing to transfer
2023/07/12 10:32:00 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 8 / 8, 100%
Deleted:                4 (files), 0 (dirs)
Renamed:                2
Elapsed time:         0.0s

2023/07/12 10:32:00 DEBUG : 6 go routines active

I'll try on a windows box to Sharepoint when I can get in front of it. The issue does seem to be that (for whatever reason) "Excluded" isn't appearing in the the DEBUG in the "windows local -> Sharepoint"

Tell me anything you'd like to see as attempted and I'll give it a whirl for you.

Of course this one works:) as backup dir is in parent folder of src and dst.

Try:

rclone sync src dst --backup-dir dst/backup --exclude "backup/"
1 Like

And I see why it "worked" ...

2023/07/12 10:32:00 DEBUG : fs cache: renaming cache item "backup/" to be canonical "/Users/Shared/TEST_Rclone/backup"

jinx ... and thanks. Not going to lie, I'm more than a bit rummy looking at this for so long!

Oddly, this does work for me on Ventura/rclone1.62.2

% rclone sync src dst --backup-dir dst/backup --exclude "backup/" -vv
2023/07/12 10:44:54 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "sync" "src" "dst" "--backup-dir" "dst/backup" "--exclude" "backup/" "-vv"]
2023/07/12 10:44:54 DEBUG : Creating backend with remote "src"
2023/07/12 10:44:54 NOTICE: Config file "/Users/peet/.config/rclone/rclone.conf" not found - using defaults
2023/07/12 10:44:54 DEBUG : fs cache: renaming cache item "src" to be canonical "/Users/Shared/TEST_Rclone/src"
2023/07/12 10:44:54 DEBUG : Creating backend with remote "dst"
2023/07/12 10:44:54 DEBUG : fs cache: renaming cache item "dst" to be canonical "/Users/Shared/TEST_Rclone/dst"
2023/07/12 10:44:54 DEBUG : Creating backend with remote "dst/backup"
2023/07/12 10:44:54 DEBUG : fs cache: renaming cache item "dst/backup" to be canonical "/Users/Shared/TEST_Rclone/dst/backup"
2023/07/12 10:44:54 DEBUG : backup: Excluded
2023/07/12 10:44:54 DEBUG : file1: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/07/12 10:44:54 DEBUG : file1: Unchanged skipping
2023/07/12 10:44:54 DEBUG : file2: Size and modification time the same (differ by 0s, within tolerance 1ns)
2023/07/12 10:44:54 DEBUG : file2: Unchanged skipping
2023/07/12 10:44:54 DEBUG : Local file system at /Users/Shared/TEST_Rclone/dst: Waiting for checks to finish
2023/07/12 10:44:54 DEBUG : Local file system at /Users/Shared/TEST_Rclone/dst: Waiting for transfers to finish
2023/07/12 10:44:54 DEBUG : Waiting for deletions to finish
2023/07/12 10:44:54 INFO  : file1 copy: Moved (server-side)
2023/07/12 10:44:54 INFO  : file1 copy: Moved into backup dir
2023/07/12 10:44:54 INFO  : file2 copy: Moved (server-side)
2023/07/12 10:44:54 INFO  : file2 copy: Moved into backup dir
2023/07/12 10:44:54 INFO  : There was nothing to transfer
2023/07/12 10:44:54 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 6 / 6, 100%
Deleted:                2 (files), 0 (dirs)
Renamed:                2
Elapsed time:         0.0s

2023/07/12 10:44:54 DEBUG : 4 go routines active

It is getting more interesting... rclone v1.63

macOS:

$ rclone sync src dst --backup-dir dst/backup --exclude "backup/"
# OK

$ rclone sync src dst --backup-dir dst/Deleted --exclude "Deleted/"
2023/07/12 17:50:16 ERROR : Fatal error received - not attempting retries
2023/07/12 17:50:16 Failed to sync: destination and parameter to --backup-dir mustn't overlap

Linux:

$ rclone sync src dst --backup-dir dst/backup --exclude "backup/"
# OK

$ rclone sync src dst --backup-dir dst/Deleted --exclude "Deleted/"
# OK

Windows:

>rclone sync src dst --backup-dir dst/backup --exclude "backup/"
# OK

>rclone sync src dst --backup-dir dst/Deleted --exclude "Deleted/"
2023/07/12 17:57:00 ERROR : Fatal error received - not attempting retries
2023/07/12 17:57:00 Failed to sync: destination and parameter to --backup-dir mustn't overlap

so @peet1 you know what to do to make your sync to work.

Until this bug (as clearly there is one) is fixed.

Holy crap. It's the CAPITAL letters ... it fails on capital letters. I'd tried renaming my "--backup-dir" to many, many different things even single letters like "z". But, all my destinations for these copies have either included CamelCase or ALLCAPS, so "--backup-dir RemoteDest:z/" and "--backup-dir REMOTE:z/" would fail.

Sheesh. All I can say is that I'm glad to not be programming ... anything.

Just an FYI, when all capital characters are stripped out of the remote name, and the "backup-dir" does not have a capital letter in it, it functions as intended.

> rclone sync C:\SPO root: --backup-dir root:/z_recycle --exclude z_recycle/ -vv
2023/07/12 11:28:47 DEBUG : rclone: Version "v1.63.0" starting with parameters ["C:\\Windows\\system32\\rclone.exe" "sync" "C:\\SPO" "root:" "--backup-dir" "root:/z_recycle" "--exclude" "z_recycle/" "-vv"]
2023/07/12 11:28:47 DEBUG : Creating backend with remote "C:\\SPO"
2023/07/12 11:28:47 DEBUG : Using config file from "C:\\Users\\wlfadmin.WLF\\AppData\\Roaming\\rclone\\rclone.conf"
2023/07/12 11:28:47 DEBUG : fs cache: renaming cache item "C:\\SPO" to be canonical "//?/C:/SPO"
2023/07/12 11:28:47 DEBUG : Creating backend with remote "root:"
2023/07/12 11:28:47 DEBUG : Creating backend with remote "root:/z_recycle"
2023/07/12 11:28:48 DEBUG : fs cache: renaming cache item "root:/z_recycle" to be canonical "root:z_recycle"
2023/07/12 11:28:48 DEBUG : z_recycle: Excluded
2023/07/12 11:28:48 DEBUG : OneDrive root '': Waiting for checks to finish
2023/07/12 11:28:48 DEBUG : file02.txt: Size and modification time the same (differ by -357.3742ms, within tolerance 1s)
2023/07/12 11:28:48 DEBUG : file02.txt: Unchanged skipping
2023/07/12 11:28:48 DEBUG : file01.txt: Size and modification time the same (differ by -357.3742ms, within tolerance 1s)
2023/07/12 11:28:48 DEBUG : file01.txt: Unchanged skipping
2023/07/12 11:28:48 DEBUG : OneDrive root '': Waiting for transfers to finish
2023/07/12 11:28:48 DEBUG : Waiting for deletions to finish
2023/07/12 11:28:48 INFO  : There was nothing to transfer
2023/07/12 11:28:48 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 2 / 2, 100%
Elapsed time:         1.3s

2023/07/12 11:28:48 DEBUG : 6 go routines active

I hope this gives you enough to work with. And, seriously, thank you for all you do! rclone is the bee's knees.

Forgot to say Thanks! You would believe how much time, effort and silly work-arounds this will save me on SharePoint migrations.

Since the SharePoint API can't reliably delete files "--backup-dir" is crucial to getting data into SPO.

Cheers.Peet

1 Like

@ncw - I have never seen such rclone behaviour and I use it quite a lot on various systems. It let me believe that it is not some general issues but something introduced by these two PRs:

I have looked at the code but I can not see why it would be sensitive to backup-dir with capital letters and only on Windows and macOS.

Just to add another data point, I tested on macOS 12.6.7 (Intel) in a case sensitive disk image (my main drive is the default case insensitive) and it did, in fact, fail with capital letters:

Works:

$ rclone sync src dst --backup-dir dst/backup --exclude "backup/"

Fails:

$ rclone sync src dst --backup-dir dst/B --exclude "B/"

I do not think it depends on filesystem case sensitivity - your test actually shows it.

It always fails on macOS and Windows if backup directory name contains capital letters.

I should have been more clear. I was trying to see if that made a difference. It did not.

1 Like

This is very weird bug... I have tried to analyse the code - but there is nothing obvious there indicating problems. And all works on Linux.. if it was only Windows affected I would be less surprised. macOS is proper Unix and such things if working on Linux work on macOS usually.

$ rclone sync src dst --backup-dir dst/Backup --exclude 'Backup/' --dump filters
2023/07/13 17:01:58 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
- (^|/)Backup/.*$
--- Directory filter rules ---
- (^|/)Backup/.*$
--- end filters ---
2023/07/13 17:01:58 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "sync" "src" "dst" "--backup-dir" "dst/Backup" "--exclude" "Backup/" "--dump" "filters"]
2023/07/13 17:01:58 DEBUG : Creating backend with remote "src"
2023/07/13 17:01:58 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2023/07/13 17:01:58 DEBUG : fs cache: renaming cache item "src" to be canonical "/Users/kptsky/Temp/test/src"
2023/07/13 17:01:58 DEBUG : Creating backend with remote "dst"
2023/07/13 17:01:58 DEBUG : fs cache: renaming cache item "dst" to be canonical "/Users/kptsky/Temp/test/dst"
2023/07/13 17:01:58 DEBUG : Creating backend with remote "dst/Backup"
2023/07/13 17:01:58 DEBUG : fs cache: renaming cache item "dst/Backup" to be canonical "/Users/kptsky/Temp/test/dst/Backup"
2023/07/13 17:01:58 ERROR : Fatal error received - not attempting retries
2023/07/13 17:01:58 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (fatal error encountered)
Elapsed time:         0.0s

2023/07/13 17:01:58 DEBUG : 5 go routines active
2023/07/13 17:01:58 Failed to sync: destination and parameter to --backup-dir mustn't overlap

filters are ok:

--- start filters ---
--- File filter rules ---
- (^|/)Backup/.*$
--- Directory filter rules ---
- (^|/)Backup/.*$
--- end filters ---

Backup directory is OK:

2023/07/13 17:01:58 DEBUG : fs cache: renaming cache item "dst/Backup" to be canonical "/Users/kptsky/Temp/test/dst/Backup"

but it fails at the end anyway

Thank you for your investigations.

I've discovered that this is related to whether the local file system is case sensitive or not

Fails

$ rclone-v1.63.0 -l sync src dst --backup-dir dst/Deleted --exclude "Deleted/" --local-case-insensitive
2023/07/14 11:02:07 ERROR : Fatal error received - not attempting retries
2023/07/14 11:02:07 Failed to sync: destination and parameter to --backup-dir mustn't overlap
$

Works

$ rclone-v1.63.0 -l sync src dst --backup-dir dst/Deleted --exclude "Deleted/" --local-case-sensitive
$ 

So thankfully it isn't OS dependent! I will investigate further but this should be enough to pin down the problem when I'm back at my desk!

1 Like