Sync --backup-dir repeatedly backs up one file even if that file is not touched

Suspected Bug
1

What is the problem you are having with rclone?

I'm doing backup with versioning, of several drives on my Win computer using a PowerShell script.

However, each time sync is run, only one file out of the tens of thousands gets repeatedly copied in the versioned backup directory even though that file is never touched.

I have read all the related posts in the forum but no clue so far.

Then I conducted some tests suspecting that there was something peculiar in the file name itself, which was "Tax Deductible - <U+202A>Inequality Media<U+202C> receipt [#1345-8796].pdf" (119KB)

I copied this file name to an empty text file in the same folder "Tax Deductible - <U+202A>Inequality Media<U+202C> receipt [#1345-8796].txt" (0KB) and sync did the same thing. Ultimately, copied various combinations of the above file name in the same test directory, until I narrowed down the problem to some invisible char difference. I'm only giving the file names that will exhibit the failure and pass conditions for the sake of brevity (all are 0 byte empty text files):

Tax Deductible - <U+202A>Inequality Media<U+202C> receipt.txt Fails
Tax Deductible.txt Fails
Media<U+202C>.txt Fails
abc.txt Passes
Manually typed Media.txt Passes

Interestingly, the above two Media.txt files, both of which show the EXACT same name in Folder Explorer, can coexist in the same folder! (This is the first time I've seen such a behavior in Windows).

So there must be some encoding difference in the characters used in the original file name (the PDF), which is making rclone think that this file always keeps changing, and so it always keeps putting a copy in the backup versioning folder verDir.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.2486 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

OneDrive Personal

The command you were trying to run (eg rclone copy /tmp remote:tmp)

.\rclone.exe sync c:\rclone\testDir od:testDir --backup-dir od:verDir

The rclone config contents with secrets removed.

[od]
type = onedrive
token = 
drive_id = 
drive_type = personal

A log from the command with the -vv flag

.\rclone.exe : 2023/01/28 16:53:04 DEBUG : rclone: Version "v1.61.1" starting with parameters ["C:\\rclone\\rclone.exe" "sync" 
"c:\\rclone\\testDir" "od:testDir" "--backup-dir" "od:verDir" "-vv"]
At line:1 char:1
+ .\rclone.exe sync c:\rclone\testDir od:testDir --backup-dir od:verDir ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2023/01/28 16:5...:verDir" "-vv"]:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2023/01/28 16:53:04 DEBUG : Creating backend with remote "c:\\rclone\\testDir"
2023/01/28 16:53:04 DEBUG : Using config file from "C:\\Users\\Ashim Jain\\AppData\\Roaming\\rclone\\rclone.conf"
2023/01/28 16:53:04 DEBUG : fs cache: renaming cache item "c:\\rclone\\testDir" to be canonical "//?/c:/rclone/testDir"
2023/01/28 16:53:04 DEBUG : Creating backend with remote "od:testDir"
2023/01/28 16:53:04 DEBUG : Creating backend with remote "od:verDir"
2023/01/28 16:53:05 DEBUG : abc.txt: Size and modification time the same (differ by -849.1543ms, within tolerance 1s)
2023/01/28 16:53:05 DEBUG : abc.txt: Unchanged skipping
2023/01/28 16:53:05 DEBUG : Abcd.txt: Size and modification time the same (differ by -461.6576ms, within tolerance 1s)
2023/01/28 16:53:05 DEBUG : Abcd.txt: Unchanged skipping
2023/01/28 16:53:05 DEBUG : Media.txt: Size and modification time the same (differ by -723.1045ms, within tolerance 1s)
2023/01/28 16:53:05 DEBUG : Media.txt: Unchanged skipping
2023/01/28 16:53:05 DEBUG : Tax Deductible.txt: Size and modification time the same (differ by -897.0414ms, within tolerance 
1s)
2023/01/28 16:53:05 DEBUG : Tax Deductible.txt: Unchanged skipping
2023/01/28 16:53:05 DEBUG : OneDrive root 'testDir': Waiting for checks to finish
2023/01/28 16:53:05 DEBUG : OneDrive root 'testDir': Waiting for transfers to finish
2023/01/28 16:53:05 DEBUG : Media‬.txt: Starting singlepart upload
2023/01/28 16:53:05 DEBUG : Tax Deductible  ‪Inequality Media‬ receipt.txt: Starting singlepart upload
2023/01/28 16:53:05 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].pdf: Starting multipart upload
2023/01/28 16:53:05 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].txt: Starting multipart upload
2023/01/28 16:53:06 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].pdf: Uploading segment 0/121682 size 
121682
2023/01/28 16:53:06 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].txt: Uploading segment 0/16 size 16
2023/01/28 16:53:06 DEBUG : Tax Deductible  ‪Inequality Media‬ receipt.txt: sha1 = 
da39a3ee5e6b4b0d3255bfef95601890afd80709 OK
2023/01/28 16:53:06 INFO  : Tax Deductible  ‪Inequality Media‬ receipt.txt: Copied (new)
2023/01/28 16:53:06 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt.txt: Starting singlepart upload
2023/01/28 16:53:06 DEBUG : Media‬.txt: sha1 = da39a3ee5e6b4b0d3255bfef95601890afd80709 OK
2023/01/28 16:53:06 INFO  : Media‬.txt: Copied (new)
2023/01/28 16:53:06 DEBUG : ‪Inequality Media‬ receipt.txt: Starting singlepart upload
2023/01/28 16:53:06 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].pdf: sha1 = 
959bcf426ec19b32123ef6154598deedc244986b OK
2023/01/28 16:53:06 INFO  : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].pdf: Copied (new)
2023/01/28 16:53:06 DEBUG : ‪Inequality Media‬.txt: Starting singlepart upload
2023/01/28 16:53:06 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].txt: sha1 = 
eb7ff18b50ae38e58d9c5308ecfcdc3bfa50359c OK
2023/01/28 16:53:06 INFO  : Tax Deductible - ‪Inequality Media‬ receipt [#1345-8796].txt: Copied (new)
2023/01/28 16:53:06 DEBUG : ‪Inequality.txt: Starting singlepart upload
2023/01/28 16:53:07 DEBUG : ‪Inequality Media‬.txt: sha1 = da39a3ee5e6b4b0d3255bfef95601890afd80709 OK
2023/01/28 16:53:07 INFO  : ‪Inequality Media‬.txt: Copied (new)
2023/01/28 16:53:07 DEBUG : Tax Deductible - ‪Inequality Media‬ receipt.txt: sha1 = 
da39a3ee5e6b4b0d3255bfef95601890afd80709 OK
2023/01/28 16:53:07 INFO  : Tax Deductible - ‪Inequality Media‬ receipt.txt: Copied (new)
2023/01/28 16:53:07 DEBUG : ‪Inequality Media‬ receipt.txt: sha1 = da39a3ee5e6b4b0d3255bfef95601890afd80709 OK
2023/01/28 16:53:07 INFO  : ‪Inequality Media‬ receipt.txt: Copied (new)
2023/01/28 16:53:07 DEBUG : ‪Inequality.txt: sha1 = da39a3ee5e6b4b0d3255bfef95601890afd80709 OK
2023/01/28 16:53:07 INFO  : ‪Inequality.txt: Copied (new)
2023/01/28 16:53:07 DEBUG : Waiting for deletions to finish
2023/01/28 16:53:07 INFO  : Tax Deductible - Inequality Media receipt.txt: Deleted
2023/01/28 16:53:07 INFO  : Inequality Media receipt.txt: Deleted
2023/01/28 16:53:07 INFO  : Tax Deductible - Inequality Media receipt [#1345-8796].txt: Deleted
2023/01/28 16:53:07 INFO  : Tax Deductible - Inequality Media receipt [#1345-8796].pdf: Deleted
2023/01/28 16:53:08 INFO  : Tax Deductible - Inequality Media receipt.txt: Moved (server-side)
2023/01/28 16:53:08 INFO  : Tax Deductible - Inequality Media receipt.txt: Moved into backup dir
2023/01/28 16:53:08 INFO  : Tax Deductible - Inequality Media receipt [#1345-8796].txt: Moved (server-side)
2023/01/28 16:53:08 INFO  : Tax Deductible - Inequality Media receipt [#1345-8796].txt: Moved into backup dir
2023/01/28 16:53:08 INFO  : Inequality Media receipt.txt: Moved (server-side)
2023/01/28 16:53:08 INFO  : Inequality Media receipt.txt: Moved into backup dir
2023/01/28 16:53:08 INFO  : Tax Deductible - Inequality Media receipt [#1345-8796].pdf: Moved (server-side)
2023/01/28 16:53:08 INFO  : Tax Deductible - Inequality Media receipt [#1345-8796].pdf: Moved into backup dir
2023/01/28 16:53:08 INFO  : Inequality.txt: Deleted
2023/01/28 16:53:08 INFO  : Tax Deductible  Inequality Media receipt.txt: Deleted
2023/01/28 16:53:08 INFO  : Inequality Media.txt: Deleted
2023/01/28 16:53:08 INFO  : Tax Deductible  Inequality Media receipt.txt: Moved (server-side)
2023/01/28 16:53:08 INFO  : Tax Deductible  Inequality Media receipt.txt: Moved into backup dir
2023/01/28 16:53:08 INFO  : Inequality Media.txt: Moved (server-side)
2023/01/28 16:53:08 INFO  : Inequality Media.txt: Moved into backup dir
2023/01/28 16:53:09 INFO  : Inequality.txt: Moved (server-side)
2023/01/28 16:53:09 INFO  : Inequality.txt: Moved into backup dir
2023/01/28 16:53:09 INFO  : 
Transferred:   	  237.691 KiB / 237.691 KiB, 100%, 78.967 KiB/s, ETA 0s
Checks:                25 / 25, 100%
Deleted:               14 (files), 0 (dirs)
Renamed:                7
Transferred:            8 / 8, 100%
Elapsed time:         4.6s
2023/01/28 16:53:09 DEBUG : 8 go routines active
2

It might be because of the invalid characters as they have to be converted to be uploaded.

The "#" isn't allowed in OneDrive/Sharepoint.

3

In the final test with "Media.txt", there was no "#" but it was still failing.

No problem, I've resolved in a different way. Just wanted to bring this to your attention.

Thank you for your valuable and prompt support!

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.