Chaning directory case makes rclone lose it

jared · January 5, 2023, 2:55am

What is the problem you are having with rclone?

if the letter case of a directory is changed, rclone deletes the existing case version, but doesn't upload new one. running the sync again then uploads it. this is possibly a microsoft bug, because rclone logs a checksum it received. i suspect it didn't make up the checksum.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

os/version: Microsoft Windows Server 2019 Datacenter 1809 (64 bit)
os/kernel: 10.0.17763.3406 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.19.4
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Microsoft Sharepoint

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync --verbose -vv --progress --transfers=6 "E:\Jared\Test Directory" "ServerSharePointBackupnew:Jared/Test Directory"

The rclone config contents with secrets removed.

[ServerSharePoint]
type = onedrive
token = yes it works
drive_id = asdfasdf
drive_type = documentLibrary

[ServerSharePointBackupnew]
type = crypt
remote = ServerSharePoint:Server New
directory_name_encryption = false
password = yes i know

A log from the command with the `-vv` flag

i will paste 3 logs here. they are of syncs of a folder called "tEST being renamed to Test" and said folder contains a file "test.txt" they are as follows:

sync with no changes to show everything running it's normal course
sync with the directory name changed, showing rlone uploading the new directory and kicking out the old and receiving the checksum.
rclone again uploading the new directory, this time it actually appears in the remote. i'm hoping ya'll can take my word for it that after sync 2 (shown in log 2) i can't see the file on the web gui for said remote. there is also the added evidence that with sync 3 (shown in log 3) rclone goes through the upload again and says "copied (new)" again.

log 1.

2023/01/04 20:42:03 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "--verbose" "-vv" "--progress" "--transfers=6" "E:\\Jared\\Test Directory" "ServerSharePointBackupnew:Jared/Test Directory" "--log-file=C:\\Users\\Administrator\\Desktop\\log.txt"]
2023/01/04 20:42:03 DEBUG : Creating backend with remote "E:\\Jared\\Test Directory"
2023/01/04 20:42:03 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/01/04 20:42:03 DEBUG : fs cache: renaming cache item "E:\\Jared\\Test Directory" to be canonical "//?/E:/Jared/Test Directory"
2023/01/04 20:42:03 DEBUG : Creating backend with remote "ServerSharePointBackupnew:Jared/Test Directory"
2023/01/04 20:42:03 DEBUG : Creating backend with remote "ServerSharePoint:Server New/Jared/s09dg7f09ds8f7g9sd8f79g78d0sg7s0d9g7f"
2023/01/04 20:42:05 DEBUG : Creating backend with remote "ServerSharePoint:Server New/Jared/Test Directory"
2023/01/04 20:42:07 DEBUG : Encrypted drive 'ServerSharePointBackupnew:Jared/Test Directory': Waiting for checks to finish
2023/01/04 20:42:07 DEBUG : tEST/test.txt: Size and modification time the same (differ by -641.8295ms, within tolerance 1s)
2023/01/04 20:42:07 DEBUG : tEST/test.txt: Unchanged skipping
2023/01/04 20:42:07 DEBUG : Encrypted drive 'ServerSharePointBackupnew:Jared/Test Directory': Waiting for transfers to finish
2023/01/04 20:42:07 DEBUG : Waiting for deletions to finish
2023/01/04 20:42:07 INFO  : There was nothing to transfer
2023/01/04 20:42:07 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         4.5s

2023/01/04 20:42:07 DEBUG : 4 go routines active

log 2.

2023/01/04 20:44:30 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "--verbose" "-vv" "--progress" "--transfers=6" "E:\\Jared\\Test Directory" "ServerSharePointBackupnew:Jared/Test Directory" "--log-file=C:\\Users\\Administrator\\Desktop\\log.txt"]
2023/01/04 20:44:30 DEBUG : Creating backend with remote "E:\\Jared\\Test Directory"
2023/01/04 20:44:30 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/01/04 20:44:30 DEBUG : fs cache: renaming cache item "E:\\Jared\\Test Directory" to be canonical "//?/E:/Jared/Test Directory"
2023/01/04 20:44:30 DEBUG : Creating backend with remote "ServerSharePointBackupnew:Jared/Test Directory"
2023/01/04 20:44:30 DEBUG : Creating backend with remote "ServerSharePoint:Server New/Jared/s09dg7f09ds8f7g9sd8f79g78d0sg7s0d9g7f"
2023/01/04 20:44:33 DEBUG : Creating backend with remote "ServerSharePoint:Server New/Jared/Test Directory"
2023/01/04 20:44:35 DEBUG : Encrypted drive 'ServerSharePointBackupnew:Jared/Test Directory': Waiting for checks to finish
2023/01/04 20:44:35 DEBUG : Encrypted drive 'ServerSharePointBackupnew:Jared/Test Directory': Waiting for transfers to finish
2023/01/04 20:44:35 DEBUG : Test/tpb8mh3pae5u8mlk35jf4nhra0: Starting multipart upload
2023/01/04 20:44:36 DEBUG : Test/tpb8mh3pae5u8mlk35jf4nhra0: Uploading segment 0/32 size 32
2023/01/04 20:44:36 DEBUG : Test/test.txt: quickxor = 74f43a7cdef09e1283681c1d0132bb7a36c7eec9 OK
2023/01/04 20:44:36 INFO  : Test/test.txt: Copied (new)
2023/01/04 20:44:36 DEBUG : Waiting for deletions to finish
2023/01/04 20:44:37 INFO  : tEST/test.txt: Deleted
2023/01/04 20:44:37 INFO  : tEST: Removing directory
2023/01/04 20:44:37 DEBUG : Encrypted drive 'ServerSharePointBackupnew:Jared/Test Directory': deleted 1 directories
2023/01/04 20:44:37 INFO  : 
Transferred:   	         32 B / 32 B, 100%, 16 B/s, ETA 0s
Checks:                 1 / 1, 100%
Deleted:                1 (files), 1 (dirs)
Transferred:            1 / 1, 100%
Elapsed time:         7.0s

2023/01/04 20:44:37 DEBUG : 10 go routines active

log 3.

2023/01/04 20:45:47 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "--verbose" "-vv" "--progress" "--transfers=6" "E:\\Jared\\Test Directory" "ServerSharePointBackupnew:Jared/Test Directory" "--log-file=C:\\Users\\Administrator\\Desktop\\log.txt"]
2023/01/04 20:45:47 DEBUG : Creating backend with remote "E:\\Jared\\Test Directory"
2023/01/04 20:45:47 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/01/04 20:45:47 DEBUG : fs cache: renaming cache item "E:\\Jared\\Test Directory" to be canonical "//?/E:/Jared/Test Directory"
2023/01/04 20:45:47 DEBUG : Creating backend with remote "ServerSharePointBackupnew:Jared/Test Directory"
2023/01/04 20:45:47 DEBUG : Creating backend with remote "ServerSharePoint:Server New/Jared/s09dg7f09ds8f7g9sd8f79g78d0sg7s0d9g7f"
2023/01/04 20:45:49 DEBUG : Creating backend with remote "ServerSharePoint:Server New/Jared/Test Directory"
2023/01/04 20:45:51 DEBUG : Encrypted drive 'ServerSharePointBackupnew:Jared/Test Directory': Waiting for checks to finish
2023/01/04 20:45:51 DEBUG : Encrypted drive 'ServerSharePointBackupnew:Jared/Test Directory': Waiting for transfers to finish
2023/01/04 20:45:51 DEBUG : Test/tpb8mh3pae5u8mlk35jf4nhra0: Starting multipart upload
2023/01/04 20:45:52 DEBUG : Test/tpb8mh3pae5u8mlk35jf4nhra0: Uploading segment 0/32 size 32
2023/01/04 20:45:52 DEBUG : Test/test.txt: quickxor = 677e1984d8df14020d6540e4f538fd3dbeff12e9 OK
2023/01/04 20:45:52 INFO  : Test/test.txt: Copied (new)
2023/01/04 20:45:52 DEBUG : Waiting for deletions to finish
2023/01/04 20:45:52 INFO  : 
Transferred:   	         32 B / 32 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         5.6s

2023/01/04 20:45:52 DEBUG : 8 go routines active

Ole · January 5, 2023, 10:54am

Hi jared,

I suspect an issue with directory_name_encryption = false on backends where rmdir tEST will successfully remove Test, that is SharePoint, Windows,...

Are you able to reproduce if using a sync target something like this (under Windows):

[LocalTestCrypt]
type = crypt
remote = E:/Jared/TestCrypt
directory_name_encryption = false
password = notverysecret

Animosity022 · January 5, 2023, 12:35pm

Is that an issue with case insensitive remotes and just a bad order of operations? I don't have a remote like that to test on.

It seems it gets confused and the move works but the delete also happens after.

I'd imagine it's really easy to test that scenario as you'd have a folder and just adjust the name on a non case sensitive remote and it would be easy to replicate.

It is early for me so might not be awake yet either

ncw · January 5, 2023, 2:31pm

If you do rclone backend features ServerSharePointBackupnew: what does it say for the "CaseInsensitive" flag?

That should be false and in that case rclone should deal with changed cases of things, but I suspect it might be true from the symptoms.

Hmm, that effectively makes file names case sensitive and directory names case insensitive which is where the problem is coming from.

Rclone (I suspect) is marking the backend as case sensitive, but to mark it as case insensitive isn't right either as the file names will be case sensitive...

jared · January 5, 2023, 3:09pm

yes. the same thing happens locally in windows. here are both logs, showing the first sync after case change which result in the empty directory, and also the followup sync that shows rclone doing the sync again saying "Copied (new)"

2023/01/05 09:02:13 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "--verbose" "--progress" "E:\\test" "ProdCrypt:" "-vv" "--log-file=C:\\Users\\Administrator\\Desktop\\log.txt"]
2023/01/05 09:02:13 DEBUG : Creating backend with remote "E:\\test"
2023/01/05 09:02:13 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/01/05 09:02:13 DEBUG : fs cache: renaming cache item "E:\\test" to be canonical "//?/E:/test"
2023/01/05 09:02:13 DEBUG : Creating backend with remote "ProdCrypt:"
2023/01/05 09:02:13 DEBUG : Creating backend with remote "C:\\Users\\Administrator\\Desktop\\Case Test"
2023/01/05 09:02:13 DEBUG : fs cache: renaming cache item "C:\\Users\\Administrator\\Desktop\\Case Test" to be canonical "//?/C:/Users/Administrator/Desktop/Case Test"
2023/01/05 09:02:13 DEBUG : Encrypted drive 'ProdCrypt:': Waiting for checks to finish
2023/01/05 09:02:13 DEBUG : Encrypted drive 'ProdCrypt:': Waiting for transfers to finish
2023/01/05 09:02:13 DEBUG : Baghouse/IMG_1528.jpg: md5 = e218d45bad1c91d14acaa22709c8f1d9 OK
2023/01/05 09:02:13 INFO  : Baghouse/IMG_1528.jpg: Copied (new)
2023/01/05 09:02:13 DEBUG : Baghouse/IMG_1525.jpg: md5 = 0998282229d190c3fc717631afad9955 OK
2023/01/05 09:02:13 INFO  : Baghouse/IMG_1525.jpg: Copied (new)
2023/01/05 09:02:13 DEBUG : Baghouse/IMG_1526.jpg: md5 = eb1d31b2a3ab080f2aa68a498956986b OK
2023/01/05 09:02:13 INFO  : Baghouse/IMG_1526.jpg: Copied (new)
2023/01/05 09:02:13 DEBUG : Baghouse/IMG_1527.jpg: md5 = f1afafa1b4bb3e05d702352a9eed1f81 OK
2023/01/05 09:02:13 INFO  : Baghouse/IMG_1527.jpg: Copied (new)
2023/01/05 09:02:13 DEBUG : Baghouse/IMG_1530.jpg: md5 = 3760dee6ee57650f028c0bf29bb85158 OK
2023/01/05 09:02:13 INFO  : Baghouse/IMG_1530.jpg: Copied (new)
2023/01/05 09:02:13 DEBUG : Waiting for deletions to finish
2023/01/05 09:02:13 INFO  : bAGHOUSE/IMG_1526.jpg: Deleted
2023/01/05 09:02:13 INFO  : bAGHOUSE/IMG_1527.jpg: Deleted
2023/01/05 09:02:13 INFO  : bAGHOUSE/IMG_1528.jpg: Deleted
2023/01/05 09:02:13 INFO  : bAGHOUSE/IMG_1525.jpg: Deleted
2023/01/05 09:02:13 INFO  : bAGHOUSE/IMG_1530.jpg: Deleted
2023/01/05 09:02:13 INFO  : bAGHOUSE: Removing directory
2023/01/05 09:02:13 DEBUG : Encrypted drive 'ProdCrypt:': deleted 1 directories
2023/01/05 09:02:13 INFO  : 
Transferred:   	    7.261 MiB / 7.261 MiB, 100%, 0 B/s, ETA -
Checks:                 5 / 5, 100%
Deleted:                5 (files), 1 (dirs)
Transferred:            5 / 5, 100%
Elapsed time:         0.1s

2023/01/05 09:02:13 DEBUG : 3 go routines active



2023/01/05 09:02:36 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "--verbose" "--progress" "E:\\test" "ProdCrypt:" "-vv" "--log-file=C:\\Users\\Administrator\\Desktop\\log.txt"]
2023/01/05 09:02:36 DEBUG : Creating backend with remote "E:\\test"
2023/01/05 09:02:36 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2023/01/05 09:02:36 DEBUG : fs cache: renaming cache item "E:\\test" to be canonical "//?/E:/test"
2023/01/05 09:02:36 DEBUG : Creating backend with remote "ProdCrypt:"
2023/01/05 09:02:36 DEBUG : Creating backend with remote "C:\\Users\\Administrator\\Desktop\\Case Test"
2023/01/05 09:02:36 DEBUG : fs cache: renaming cache item "C:\\Users\\Administrator\\Desktop\\Case Test" to be canonical "//?/C:/Users/Administrator/Desktop/Case Test"
2023/01/05 09:02:36 DEBUG : Encrypted drive 'ProdCrypt:': Waiting for checks to finish
2023/01/05 09:02:36 DEBUG : Encrypted drive 'ProdCrypt:': Waiting for transfers to finish
2023/01/05 09:02:36 DEBUG : Baghouse/IMG_1526.jpg: md5 = 2659060b8eb4195e57e64bc04b141f70 OK
2023/01/05 09:02:36 INFO  : Baghouse/IMG_1526.jpg: Copied (new)
2023/01/05 09:02:36 DEBUG : Baghouse/IMG_1525.jpg: md5 = 5ff189e2db298f92b9a4a3af712aff50 OK
2023/01/05 09:02:36 INFO  : Baghouse/IMG_1525.jpg: Copied (new)
2023/01/05 09:02:36 DEBUG : Baghouse/IMG_1527.jpg: md5 = 2e95f8fec40436523e2c407731ae6523 OK
2023/01/05 09:02:36 INFO  : Baghouse/IMG_1527.jpg: Copied (new)
2023/01/05 09:02:36 DEBUG : Baghouse/IMG_1530.jpg: md5 = db0d7a6d88c9bb05e8caac1f62be04de OK
2023/01/05 09:02:36 INFO  : Baghouse/IMG_1530.jpg: Copied (new)
2023/01/05 09:02:36 DEBUG : Baghouse/IMG_1528.jpg: md5 = 353b1f9d19888a6cc81d18545d8d5acd OK
2023/01/05 09:02:36 INFO  : Baghouse/IMG_1528.jpg: Copied (new)
2023/01/05 09:02:36 DEBUG : Waiting for deletions to finish
2023/01/05 09:02:36 INFO  : 
Transferred:   	    7.261 MiB / 7.261 MiB, 100%, 0 B/s, ETA -
Transferred:            5 / 5, 100%
Elapsed time:         0.1s

2023/01/05 09:02:36 DEBUG : 3 go routines active

.
.

chuck norris doesn't sleep. he waits.....

.
.

it is in fact false

i'm glad you guys are the one handling this (a)

jared

p.s. when using the same situation with rclone copy for the local windows test, it copy the files over and over again saying "Copied (new)" each time. the directory case will then not be changed.

Ole · January 5, 2023, 4:55pm

Thanks!

Bug confirmed, here is a short PowerShell script to demonstrate the issue on Windows:

rclone config create winCrypt crypt directory_name_encryption=false remote="./testfoldercrypt" password=QsveQEXQq1rbMcrayfTUEuI
rclone touch ./testfolder/someFOLDER/somefile

rclone sync ./testfolder winCrypt: -v
rclone lsl winCrypt:
Rename-Item -Path ./testfolder/someFOLDER -NewName SOMEfolder
rclone sync ./testfolder winCrypt: -vv
rclone lsl winCrypt:
# The crypt should have contained somefile, but is now empty (data loss)

This is what I see:

PS > .\rclone\rclone backend features ./testfolder | Select-String Case
                "CaseInsensitive": true,
PS > .\rclone\rclone backend features winCrypt | Select-String Case     
                "CaseInsensitive": true,

@ncw I have the issue next to an rclone debugger on Windows in case there is anything you would like me to trace/debug. I doubt I can find/fix it myself.

ncw · January 5, 2023, 5:53pm

I got that backwards (as usual). If it was true then it would have worked properly.

Nice

I think (hope) you forgot the : on the end of the remote.

I'm not sure what to do here...

Setting "CaseInsensitive": true is easy enough and will fix the directories, but not the files....

Try commenting out this line in the crypt backend

		CaseInsensitive:         cipher.NameEncryptionMode() == NameEncryptionOff,

And I think that should fix the problem.

If it does I'll make a test binary with a proper fix. (I'd usually do that now, but I'm up to my elbows in a massive VFS refactor! )

Ole · January 5, 2023, 7:57pm

Sorry, you are right!

PS > .\rclone\rclone backend features winCrypt: | Select-String Case     
                "CaseInsensitive": false,

I have tried, but it makes no difference in the above test case and winCrypt: backendfeatures remains "CaseInsensitive": false. Tried saving + building twice, so pretty sure I did it right - but still human as proven above.

I can see the dilemma and how it is difficult to solve

ncw · January 6, 2023, 10:02am

Sorry, I think I told you to do the wrong thing.

Try, instead of commenting out the line, put CaseInsensitive: true - its a mask and if it is true, it should allow the underlying Fs value to be used.

I'd be interested to see if (provided that works as expected) this causes similar problems with files. If you make a file called hello.txt and rename it to HELLO.txt does that do the right thing? Can you still read the file.

Ole · January 9, 2023, 9:49am

I think I have got it, you are probably considering this fix to crypt.go:

  CaseInsensitive:  !cipher.dirNameEncrypt || cipher.NameEncryptionMode() == NameEncryptionOff,

which will then return true and fix the above sync issue:

> rclone config create winCrypt crypt directory_name_encryption=false remote="./testfoldercrypt" password=QsveQEXQq1rbMcrayfTUEuI
[winCrypt]
type = crypt
directory_name_encryption = false
remote = ./testfoldercrypt
password = *** ENCRYPTED ***
> rclone backend features winCrypt: | Select-String CaseInsensitive
                "CaseInsensitive": true,
> echo "hello WORLD" | rclone rcat ./testfolder/someFOLDER/someFILE.txt
> rclone sync ./testfolder winCrypt: -v
2023/01/09 09:36:49 INFO  : someFOLDER/someFILE.txt: Copied (new)
> rclone lsl winCrypt:
       13 2023-01-09 09:36:49.674261300 someFOLDER/someFILE.txt
> Rename-Item -Path ./testfolder/someFOLDER -NewName SOMEfolder
> rclone lsl ./testfolder
       13 2023-01-09 09:36:49.674261300 SOMEfolder/someFILE.txt
> rclone sync ./testfolder winCrypt: -v
2023/01/09 09:36:50 INFO  : There was nothing to transfer
> rclone lsl winCrypt:
       13 2023-01-09 09:36:49.674261300 someFOLDER/someFILE.txt

We do however (as we both expected) still have some interesting situations when accessing the file (content) with a different casing, e.g. the modified casing in the source folder:

PS> Rename-Item -Path ./testfolder/SOMEfolder/someFILE.txt -NewName SOMEfile.txt
PS> rclone lsl ./testfolder
13 2023-01-09 09:36:49.674261300 SOMEfolder/SOMEfile.txt
PS> rclone sync ./testfolder winCrypt: -v
2023/01/09 09:36:50 INFO : There was nothing to transfer
PS> rclone lsl winCrypt:
13 2023-01-09 09:36:49.674261300 someFOLDER/someFILE.txt
PS> rclone cat winCrypt:someFOLDER/someFILE.txt
hello WORLD
PS> rclone cat winCrypt:someFOLDER/SOMEfile.txt
2023/01/09 09:36:50 ERROR : : error listing: directory not found
2023/01/09 09:36:50 Failed to cat with 2 errors: last error was: directory not found
PS> rclone cat winCrypt:SOMEfolder/someFILE.txt
hello WORLD
PS> rclone cat winCrypt:SOMEfolder/SOMEfile.txt
2023/01/09 09:36:51 ERROR : : error listing: directory not found
2023/01/09 09:36:51 Failed to cat with 2 errors: last error was: directory not found

and the crypt allows behaviors (e.g. duplicate files) that I wouldn't expect on a CaseInsensitive filesystem:

PS> echo "hello world" | rclone rcat winCrypt:somefolder/somefile.txt
PS> rclone lsl winCrypt:
       13 2023-01-09 09:36:49.674261300 someFOLDER/someFILE.txt
       13 2023-01-09 09:36:51.380501900 someFOLDER/somefile.txt

These observations made me curios and I then tried a crypt with filename_encryption=off on Windows:

PS> rclone config create winCrypt2 crypt filename_encryption=off remote="./testfoldercrypt2" password=QsveQEXQq1rbMcrayfTUEuI
[winCrypt2]
type = crypt
password = *** ENCRYPTED ***
filename_encryption = off
remote = ./testfoldercrypt2
PS> rclone backend features winCrypt2: | Select-String CaseInsensitive
                "CaseInsensitive": true,
PS> rclone sync ./testfolder winCrypt2: -v
2023/01/09 09:36:52 INFO  : SOMEfolder/SOMEfile.txt: Copied (new)
PS> rclone lsl winCrypt2:
       13 2023-01-09 09:36:49.674261300 SOMEfolder/SOMEfile.txt
PS> rclone cat winCrypt2:someFOLDER/someFILE.txt
PS> rclone cat winCrypt2:someFOLDER/SOMEfile.txt
hello WORLD
PS> rclone cat winCrypt2:SOMEfolder/someFILE.txt
PS> rclone cat winCrypt2:SOMEfolder/SOMEfile.txt
hello WORLD
PS> echo "hello world" | rclone rcat winCrypt2:somefolder/somefile.txt
PS> rclone lsl winCrypt2:
       13 2023-01-09 09:36:52.905936300 SOMEfolder/SOMEfile.txt

This looks more like a CaseInsensitive filesystem, but I am surprised that 2 of the 4 cat's just returned empty contents without any errors - it looks like a bug. I haven't investigated further.

So to conclude:

I think the fix to CaseInsensitive is better than nothing and has a local branch ready to send a PR, but also think we should consider describing directory_name_encryption=false and filename_encryption=off as experimental on CaseInsensitive remotes until somebody has had the time to fully test and fix the above oddities.

ncw · January 10, 2023, 10:43am

Excellent work.

Yes I agree. Do you want to send a PR with the fix and the notes in the docs?

Ole · January 11, 2023, 12:52pm

Thanks!

Sure, PR ready for review:

github.com/rclone/rclone

crypt: fix for unencrypted directory names on case insensitive remotes

rclone:master ← olefrost:fix-crypt-directory-name-encryption

opened 12:47PM - 11 Jan 23 UTC

olefrost

+3 -1

#### What is the purpose of this change? rclone sync erroneously deleted fold…ers renamed to a different case on crypts where directory name encryption was disabled and the underlying remote was case insensitive. Example: Renaming the folder Test to tEST before a sync to a crypt having remote=OneDrive:crypt and directory_name_encryption=false could result in the folder and all its content being deleted. The following sync would correctly create the tEST folder and upload all of the content. Additional tests have revealed other potential issues when using filename_encryption=off or directory_name_encryption=false on case insensitive remotes. The documentation has been updated to warn about potential problems when using these combinations. #### Was the change discussed in an issue or in the forum before? https://forum.rclone.org/t/chaning-directory-case-makes-rclone-lose-it/35268 #### Checklist - [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix). - [x] I have added tests for all changes in this PR if appropriate. - [x] I have added documentation for the changes if appropriate. - [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [x] I'm done, this Pull Request is ready for review :-)

Ole · January 12, 2023, 8:30am

Hi @jared,

The fix is now available in the latest beta.

jared · January 12, 2023, 4:49pm

ok, so just to be clear, the only area where things are still a little hazy is when directory AND file name encryption are turned off?

Ole · January 12, 2023, 5:27pm

No sorry, there are also hazy areas with just directory_name_encryption=false after the fix.

I don't think you will notice/hit the hazy areas I found, if the crypt is solely used as target for backup syncs.

However the existence of such hazy areas indicates low test coverage and low usage to me, which typically means there are more undiscovered oddities and corner cases waiting to be discovered.

I don't know your dependency on the data, but suggest you do regular checks using cryptcheck and/or check --download.

These are the known hazy areas with directory_name_encryption=false after the fix:

Ole:

We do however (as we both expected) still have some interesting situations when accessing the file (content) with a different casing, e.g. the modified casing in the source folder:

PS> Rename-Item -Path ./testfolder/SOMEfolder/someFILE.txt -NewName SOMEfile.txt
PS> rclone lsl ./testfolder
13 2023-01-09 09:36:49.674261300 SOMEfolder/SOMEfile.txt
PS> rclone sync ./testfolder winCrypt: -v
2023/01/09 09:36:50 INFO : There was nothing to transfer
PS> rclone lsl winCrypt:
13 2023-01-09 09:36:49.674261300 someFOLDER/someFILE.txt
PS> rclone cat winCrypt:someFOLDER/someFILE.txt
hello WORLD
PS> rclone cat winCrypt:someFOLDER/SOMEfile.txt
2023/01/09 09:36:50 ERROR : : error listing: directory not found
2023/01/09 09:36:50 Failed to cat with 2 errors: last error was: directory not found
PS> rclone cat winCrypt:SOMEfolder/someFILE.txt
hello WORLD
PS> rclone cat winCrypt:SOMEfolder/SOMEfile.txt
2023/01/09 09:36:51 ERROR : : error listing: directory not found
2023/01/09 09:36:51 Failed to cat with 2 errors: last error was: directory not found

and the crypt allows behaviors (e.g. duplicate files) that I wouldn't expect on a CaseInsensitive filesystem:

PS> echo "hello world" | rclone rcat winCrypt:somefolder/somefile.txt
PS> rclone lsl winCrypt:
       13 2023-01-09 09:36:49.674261300 someFOLDER/someFILE.txt
       13 2023-01-09 09:36:51.380501900 someFOLDER/somefile.txt

jared · January 13, 2023, 2:33pm

wow. looks like i stumbled upon a real good one there. all thanks to some girl at the local school division renaming one of the folders in her private movie stash

i'd rather not meddle in this area if i can help it.

so, thanks for the help and for the good work and for the awesome tool. it's nice to know i'm partially responsible for some of it in some small way and yes, as per your advice, we do the occasional cryptcheck anyway.

,
jared

Ole · January 13, 2023, 2:55pm

You are welcome, happy we got it fixed!

You did an excellent job in spotting and reducing the issue to a fun and challenging task for me

Ole

ncw · January 13, 2023, 3:14pm

Yes you did. If there was a can of worms emoji I would stick one in here!

The root problem is that rclone expects file systems to be case insensitive or case sensitive, but your particular config creates a file system where the directories are case insensitive and the files are case sensitive (or the other way round depending), which confuses rclone!

Thanks to @Ole for making the doc fix.

jared · January 13, 2023, 3:31pm

i'd be happy to turn on directory name encryption and keep things simple, but i exceed the character length limit that sharepoint has. it's annoying, but so it goes with sharepoint

ncw · January 13, 2023, 3:42pm

You could try base32768 encoding for the names which was put in specifically for onedrive and should be shorter.