Problem with long file names syncing to encrypted Koofr folder

What is the problem you are having with rclone?

Error when syncing files with long file names (> 138 characters) to encrypted folder on Koofr, or when archiving them using --backu-dir

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

  • os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
  • os/kernel: 10.0.22635.3066 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.22.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Koofr (encrypted Vault = rclone crypt)

The command you were trying to run (eg rclone copy /tmp remote:tmp)


:: Archivesync v1.3 by Stigma, credit to asdfdsa(Jojo) for pseudocode assistance, debugging and emotional support ;)
@echo off

::Before we start, get a file-lock on this script to prevent concurrent instances
call :getLock
exit /b

:getLock
:: The CALL will fail if another process already has a write lock on the script
call :main 9>>"%~f0"
exit /b

:main
:: Body of script goes here. Only one process can ever get here

:: --------------SETTINGS START ----------------

:: set this to the folder where rclone.exe is located on your system
set "rclonepath=C:\rclone"

:: Set this to the folder (or driveletter) you want to protect with backup and archive
set "sourcepath=D:\Eigene Dokumente"

:: Set this the the folder (usually on a remote) you want to save the backup and archive to (does not necessarily have to be an rclone remote)
set "destpath=Koofr3-Rogers-Vault:\BACKUP\Computer2Stay\Eigene Dokumente"

:: Set this to the folder (usually on a remote) which will contain old "deleted" or "overwritten" revisions of files. I suggest keeping it next to your backup folder but it could be put anywhere.
set "archivepath=Koofr3-Rogers-Vault:\BACKUP\Computer2Stay\oldfiles\Eigene Dokumente"

:: Set this path to where you want the logfile to be made.
set "logfilepath=C:\rclone\logfiles"

:: Set the detail of logging - from least verbose to most: ERROR or NOTICE or INFO or DEBUG (default, NOTICE, is usually sufficient)
:: see documentaion for more info : https://rclone.org/docs/#log-level-level
set "loglevel=INFO"

:: Set any other non-essential flags you want rclone to use. Can leave as empty set "flags=" if you want none
set "flags=--progress --drive-chunk-size 64M --retries 1 --exclude=/Outlook-Dateien/** --exclude=/_gsdata_/**"

::------------------SETTINGS END------------------
::----------------MAIN SCRIPT START --------------

::Various timestamp formatting in separate function (you can change timestamp there if you want).
call :FORMATTIME

::Make the logfile directory if it doesn't already exist
if not exist "%logfilepath%\" mkdir "%logfilepath%"

echo Synchronisierung "Eigene Dokumente" nach Koofr gestartet ...
echo:

:: Now let us sync. This makes a mirror of sourcepath to destpath (including removing files if required), and any files that get "overwritten" or "deleted" as a
:: result from destpath, will be moved into archive and and timestamped instead - effectively creating a full archive of all revisions of files you have ever had.
%rclonepath%\rclone sync "%sourcepath%" "%destpath%" %flags% --backup-dir="%archivepath%" --suffix=_DEL%datesuffix% --suffix-keep-extension --log-file="%logfilepath%\%date%.log" --log-level=%loglevel%
echo:

::If exit code of above command was anything but normal, display an error and pause
if not %ERRORLEVEL% equ 0 (
	echo FEHLER! rclone hat eine Error-Meldung ausgegeben. Weitere Infos siehe logfile.
	pause
	exit /b 1
) else (
	echo Syncronisierung erfolgreich abgeschlossen!
	pause
	exit /b 0
)

::----------------MAIN SCRIPT END -----------------
::--------------HELPER FUNCTONS START--------------
:FORMATTIME

for /f "usebackq skip=1 tokens=1-6" %%g in (`wmic Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year ^| findstr /r /v "^$"`) do (
    set day=00%%g
    set hours=00%%h
    set minutes=00%%i
    set month=00%%j
    set seconds=00%%k
    set year=%%l
	)
set month=%month:~-2%
set day=%day:~-2%
set hh=%hours:~-2%
set mm=%minutes:~-2%
set ss=%seconds:~-2%

:: This can be easily modified to your liking if you prefer another timestamp format (for archive and logs) - credit to asdfdsa(Jojo)
set date=%year%-%month%-%day%_%hh%h%mm%m
set datesuffix=%year%%month%%day%T%hh%%mm%
	
exit /b 0
::----------------HELPER FUNCTONS END--------------

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[Koofr2-Roger]
type = koofr
provider = koofr
user = XXX
password = XXX

[Koofr3-Rogers-Vault]
type = crypt
remote = Koofr2-Roger:/Rogers Vault
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

2024/06/25 12:57:37 DEBUG : rclone: Version "v1.67.0" starting with parameters ["C:\\rclone\\rclone" "sync" "D:\\Eigene Dokumente" "Koofr3-Rogers-Vault:\\BACKUP\\Computer2Stay\\Eigene Dokumente" "--progress" "--drive-chunk-size" "64M" "--retries" "1" "--exclude=/Outlook-Dateien/**" "--exclude=/_gsdata_/**" "--backup-dir=Koofr3-Rogers-Vault:\\BACKUP\\Computer2Stay\\oldfiles\\Eigene Dokumente" "--suffix=_DEL20240625T1257" "--suffix-keep-extension" "--log-file=C:\\rclone\\logfiles\\2024-06-25_12h57m.log" "--log-level=DEBUG"]
2024/06/25 12:57:37 DEBUG : Creating backend with remote "D:\\Eigene Dokumente"
2024/06/25 12:57:37 DEBUG : Using config file from "C:\\rclone\\rclone.conf"
2024/06/25 12:57:37 DEBUG : fs cache: renaming cache item "D:\\Eigene Dokumente" to be canonical "//?/D:/Eigene Dokumente"
2024/06/25 12:57:37 DEBUG : Creating backend with remote "Koofr3-Rogers-Vault:\\BACKUP\\Computer2Stay\\Eigene Dokumente"
2024/06/25 12:57:37 DEBUG : Creating backend with remote "Koofr2-Roger:/Rogers Vault/fdk07po4vguu3ug0hogfdl0p90/na2oahc2ub6mcert1g14659uno/n4pdqtdgiqpr7dpm20vt8462t02bngmtbab3bg8mhiqlhbuab8ng"
2024/06/25 12:57:38 DEBUG : fs cache: renaming cache item "Koofr3-Rogers-Vault:\\BACKUP\\Computer2Stay\\Eigene Dokumente" to be canonical "Koofr3-Rogers-Vault:/BACKUP/Computer2Stay/Eigene Dokumente"
2024/06/25 12:57:38 DEBUG : Creating backend with remote "Koofr3-Rogers-Vault:\\BACKUP\\Computer2Stay\\oldfiles\\Eigene Dokumente"
2024/06/25 12:57:38 DEBUG : Creating backend with remote "Koofr2-Roger:/Rogers Vault/fdk07po4vguu3ug0hogfdl0p90/na2oahc2ub6mcert1g14659uno/ddtmtqrl9mclqnc5rhklad86jg/n4pdqtdgiqpr7dpm20vt8462t02bngmtbab3bg8mhiqlhbuab8ng"
2024/06/25 12:57:38 DEBUG : fs cache: renaming cache item "Koofr3-Rogers-Vault:\\BACKUP\\Computer2Stay\\oldfiles\\Eigene Dokumente" to be canonical "Koofr3-Rogers-Vault:/BACKUP/Computer2Stay/oldfiles/Eigene Dokumente"
2024/06/25 12:57:38 DEBUG : Outlook-Dateien: Excluded
2024/06/25 12:57:38 DEBUG : _gsdata_: Excluded
2024/06/25 12:57:38 DEBUG : _gsdata_: Excluded
2024/06/25 12:57:38 DEBUG : Outlook-Dateien: Excluded
2024/06/25 12:57:38 DEBUG : desktop.ini: Size and modification time the same (differ by -177.1µs, within tolerance 1ms)
2024/06/25 12:57:38 DEBUG : desktop.ini: Unchanged skipping
2024/06/25 12:57:38 DEBUG : 123abc/138_characters_123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123.docx: Sizes differ (src 12214 vs dst 12150)
2024/06/25 12:57:38 DEBUG : 123abc/138_characters_123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123.docx: Can't move, switching to copy
2024/06/25 12:57:39 ERROR : 123abc/138_characters_123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123.docx: Failed to copy: Invalid response status! Got 400, expected [200]; headers: map[Cache-Control:[no-cache, no-store, must-revalidate] Content-Length:[0] Content-Type:[text/plain] Date:[Tue, 25 Jun 2024 12:57:39 GMT] Expires:[0] Pragma:[no-cache] X-Request-Id:[f6e767c3-55e0-4480-8692-0d87203d813b] X-User-Id:[7bd2a49b-c6fd-475a-8cac-16b27c764beb]], content: 
2024/06/25 12:57:39 ERROR : 123abc/138_characters_123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123.docx: Not deleting source as copy failed: Invalid response status! Got 400, expected [200]; headers: map[Cache-Control:[no-cache, no-store, must-revalidate] Content-Length:[0] Content-Type:[text/plain] Date:[Tue, 25 Jun 2024 12:57:39 GMT] Expires:[0] Pragma:[no-cache] X-Request-Id:[f6e767c3-55e0-4480-8692-0d87203d813b] X-User-Id:[7bd2a49b-c6fd-475a-8cac-16b27c764beb]], content: 
2024/06/25 12:59:21 DEBUG : Encrypted drive 'Koofr3-Rogers-Vault:/BACKUP/Computer2Stay/Eigene Dokumente': Waiting for checks to finish
2024/06/25 12:59:21 DEBUG : Encrypted drive 'Koofr3-Rogers-Vault:/BACKUP/Computer2Stay/Eigene Dokumente': Waiting for transfers to finish
2024/06/25 12:59:21 ERROR : Encrypted drive 'Koofr3-Rogers-Vault:/BACKUP/Computer2Stay/Eigene Dokumente': not deleting files as there were IO errors
2024/06/25 12:59:21 ERROR : Encrypted drive 'Koofr3-Rogers-Vault:/BACKUP/Computer2Stay/Eigene Dokumente': not deleting directories as there were IO errors
2024/06/25 12:59:21 ERROR : Attempt 1/1 failed with 1 errors and: Invalid response status! Got 400, expected [200]; headers: map[Cache-Control:[no-cache, no-store, must-revalidate] Content-Length:[0] Content-Type:[text/plain] Date:[Tue, 25 Jun 2024 12:57:39 GMT] Expires:[0] Pragma:[no-cache] X-Request-Id:[f6e767c3-55e0-4480-8692-0d87203d813b] X-User-Id:[7bd2a49b-c6fd-475a-8cac-16b27c764beb]], content: 
2024/06/25 12:59:21 INFO  : 
Transferred:   	   11.912 KiB / 11.912 KiB, 100%, 2 B/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:             56160 / 56160, 100%
Elapsed time:      1m44.2s

2024/06/25 12:59:21 DEBUG : 4 go routines active
2024/06/25 12:59:21 Failed to sync: Invalid response status! Got 400, expected [200]; headers: map[Cache-Control:[no-cache, no-store, must-revalidate] Content-Length:[0] Content-Type:[text/plain] Date:[Tue, 25 Jun 2024 12:57:39 GMT] Expires:[0] Pragma:[no-cache] X-Request-Id:[f6e767c3-55e0-4480-8692-0d87203d813b] X-User-Id:[7bd2a49b-c6fd-475a-8cac-16b27c764beb]], content: 
 

I deleted the tenthousands of DEBUG lines about other files from the log file of course.

I modified the versioning backup script, which thestigma has posted in this forum. In his original script, every time when files were moved to the backup directory a new folder with date and time was created. I modified the script in a way, that no subfolders are created, but the original file structure of the source ist being remained. Instead I used the suffix flag to add an extension with date and time to the archived files, so that I have all former versions next to each other in the same folder.
This works fine exept when the file names are too long.
I tested a bit and found out, that the maximum file name length for an encrypted folder on Koofr is about 138 characters for a .docx file, maybe even shorter when the file name extension is longer.
My questions:
1.) Is there are way to modify the script, so that filenames that are loger than 130 characters will be automatically shortened to the first 130 characters during upload?
2.) Is there a way to change the script, so that filenames in the Koofr remote, that are longer than 120 characters will automatically be shortened to 120 characters, before the suffix is added and they are moved to the archive folder?

I won't comment about your script as I am not familiar with Windows but you could check if maybe more efficient encrypted names encoding can be used (it would allow longer files' names).

Koofr is case insensitive so base64 is not an option but you can check if base32768 would work, run:

rclone test info --check-length --check-base32768 Koofr2-Roger:test

and post results here

Thank you for your answer. I tried to run the script before with --crypt-filename-encoding base32768 already but it didn´t solve the problem.
I just executed the command you posted for the unencrypted Koofr2-Roger and the encrypted Koor3-Rogers-Vault with the following results:

rclone test info --check-length --check-base32768 Koofr2-Roger:test
2024/06/25 14:02:56 NOTICE: koofr:34aff57d-4bfb-4e4e-ab48-637ae79fcf0b:test/rclone-test-info-macunar8/test-base32768: 0 differences found
2024/06/25 14:02:56 NOTICE: koofr:34aff57d-4bfb-4e4e-ab48-637ae79fcf0b:test/rclone-test-info-macunar8/test-base32768: 1028 matching files
// Koofr2-Roger
maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 255 // for 2 byte unicode characters
maxFileLength = 255 // for 3 byte unicode characters
maxFileLength = -1 // for 4 byte unicode characters
base32768isOK = true // make sure maxFileLength for 2 byte unicode chars is the same as for 1 byte characters

rclone test info --check-length --check-base32768 Koofr3-Rogers-Vault:test
2024/06/25 14:05:36 EME operates on 1 to 128 block-cipher blocks, you passed 513
2024/06/25 14:05:36 EME operates on 1 to 128 block-cipher blocks, you passed 257
2024/06/25 14:05:36 EME operates on 1 to 128 block-cipher blocks, you passed 129
2024/06/25 14:05:43 EME operates on 1 to 128 block-cipher blocks, you passed 1025
2024/06/25 14:05:43 EME operates on 1 to 128 block-cipher blocks, you passed 513
2024/06/25 14:05:43 EME operates on 1 to 128 block-cipher blocks, you passed 257
2024/06/25 14:05:43 EME operates on 1 to 128 block-cipher blocks, you passed 129
2024/06/25 14:05:52 EME operates on 1 to 128 block-cipher blocks, you passed 1537
2024/06/25 14:05:52 EME operates on 1 to 128 block-cipher blocks, you passed 769
2024/06/25 14:05:52 EME operates on 1 to 128 block-cipher blocks, you passed 385
2024/06/25 14:05:52 EME operates on 1 to 128 block-cipher blocks, you passed 193
2024/06/25 14:06:01 EME operates on 1 to 128 block-cipher blocks, you passed 2049
2024/06/25 14:06:01 EME operates on 1 to 128 block-cipher blocks, you passed 1025
2024/06/25 14:06:01 EME operates on 1 to 128 block-cipher blocks, you passed 513
2024/06/25 14:06:01 EME operates on 1 to 128 block-cipher blocks, you passed 257
2024/06/25 14:06:01 EME operates on 1 to 128 block-cipher blocks, you passed 129
2024/06/25 14:10:36 NOTICE: Encrypted drive 'Koofr3-Rogers-Vault:test/rclone-test-info-xukubus1/test-base32768': 0 differences found
2024/06/25 14:10:36 NOTICE: Encrypted drive 'Koofr3-Rogers-Vault:test/rclone-test-info-xukubus1/test-base32768': 1028 hashes could not be checked
2024/06/25 14:10:36 NOTICE: Encrypted drive 'Koofr3-Rogers-Vault:test/rclone-test-info-xukubus1/test-base32768': 1028 matching files
// Koofr3-Rogers-Vault
maxFileLength = 143 // for 1 byte unicode characters
maxFileLength = 71 // for 2 byte unicode characters
maxFileLength = 47 // for 3 byte unicode characters
maxFileLength = 35 // for 4 byte unicode characters
base32768isOK = true // make sure maxFileLength for 2 byte unicode chars is the same as for 1 byte characters

It looks good.

It should allow you to store much longer filenames.

I would switch to base32768 encoding.

Again I have no idea about your script. But this encoding will let you store filenames at least 2 times longer I think. With default base32 you can only encode 160 bytes but with base32768 478 bytes.

But Koofr2-Roger ist the unencrypted remote.
I have the problem with the encrypted remote Koofr3-Rogers-Vault.
For this your command gives a maximum length of 143 files, if I understand it right. This would be identical to my tests (about 138 characters plus .docx)

Default base32 allows to encode only 5 bits per character. base32768 allows 15.

And you just tested:

maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 255 // for 2 byte unicode characters

that max number of characters is the same for 1 or 2 bytes unicode.

Try yourself.

Create new crypt in config file:

[Koofr3-Rogers-Vault-32768]
type = crypt
remote = Koofr2-Roger:test32768
filename_encoding = base32768
password = XXX
password2 = XXX

and test.

Create first test32768 directory in Koofr2-Roger remote

Hi, thank you for your advice. You were right, encrypt with base32768 filename encoding is accepting longer filenames. But when I use this, it isn´t compatible with Koofr´s Vault-feature. Koofr offers a feature to access an encrypted folder by internet browser or with their Vault-App for Android. This feature uses rclone encryption and therefore is compatible with rclone. But as far as I have just tested it is working only with the default encryption settings. I think I have to decide to either shorten my file names or give up using the Vault-Features of Koofr.
Or maybe somene else has an idea how I can automatically shorten long filenames with the script?

Or talk to whoever provides this feature to make encoding flexible. It should be very easy to implement.

I will do. Thank you. Have a nice day.

@kapitainsky I contacted Koofr, and they told me, that their cloud only supports base32 file name encryption, because it is not case sensitive or UTF-16. I experimented a little bit with a crypt remote with base32768 file name encryption, and in fact I got some errors when up- and downloading files.
It seems that the rclone command --check-base32768 isn´t really reliable?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.