Syncing folders with special characters to ntfs partition

[mhermann]

Hi,

First of all, I'd like to say thank you to all the devs who worked on rclone to make this tool as awesome as it is. Your work is much appreciated!

What is the problem you are having with rclone?

I'm trying to sync files and folders from a linux partition to an ntfs-formatted partition on the same machine. Some of the folders contain special characters (specifically a colon). The sync fails at creating the folders containing the special characters on the ntfs destination.

I've read on multiple sites, that rclone replaces invalid characters automatically, which doesn't seem to happen. Is there any way I can make the sync work without renaming the source folders?

I've tried the --local-unicode-normalization, which seems to help with files, but not folder names. Is this a limitation/bug?

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

• os/version: debian 11.6 (64 bit)
• os/kernel: 5.10.0-21-amd64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.19.4
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

None. Syncing files locally.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

mkdir -p test/test:test/ && touch test/test:test/test.txt
rclone sync test /path/on/ntfs/partition

The rclone config contents with secrets removed.

not relevant

A log from the command with the -vv flag

DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "-vv" "test" "/path/on/ntfs/partition/test"]
DEBUG : Creating backend with remote "test"
DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
DEBUG : fs cache: renaming cache item "test" to be canonical "/home/user/test"
DEBUG : Creating backend with remote "/path/on/ntfs/partition/test"
DEBUG : Local file system at /path/on/ntfs/partition/test: Waiting for checks to finish
DEBUG : Local file system at /path/on/ntfs/partition/test: Waiting for transfers to finish
ERROR : test:test/test.txt: Failed to copy: mkdir /path/on/ntfs/partition/test/test:test: invalid argument
ERROR : Local file system at /path/on/ntfs/partition/test: not deleting files as there were IO errors
ERROR : Local file system at /path/on/ntfs/partition/test: not deleting directories as there were IO errors
ERROR : Attempt 1/3 failed with 1 errors and: mkdir /path/on/ntfs/partition/test/test:test: invalid argument
DEBUG : Local file system at /path/on/ntfs/partition/test: Waiting for checks to finish
DEBUG : Local file system at /path/on/ntfs/partition/test: Waiting for transfers to finish
ERROR : test:test/test.txt: Failed to copy: mkdir /path/on/ntfs/partition/test/test:test: invalid argument
ERROR : Local file system at /path/on/ntfs/partition/test: not deleting files as there were IO errors
ERROR : Local file system at /path/on/ntfs/partition/test: not deleting directories as there were IO errors
ERROR : Attempt 2/3 failed with 1 errors and: mkdir /path/on/ntfs/partition/test/test:test: invalid argument
DEBUG : Local file system at /path/on/ntfs/partition/test: Waiting for checks to finish
DEBUG : Local file system at /path/on/ntfs/partition/test: Waiting for transfers to finish
ERROR : test:test/test.txt: Failed to copy: mkdir /path/on/ntfs/partition/test/test:test: invalid argument
ERROR : Local file system at /path/on/ntfs/partition/test: not deleting files as there were IO errors
ERROR : Local file system at /path/on/ntfs/partition/test: not deleting directories as there were IO errors
ERROR : Attempt 3/3 failed with 1 errors and: mkdir /path/on/ntfs/partition/test/test:test: invalid argument
INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s

DEBUG : 3 go routines active
Failed to sync: mkdir /path/on/ntfs/partition/test/test:test: invalid argument

[Animosity022]

I don't think rclone understands that's a ntfs partition mounted underneath.

There's a whole section on trying to get around that.

Local Filesystem as you want to mess with the local encoding I'd imagine and replace it.

[albertony]

I agree with @Animosity022: I think rclone uses a single encoding for the local backend, OS-dependent but not filesystem dependent. The encoding can be configured, as indicated, see also background info about encoding in Overview of cloud storage systems. In addition: To copy from one local remote to another local remote, to be able to configure different encoding on source and destination, one would need to use connection string syntax.

You can try the following:

First, run the following to show your default encoding:

rclone help flags local-encoding

Now use the default encoding shown by that command as a starting point, and add/remove identifiers on source and/or destination to fit your need. E.g. given a default encoding of Slash,Dot, if you want to copy files from source as is but with additional encoding of colon on your destination:

rclone sync test ':local,encoding="Slash,Dot,Colon":/path/on/ntfs/partition/' --dry-run

Remove the --dry-run to actually do the sync, when you have verified it does what you want.

Edit: Added quoting to the sync command, which was necessary to make it work on linux, as discovered by @mhermann and mentioned in next post.

[mhermann]

Ah, thank you. That worked. I had to use quotes in two places though:

rclone sync test ':local,encoding="Slash,Dot,Colon":/path/on/ntfs/partition/'

The sync back from ntfs file system to Linux works without issues too.

[albertony]

Great!

@mhermann

I had to use quotes in two places though

Aha, thank you for reporting back!