Error when copy to cloud InvalidArgumentException

Modesty · February 27, 2022, 10:32am

Hi

I get error and something strange is observed.
File u2028hus 1-4.pdf don't exist either in source or destination (jottacloud).

My command is:

rclone copy /volume1/xxxxxxx/ NAS:xxxxxx --log-file=Logxxxxxx.txt

Any tip I can get from you?

I have loged into jottacloud and my NAS and searched for file "u2028hus 1-4.pdf" but file don't exist anyplace.

Error:

2022/02/26 17:23:49 ERROR : Latvija/Price enquiry/2021.05.10 - B2 - 10 - Fjellsrude - Brannkonsult/Drawings/PDF/A40-02 Fasade Nordøst og Sørøst hus 1-4.pdf: Failed to copy: HTTP error 400 (400 Bad Request) returned body:"{"code":400,"message":"Unable to validate decoded path:/xxxxxx/Jotta/Archive/yyyyyyyyyy/Latvija/Price enquiry/2021.05.10 - B2 - 10 - Fjellsrude -Brannkonsult/Drawings/PDF/A40-02 Fasade Nordøst og Sørøst\u2028hus 1-4.pdf","cause":"","error_id":"InvalidArgumentException","x-id":"587318965507"}"

This is screenshot of server and mounted drive from Jottacloud
On server there is 5 more files, and they are not copied to cloud.

albertony · February 27, 2022, 11:46am

The filename contains a special character, shown simply as space in GUI, represented by Unicode sequence \u2028 in log. Did you upload it with rclone, or Jottacloud's official client, or some other method?

Modesty · February 27, 2022, 3:02pm

Hi,

I'm 99,9% sure I used rclone copy due to first time I run backup on my new NAS, i have not used JC client on this NAS
I could (0,1%) have copied from JC web, from old backup to new (new is in Archive area of JC)

M

ncw · February 27, 2022, 3:29pm

Unicode 0x2028 is a line separator apparently.

That is exactly the sort of character that cloud storage providers don't allow in their filenames as it can confuse stuff. I don't know whether that character is allowed or not though.

Modesty · February 28, 2022, 8:10am

I did a MD5 check sum and presented as ANSI Window char set

As you see Norwegian leter å is NOT the same even if it looks like it in explorer:

Under is same info as in picture, pasted as text, letter å is the same:

5f18d62812c92e3dda433e1909400f89 *Kirkeåsveien 20, B1 14.09.2021.doc
5f18d62812c92e3dda433e1909400f89 *Kirkeåsveien 20, B1 14.09.2021.doc

In this directory users with MAC and PC are saving, this spesial file i know is saved from a MAC.

Is there a way to sort out this types of "duplicates" and replace the one letter and save it like
Kirke(aa1)sveien 20, B1 14.09.2021.doc
Kirke(aa2)sveien 20, B1 14.09.2021.doc

where number is n=n+1?
Maybe a script or something

albertony · February 28, 2022, 9:09am

You could check out rclone dedupe and the --by-hash option. Not sure if it fits your need 100%, and I have not really been using it myself, but may be worth looking into..

ncw · February 28, 2022, 11:11am

Python agrees

>>> "Kirkeåsveien" == "Kirkeåsveien"
False
>>> "Kirkeåsveien".encode("utf-8"), "Kirkeåsveien".encode("utf-8")
(b'Kirkea\xcc\x8asveien', b'Kirke\xc3\xa5sveien')
>>>

If you want rclone to ignore these kind of differences then use the

  --no-unicode-normalization   Don't normalize unicode characters in filenames

And rclone will treat those files as two different files.

macOS is notorious for doing this. By default it uses an unnormalized version of UTF-8.

Modesty · March 1, 2022, 7:03am

Hi thanks for answer

I tried the command:

rclone copy /volume1/test/ NAS:test --no-unicode-normalization --progress --log-file=logtest4

on duplicates as discussed previously with UTF-8 mix:

Result:

No files copied

Content of logtest4:

Empty

Run without --no-unicode-normalization:

rclone copy /volume1/test/ NAS:test --progress --log-file=logtest5

Result:

No copy happened

Content of logtest5:

NOTICE: Strømmålerskjema.pdf: Duplicate object found in source - ignoring

Q1:
(What I dont get is why no file is copied with the switch --no-unicode-normalization? Is there an explanation?)
------> update, after a few minutes this file ended on Jottacloud.

Q2: Is there a way to get a backup of all fduplicates?:

I would like to copy all files of all duplicates in same directory, this will demand some code/script.

Preferably renaming on both source and destination and change all to correct unicode, maybe something like this:
The newest one with highest prefix like
Newest: Strømmålerskjema1.pdf
Older: Strømmålerskjema.pdf

Or as written before:
Is there a way to sort out this types of "duplicates" and replace, in source, the one letter and save it like
Strømm(aa1)lerskjema.pdf
Strømm(aa2)lerskjema.pdf

where number is n=n+1?

M

ncw · March 1, 2022, 12:24pm

I really need to see the log with -vv to understand what happened.

Modesty · March 1, 2022, 4:58pm

Hi @ncw

I edited the post, but I did not find all places, sorry. Here is what happened:

Why it took so long time? My jottacloud is mounted as a drive with rclone, probably something with refresh in Total Commander.

M

ncw · March 1, 2022, 7:41pm

Can you run your commands again with the -vv flag which will output debug information - that will help with what is going on.

Modesty · March 2, 2022, 7:01am

root@NAS:~# rclone copy /volume1/test/ NAS:test --no-unicode-normalization --progress --log-file=logtest9 -vv

logtest9:

Failed to create file system for "NAS:test": couldn't get customer info: Get >"https://api.jottacloud.com/account/v1/customer": dial tcp 185.179.130.26:443: i/o timeout

This is first error i have seen since I managed rclone to run on my NAS. This may have something to do with NAS is already running av rclone script to Jotta. I know that PC client has limitation of 6 parallell uploads.

M

ncw · March 2, 2022, 9:46am

Can I see the entire log?

Modesty · March 2, 2022, 10:41am

Hi

This was all in the logtest9, one line:

Failed to create file system for "NAS:test": couldn't get customer info: Get "https://api.jottacloud.com/account/v1/customer": dial tcp 185.179.130.26:443: i/o timeout

ncw · March 2, 2022, 12:56pm

That is a temporary networking error which will most likely be fine if you retry.

I'd like to see a full log for your actual problem.

Modesty · March 2, 2022, 5:50pm

Source:

root@NAS:~# rclone copy /volume1/test/ NAS:test --no-unicode-normalization --progress --log-file=logtest9 -vv

Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Transferred: 3 / 3, 100%
Elapsed time: 2.5s

root@NAS:~# cat logtest9
2022/03/02 07:55:35 Failed to create file system for "NAS:test": couldn't get customer info: Get "https://api.jottacloud.com/account/v1/customer": dial tcp 185.179.130.26:443: i/o timeout
2022/03/02 18:42:49 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "/volume1/test/" "NAS:test" "--no-unicode-normalization" "--progress" "--log-file=logtest9" "-vv"]
2022/03/02 18:42:49 DEBUG : Creating backend with remote "/volume1/test/"
2022/03/02 18:42:49 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/03/02 18:42:49 DEBUG : Creating backend with remote "NAS:test"
2022/03/02 18:42:49 DEBUG : NAS: Loaded invalid token from config file - ignoring
2022/03/02 18:42:50 DEBUG : Saving config "token" in section "NAS" of the config file
2022/03/02 18:42:50 DEBUG : NAS: Saved new token in config file
2022/03/02 18:42:51 DEBUG : jottacloud root 'test': Waiting for checks to finish
2022/03/02 18:42:51 DEBUG : jottacloud root 'test': Waiting for transfers to finish
2022/03/02 18:42:51 DEBUG : test.md5: md5 = 8e4ca6ffd1a7c9d1c8ad5defc0cbc53e OK
2022/03/02 18:42:51 INFO : test.md5: Copied (new)
2022/03/02 18:42:51 DEBUG : Strømmålerskjema.pdf: md5 = 75969133bfa7091c8e9917e6e0ec5e54 OK
2022/03/02 18:42:51 INFO : Strømmålerskjema.pdf: Copied (new)
2022/03/02 18:42:51 DEBUG : Strømmålerskjema.pdf: md5 = 75969133bfa7091c8e9917e6e0ec5e54 OK
2022/03/02 18:42:51 INFO : Strømmålerskjema.pdf: Copied (new)
2022/03/02 18:42:51 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Transferred: 3 / 3, 100%
Elapsed time: 2.5s

2022/03/02 18:42:51 DEBUG : 12 go routines active
root@NAS:~#

In my h: drive wich is mounted with rclone I see this 30 sek after copy, empty:

In explorer

On jotta web:

Now i waited ca 2 min and result is:

Source
md5 ANSI

Detination

1 file is sorted out

The one that is copied is 1:

ncw · March 3, 2022, 10:06am

Here you can see rclone copied both files.

However only 1 file seemed to arrive.

I did some local tests and I think Jottacloud is doing unicode normalisation on filenames so its impossible to have two filenames with the same unicode normalisation

I uploaded a file with one normalisation

$ echo hello | rclone rcat TestJottacloud:test2/Strømmålerskjema.txt

>>> "Strømmålerskjema.txt".encode("utf-8")
b'Str\xc3\xb8mma\xcc\x8alerskjema.txt'

Then listed it

$ rclone lsf TestJottacloud:test2
Strømmålerskjema.txt

And that is the normalised version

>>> "Strømmålerskjema.txt".encode("utf-8")
b'Str\xc3\xb8mm\xc3\xa5lerskjema.txt'

So I don't think jottacloud can store both those files with those filenames.

Modesty · March 4, 2022, 6:31am

Thanks for clarification @ncw

Now I need to make a script that renames these files on source before backup to jottacloud.

M

Ole · March 4, 2022, 11:04am

You might find inspiration in this thread: Duplicate object found in source - ignoring (unique name) - #12 by Ole

Modesty · March 6, 2022, 7:17am

To fix this duplicate issue I have done some testing on my files.

I have tried to find a way to automatic delete duplicates in a way that I keep the one with the letter marked under

When sort the files in Totall Commander by name the one I like to delete is the first, like this

Where I stranded is how to do this search by a script, Totall Commander don't have a function to delete one of two.

So now I probably need to make a scrip that:

Reads all files in each directory
If duplicates
How many duplicates
Delete the correct one

This is out of my know how, I even don't know where to start.

Can you point me to a place where I can start?
Is there a "script" community somewhere?

M