The upload fails when the rclone copy command transfers a specific file

What is the problem you are having with rclone?

The upload fails when the rclone copy command transfers a specific file.

What is your rclone version (output from rclone version)

rclone v1.55.1

  • os/type: windows
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: dynamic
  • go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy  F:\upload-ready2 dropbox-crypt:test --checkers 16 --fast-list --progress --stats 5s --stats-file-name-length 20 --stats-log-level DEBUG --transfers 4 -vv --crypt-show-mapping

The rclone config contents with secrets removed.

[dropbox]
type = dropbox
token = [Hidden]
client_id = [Hidden]
client_secret = [Hidden]

[dropbox-crypt]
type = crypt
remote = dropbox:rclone/crypt
filename_encryption = obfuscate
directory_name_encryption = true
password = iP7Hg5C6[Hidden]
password2 = 0XVeMW[Hidden]

A log from the command with the -vv flag

2021/04/30 18:40:40 DEBUG : Using config file from "C:\\rclone-windows-amd64\\rclone.conf"
2021/04/30 18:40:41 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "F:\\upload-ready2" "dropbox-crypt:test" "--checkers" "16" "--fast-list" "--progress" "--stats" "5s" "--stats-file-name-length" "20" "--stats-log-level" "DEBUG" "--transfers" "4" "-vv" "--crypt-show-mapping"]
2021/04/30 18:40:41 DEBUG : Creating backend with remote "F:\\upload-ready2"
2021/04/30 18:40:41 DEBUG : fs cache: renaming cache item "F:\\upload-ready2" to be canonical "//?/F:/upload-ready2"
2021/04/30 18:40:41 DEBUG : Creating backend with remote "dropbox-crypt:test"
2021/04/30 18:40:41 DEBUG : dropbox-crypt: detected overridden config - adding "{p07Ar}" suffix to name
2021/04/30 18:40:41 DEBUG : Creating backend with remote "dropbox:rclone/crypt/192.LwKL"
2021/04/30 18:40:41 DEBUG : fs cache: renaming cache item "dropbox-crypt:test" to be canonical "dropbox-crypt{p07Ar}:test"
2021-04-30 18:40:42 DEBUG : Encrypted drive 'dropbox-crypt{p07Ar}:test': Waiting for checks to finish
2021-04-30 18:40:42 DEBUG : Encrypted drive 'dropbox-crypt{p07Ar}:test': Waiting for transfers to finish
2021-04-30 18:40:43 ERROR : Data/Picture/카메라/흔한 반도의 집.JPG: Failed to copy: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
2021-04-30 18:40:43 ERROR : Attempt 1/3 failed with 1 errors and: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
2021-04-30 18:40:43 DEBUG : Encrypted drive 'dropbox-crypt{p07Ar}:test': Waiting for checks to finish
2021-04-30 18:40:43 DEBUG : Encrypted drive 'dropbox-crypt{p07Ar}:test': Waiting for transfers to finish
2021-04-30 18:40:44 ERROR : Data/Picture/카메라/흔한 반도의 집.JPG: Failed to copy: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
2021-04-30 18:40:44 ERROR : Attempt 2/3 failed with 1 errors and: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
2021-04-30 18:40:44 DEBUG : Encrypted drive 'dropbox-crypt{p07Ar}:test': Waiting for checks to finish
2021-04-30 18:40:44 DEBUG : Encrypted drive 'dropbox-crypt{p07Ar}:test': Waiting for transfers to finish
2021-04-30 18:40:46 ERROR : Data/Picture/카메라/흔한 반도의 집.JPG: Failed to copy: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
2021-04-30 18:40:46 ERROR : Attempt 3/3 failed with 1 errors and: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
Transferred:        3.745M / 3.745 MBytes, 100%, 1.057 MBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         5.7s
2021/04/30 18:40:46 DEBUG :
Transferred:        3.745M / 3.745 MBytes, 100%, 1.057 MBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         5.7s

2021/04/30 18:40:46 DEBUG : 5 go routines active
2021/04/30 18:40:46 Failed to copy: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH

There seems to be a problem with the file name after obfuscating it.

invalid:
흔한 반도의 집.JPG
흔한 반도의 집(1).JPG

valid:
a흔한 반도의 집.JPG
흔한 반도의 집a.JPG
흔한 반도의 집1.JPG
흔한 반도의 집(.JPG
흔한 반도의 집(1.JPG
흔한 반도의 집).JPG

Can you find out what that files obfuscated name is?

rclone backend encode dropbox-crypt: "Data/Picture/카메라/흔한 반도의 집.JPG"

If you could do a few of them, maybe we can work out the problem.

Calling @sweh also!

I don't know what character set is being used, here; all I see on my screen are squares, which makes me think it's using a font characterset not on my machine.

What OS is this from, and what locale settings?

The following command was executed at the command prompt :

rclone backend encode dropbox-crypt: "Data/Picture/카메라/흔한 반도의 집.JPG" -vv --log-file=rclone.log

Contents of rclone.log :

2021/05/01 07:33:20 DEBUG : Using config file from "C:\rclone-windows-amd64\rclone.conf"
2021/05/01 07:33:20 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "backend" "encode" "dropbox-crypt:" "Data/Picture/카메라/흔한 반도의 집.JPG" "-vv" "--log-file=rclone.log"]
2021/05/01 07:33:20 DEBUG : Creating backend with remote "dropbox:rclone/crypt"
2021/05/01 07:33:21 DEBUG : 3 go routines active

Output of CMD :

122.axQx/220.kDxOPMz/68.캆멦랎/4.힥햭 뱩댕잩 줢.OUL

The following is the Unicode code for the output :
4.힥햭 뱩댕잩 줢.OUL
U+0034 : DIGIT FOUR
U+002E : FULL STOP {period, dot, decimal point}
U+D7A5 : reserved
U+D5AD : HANGUL SYLLABLE HYAEG
U+0020 : SPACE [SP]
U+BC69 : HANGUL SYLLABLE BYAEG
U+B315 : HANGUL SYLLABLE DAENG
U+C7A9 : HANGUL SYLLABLE JAT
U+0020 : SPACE [SP]
U+C922 : HANGUL SYLLABLE JWEOLM
=> U+D7A5 appears to be invalid characters?

I am using Windows 10 1909 as a Korean locale. The encoding is probably "cp949".

U+D7A4 to F were reserved on the Hangul Syllables, one of the Unicode blocks.

Hmm, in theory there's a test to stop "invalid characters" from appearing

                case runeValue >= 0x100:
                        // Some random Unicode range; we have no good rules here
                        thisdir := (dir % 127) + 1
                        base := int(runeValue - runeValue%256)
                        newRune := rune(base + (int(runeValue)-base+thisdir)%256)
                        // If the new character isn't a valid UTF8 char
                        // then don't rotate it.  Quote it instead
                        if !utf8.ValidRune(newRune) {
                                _, _ = result.WriteRune(obfuscQuoteRune)
                                _, _ = result.WriteRune(runeValue)
                        } else {
                                _, _ = result.WriteRune(newRune)
                        }

So it looks like the Go library isn't flagging it as bad. I'm not sure what we can do about that...

UPDATE: [U+D7A5].txt file is successfully uploaded to the Google drive.

I think it's valid, but it can't only be used in Dropbox.

U+D7A5 seems to be a character that can not be used as a file name in the Dropbox. I have just uploaded a text file under the name [U+D7A5].txt in the web Dropbox, and an error occurred that there was an incorrect character in the file name.

Hmm. I'm not sure what we can do about that. Unless @ncw has any clever ideas!

Yes I agree that is the problem

$ echo hello | rclone rcat "dropbox:힥"
2021/05/01 14:01:44 ERROR : 힥: Failed to copy: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
2021/05/01 14:01:44 ERROR : 힥: Post request rcat error: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH
2021/05/01 14:01:44 Failed to rcat: upload failed: Error in call to API function "files/upload": Invalid path: INVALID_PATH

Where that character is

>>> hex(ord("힥"))
'0xd7a5'

According to fileformat this isn't a valid unicode character. However go (see playground) does believe it to be valid.

However the check is quite simple

// ValidRune reports whether r can be legally encoded as UTF-8.
// Code points that are out of range or a surrogate half are illegal.
func ValidRune(r rune) bool {
	switch {
	case 0 <= r && r < surrogateMin:
		return true
	case surrogateMax < r && r <= MaxRune:
		return true
	}
	return false
}

Maybe we should be using something a bit more sophisticated like unicode.IsPrint, eg

https://play.golang.org/p/6p6b5KzKCQa

However I have a little concern here - what is valid and invalid in unicode (not utf8) can and does change beween unicode revisions. Does this mean that the encoding/decoding of files is therefore dependent on unicode version?

Potentially, yes. The obfuscated results could change depending on GoLang version!

Also, it doesn't handle the case where GoLang may support a newer version than the cloud provider, so we'd be back where we started; GoLang thinks it's good but the provider thinks it's bad.

Indeed.

I'm suprised at Dropbox - I would have thought they would go for a forward compatible file nameing scheme, much more like the utf.IsValid check.

@QxXU8Yknm2 I can't think of a good solution to this... You could try name encryption istead of obfuscation? Dropbox can only have 255 char file names so this may not be the best solution.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.