What is the problem you are having with rclone?
In rclone copy, when the file name and/or folder name contains the character % followed by something that can be considered a hex digit (for example, %e3), the results of the copy are different from what is expected.
When the file contains the %?? string, the corresponding uploaded filename has a converted string.
For example, if we have "test%e3%46232.txt", the final name is "test�F232.txt" where � probably corresponds to the character ã and the %46 recognized as the character F:
https://grox.net/utils/encoding.html
If the rclone copy is used on a specified file (instead of a folder to folder copy), then the --onedrive-no-versions switch behaves as expected, generationg no new versions and just performing checks upon repeated commands.
If the rclone copy is used on a folder that contains the file with the %?? string in the filename, the checker does not recognize the identical file and overwrites, creating a new version. When rclone cleanup is performed on the specific file (with the source file name), the removal of previous versions works as intended.
If the rclone copy is used on a folder with a folder name containing the %?? string, the folder is not created in the destination, and the files within the source folder is copied to the destination folder, effectively shifting the directory tree up by one.
Most likely this is an effect of the change in 1.54.0 that allowed the use of % and # in OneDrive encoding. I have not extensively checked the other combinations, but use of # in filename looked ok. My guess is that %?? where the ?? can be treated as hex is a special case?
Thank you for your help!
What is your rclone version (output from rclone version)
rclone v1.54.0
- os/arch: windows/amd64
- go version: go1.15.7
(I also tested with 1.54.1 and can confirm the same results)
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 7 or Windows 10, 64 bit, Shell used is Command Prompt.
Which cloud storage system are you using? (eg Google Drive)
Microsoft OneDrive for Business / SharePoint
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy "<redacted>\test%e3%46232.txt" "sharepoint:Collaborations" --onedrive-no-versions -vv --ignore-size --ignore-checksum --fast-list --use-mmap -u
rclone copy "<redacted>\test" "sharepoint:Collaborations" --onedrive-no-versions -vv --ignore-size --ignore-checksum --fast-list --use-mmap -u
(in the second example, "test%e3%46232.txt" was moved into a folder called "test".
The rclone config contents with secrets removed.
[sharepoint]
type = onedrive
token = {"access_token":"...","token_type":"Bearer","refresh_token":"...","expiry":"2021-03-13T03:49:43.1040845-06:00"}
drive_id = ...
drive_type = documentLibrary
client_id = ...
client_secret = ...
A log from the command with the -vv flag
C:\usr\Programs\rclone>rclone copy "<redacted>\test%e3%46232.txt" "sharepoint:Collaborations" --onedrive-no-versions -vv
2021/03/13 02:51:16 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "<redacted>\test%e3%46232.txt" "sharepoint:Collaborations" "--onedrive-no-versions" "-vv"]
2021/03/13 02:51:16 DEBUG : Creating backend with remote "<redacted>\test%e3%46232.txt"
2021/03/13 02:51:16 DEBUG : Using config file from "<redacted>"
2021/03/13 02:51:16 DEBUG : fs cache: adding new entry for parent of "<redacted>\test%e3%46232.txt", "<redacted>"
2021/03/13 02:51:16 DEBUG : Creating backend with remote "sharepoint:Collaborations"
2021/03/13 02:51:17 DEBUG : test%e3%46232.txt: Need to transfer - File not found at Destination
2021/03/13 02:51:17 DEBUG : test%e3%46232.txt: Starting multipart upload
2021/03/13 02:51:17 DEBUG : test%e3%46232.txt: Uploading segment 0/4 size 4
2021/03/13 02:51:18 DEBUG : test%e3%46232.txt: QuickXorHash = 7428c31ce8000000000000000400000000000000 OK
2021/03/13 02:51:18 INFO : test%e3%46232.txt: Copied (new)
2021/03/13 02:51:18 INFO :
Transferred: 4 / 4 Bytes, 100%, 3 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 2.3s
2021/03/13 02:51:18 DEBUG : 6 go routines active
C:\usr\Programs\rclone>rclone copy "<redacted>\test%e3%46232.txt" "sharepoint:Collaborations" --onedrive-no-versions -vv
2021/03/13 02:51:19 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "<redacted>\test%e3%46232.txt" "sharepoint:Collaborations" "--onedrive-no-versions" "-vv"]
2021/03/13 02:51:19 DEBUG : Creating backend with remote "<redacted>\test%e3%46232.txt"
2021/03/13 02:51:19 DEBUG : Using config file from "<redacted>"
2021/03/13 02:51:19 DEBUG : fs cache: adding new entry for parent of "<redacted>\test%e3%46232.txt", "<redacted>"
2021/03/13 02:51:19 DEBUG : Creating backend with remote "sharepoint:Collaborations"
2021/03/13 02:51:20 DEBUG : test%e3%46232.txt: Size and modification time the same (differ by -505.1033ms, within tolerance 1s)
2021/03/13 02:51:20 DEBUG : test%e3%46232.txt: Unchanged skipping
2021/03/13 02:51:20 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 1 / 1, 100%
Elapsed time: 0.9s
2021/03/13 02:51:20 DEBUG : 3 go routines active
C:\usr\Programs\rclone>rclone copy "<redacted>\test%e3%46232.txt" "sharepoint:Collaborations" --onedrive-no-versions -vv --ignore-size --ignore-checksum --fast-list --use-mmap
2021/03/13 02:56:51 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "<redacted>\test%e3%46232.txt" "sharepoint:Collaborations" "--onedrive-no-versions" "-vv" "--ignore-size" "--ignore-checksum" "--fast-list" "--use-mmap"]
2021/03/13 02:56:51 DEBUG : Creating backend with remote "<redacted>\test%e3%46232.txt"
2021/03/13 02:56:51 DEBUG : Using config file from "<redacted>"
2021/03/13 02:56:51 DEBUG : fs cache: adding new entry for parent of "<redacted>\test%e3%46232.txt", "<redacted>"
2021/03/13 02:56:51 DEBUG : Creating backend with remote "sharepoint:Collaborations"
2021/03/13 02:56:52 DEBUG : test%e3%46232.txt: Size and modification time the same (differ by -505.1033ms, within tolerance 1s)
2021/03/13 02:56:52 DEBUG : test%e3%46232.txt: Unchanged skipping
2021/03/13 02:56:52 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 1 / 1, 100%
Elapsed time: 0.9s
2021/03/13 02:56:52 DEBUG : 3 go routines active
C:\usr\Programs\rclone>rclone copy "<redacted>\test%e3" "sharepoint:Collaborations" --onedrive-no-versions -vv --ignore-size --ignore-checksum --fast-list --use-mmap -u
2021/03/13 02:58:31 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "<redacted>\test%e3" "sharepoint:Collaborations" "--onedrive-no-versions" "-vv" "--ignore-size" "--ignore-checksum" "--fast-list" "--use-mmap" "-u"]
2021/03/13 02:58:31 DEBUG : Creating backend with remote "<redacted>\test%e3"
2021/03/13 02:58:31 DEBUG : Using config file from "<redacted>"
2021/03/13 02:58:31 DEBUG : fs cache: renaming cache item "<redacted>\test%e3" to be canonical "<redacted>/test%e3"
2021/03/13 02:58:31 DEBUG : Creating backend with remote "sharepoint:Collaborations"
2021/03/13 02:58:32 DEBUG : One drive root 'Collaborations': Waiting for checks to finish
2021/03/13 02:58:32 DEBUG : One drive root 'Collaborations': Waiting for transfers to finish
2021/03/13 02:58:32 DEBUG : test%e3%46232.txt: Starting multipart upload
2021/03/13 02:58:32 DEBUG : test%e3%46232.txt: Uploading segment 0/4 size 4
2021/03/13 02:58:33 INFO : test%e3%46232.txt: Copied (new)
2021/03/13 02:58:33 INFO :
Transferred: 4 / 4 Bytes, 100%, 3 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.9s
2021/03/13 02:58:33 DEBUG : 6 go routines active
C:\usr\Programs\rclone>rclone copy "<redacted>\test%e3" "sharepoint:Collaborations" --onedrive-no-versions -vv --ignore-size --ignore-checksum --fast-list --use-mmap -u
2021/03/13 02:58:35 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "<redacted>\test%e3" "sharepoint:Collaborations" "--onedrive-no-versions" "-vv" "--ignore-size" "--ignore-checksum" "--fast-list" "--use-mmap" "-u"]
2021/03/13 02:58:35 DEBUG : Creating backend with remote "<redacted>\test%e3"
2021/03/13 02:58:35 DEBUG : Using config file from "<redacted>"
2021/03/13 02:58:35 DEBUG : fs cache: renaming cache item "<redacted>\test%e3" to be canonical "<redacted>/test%e3"
2021/03/13 02:58:35 DEBUG : Creating backend with remote "sharepoint:Collaborations"
2021/03/13 02:58:35 DEBUG : One drive root 'Collaborations': Waiting for checks to finish
2021/03/13 02:58:35 DEBUG : One drive root 'Collaborations': Waiting for transfers to finish
2021/03/13 02:58:35 DEBUG : test%e3%46232.txt: Starting multipart upload
2021/03/13 02:58:36 DEBUG : test%e3%46232.txt: Uploading segment 0/4 size 4
2021/03/13 02:58:36 INFO : test%e3%46232.txt: Copied (new)
2021/03/13 02:58:36 INFO :
Transferred: 4 / 4 Bytes, 100%, 3 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.8s
2021/03/13 02:58:36 DEBUG : 6 go routines active
C:\usr\Programs\rclone>rclone copy "<redacted>\test" "sharepoint:Collaborations" --onedrive-no-versions -vv --ignore-size --ignore-checksum --fast-list --use-mmap -u
2021/03/13 02:59:00 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "<redacted>\test" "sharepoint:Collaborations" "--onedrive-no-versions" "-vv" "--ignore-size" "--ignore-checksum" "--fast-list" "--use-mmap" "-u"]
2021/03/13 02:59:00 DEBUG : Creating backend with remote "<redacted>\test"
2021/03/13 02:59:00 DEBUG : Using config file from "<redacted>"
2021/03/13 02:59:00 DEBUG : fs cache: renaming cache item "<redacted>\test" to be canonical "<redacted>/test"
2021/03/13 02:59:00 DEBUG : Creating backend with remote "sharepoint:Collaborations"
2021/03/13 02:59:01 DEBUG : One drive root 'Collaborations': Waiting for checks to finish
2021/03/13 02:59:01 DEBUG : One drive root 'Collaborations': Waiting for transfers to finish
2021/03/13 02:59:01 DEBUG : test%e3%46232.txt: Starting multipart upload
2021/03/13 02:59:01 DEBUG : test%e3%46232.txt: Uploading segment 0/4 size 4
2021/03/13 02:59:02 INFO : test%e3%46232.txt: Copied (new)
2021/03/13 02:59:02 INFO :
Transferred: 4 / 4 Bytes, 100%, 3 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 2.0s
2021/03/13 02:59:02 DEBUG : 6 go routines active
C:\usr\Programs\rclone>rclone copy "<redacted>\test" "sharepoint:Collaborations" --onedrive-no-versions -vv --ignore-size --ignore-checksum --fast-list --use-mmap -u
2021/03/13 02:59:03 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "<redacted>\test" "sharepoint:Collaborations" "--onedrive-no-versions" "-vv" "--ignore-size" "--ignore-checksum" "--fast-list" "--use-mmap" "-u"]
2021/03/13 02:59:03 DEBUG : Creating backend with remote "<redacted>\test"
2021/03/13 02:59:03 DEBUG : Using config file from "<redacted>"
2021/03/13 02:59:03 DEBUG : fs cache: renaming cache item "<redacted>\test" to be canonical "<redacted>/test"
2021/03/13 02:59:03 DEBUG : Creating backend with remote "sharepoint:Collaborations"
2021/03/13 02:59:04 DEBUG : One drive root 'Collaborations': Waiting for checks to finish
2021/03/13 02:59:04 DEBUG : One drive root 'Collaborations': Waiting for transfers to finish
2021/03/13 02:59:04 DEBUG : test%e3%46232.txt: Starting multipart upload
2021/03/13 02:59:04 DEBUG : test%e3%46232.txt: Uploading segment 0/4 size 4
2021/03/13 02:59:05 INFO : test%e3%46232.txt: Copied (new)
2021/03/13 02:59:05 INFO :
Transferred: 4 / 4 Bytes, 100%, 3 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.8s
2021/03/13 02:59:05 DEBUG : 6 go routines active
This output created a file "test�F232.txt" and when I tried creating a folder called "test%46", it did not create a folder called "testF" but rather copied the contents of it into the folder called "Collaborations" in the destination.
