How can I migrate to different filename encoding with server side copy?

comicfans · January 4, 2023, 11:01am

What is the problem you are having with rclone?

I tried to backup my files to encrypted onedrive with default filename encrypt/encoding, but lots of file failed due to 'pathTooLong'

so I try to use rclone to migrate existing backup into other filename encoding (or disable filename/dir encrypt)

then I found rclone docs

server_side_across_configs

This can be used, for example, to change file name encryption type without re-uploading all the data. Just make two crypt backends pointing to two different directories with the single changed parameter and use rclone move to move the files between the crypt remotes.

so now I have 3 config:

bare onedrive config, all default config, no advanced options edited
encrypt config1, all default config (with my own password1/password2), no advanced options edited (I've already upload lots of data into this config, but some path too long file failed)
encrypt config2, all default config (with same password1/password2 to config1), change dir crypt to false
(I want to use rclone server copy ability to copy files from config1 into this config2, with dir crypt disabled, so all of my backup migrate to new config without re-uploading, then I can continue to upload failed ones into this config)

then I try to use command:

export RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS=true
rclone -vv copyto config1:/file config2:/file

rclone shows

Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
Starting multipart upload

seems it didn't use server side copy, I'd like to know if this is possible ?

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
os/kernel: 10.0.19044.2364 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.19.4
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

encrypt over personal onedrive

The rclone config contents with secrets removed.

[bare]
type = onedrive
token = ***
drive_id = ***
drive_type = personal

[encrypt]
type = crypt
remote = bare:dir1
password = ***
password2 = ***

[goodbackup888_fn]
type = crypt
remote = bare:dir2
password = ***
password2 = ***
directory_name_encryption = false

asdffdsa · January 4, 2023, 1:39pm

hello,
please post a full debug log, where you copy a single file.

i would use copy, not copyto
rclone copy config1:file config2: -vv

comicfans · January 5, 2023, 12:54am

firstly, I tried two config with exactly same config (both has dir encryption turned on), even with this exactly same config, rclone still not using server side copy. this is log for such run

D:\asj\rclone>rclone copy config1:/git-log.txt config2:/ -vv
2023/01/05 08:49:37 DEBUG : Setting default for crypt-server-side-across-configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:49:37 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "config1:/git-log.txt" "config2:/" "-vv"]
2023/01/05 08:49:37 DEBUG : Creating backend with remote "config1:/git-log.txt"
2023/01/05 08:49:37 DEBUG : Using config file from "C:\\Users\\comicfans.DESKTOP-35AEQ6R\\.config\\rclone\\rclone.conf"
2023/01/05 08:49:37 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:49:37 DEBUG : config1: detected overridden config - adding "{Db_Y9}" suffix to name
2023/01/05 08:49:37 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:49:37 DEBUG : Creating backend with remote "bare:dir1/0a1lv6uu4en1n1nc7hnd7805sk"
2023/01/05 08:49:40 DEBUG : fs cache: adding new entry for parent of "bare:dir1/0a1lv6uu4en1n1nc7hnd7805sk", "bare:dir1"
2023/01/05 08:49:40 DEBUG : fs cache: adding new entry for parent of "config1:/git-log.txt", "config1{Db_Y9}:/git-log.txt"
2023/01/05 08:49:40 DEBUG : Creating backend with remote "config2:/"
2023/01/05 08:49:40 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:49:40 DEBUG : config2: detected overridden config - adding "{Db_Y9}" suffix to name
2023/01/05 08:49:40 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:49:41 DEBUG : Creating backend with remote "bare:dir2"
2023/01/05 08:49:42 DEBUG : fs cache: renaming cache item "config2:/" to be canonical "config2{Db_Y9}:/"
2023/01/05 08:49:43 DEBUG : git-log.txt: Need to transfer - File not found at Destination
2023/01/05 08:49:44 DEBUG : 0a1lv6uu4en1n1nc7hnd7805sk: Starting multipart upload
2023/01/05 08:49:45 DEBUG : 0a1lv6uu4en1n1nc7hnd7805sk: Uploading segment 0/4370 size 4370
2023/01/05 08:49:45 DEBUG : git-log.txt: sha1 = 034806b5196b2424ffc974043fc5ad7aba20bde1 OK
2023/01/05 08:49:45 INFO  : git-log.txt: Copied (new)
2023/01/05 08:49:45 INFO  :
Transferred:        4.268 KiB / 4.268 KiB, 100%, 2.125 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         8.6s

then I tried to only change 1 parameter (as rclone doc said), makes config2 has dir encryption turned off. since I'm coping files under root dir, thus such config shouldn't change anything. I delete target file from config2, rerun rclone copy command

2023/01/05 08:59:55 DEBUG : Setting default for crypt-server-side-across-configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:59:55 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "config1:/git-log.txt" "config2:/" "-vv"]
2023/01/05 08:59:55 DEBUG : Creating backend with remote "config1:/git-log.txt"
2023/01/05 08:59:55 DEBUG : Using config file from "C:\\Users\\comicfans.DESKTOP-35AEQ6R\\.config\\rclone\\rclone.conf"
2023/01/05 08:59:55 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:59:55 DEBUG : config1: detected overridden config - adding "{Db_Y9}" suffix to name
2023/01/05 08:59:55 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:59:55 DEBUG : Creating backend with remote "bare:dir1/0a1lv6uu4en1n1nc7hnd7805sk"
2023/01/05 08:59:58 DEBUG : fs cache: adding new entry for parent of "bare:dir1/0a1lv6uu4en1n1nc7hnd7805sk", "bare:dir1"
2023/01/05 08:59:58 DEBUG : fs cache: adding new entry for parent of "config1:/git-log.txt", "config1{Db_Y9}:/git-log.txt"
2023/01/05 08:59:58 DEBUG : Creating backend with remote "config2:/"
2023/01/05 08:59:58 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:59:58 DEBUG : config2: detected overridden config - adding "{Db_Y9}" suffix to name
2023/01/05 08:59:58 DEBUG : Setting crypt_server_side_across_configs="true" from environment variable RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
2023/01/05 08:59:58 DEBUG : Creating backend with remote "bare:dir2"
2023/01/05 09:00:00 DEBUG : fs cache: renaming cache item "config2:/" to be canonical "config2{Db_Y9}:/"
2023/01/05 09:00:01 DEBUG : git-log.txt: Need to transfer - File not found at Destination
2023/01/05 09:00:02 DEBUG : 0a1lv6uu4en1n1nc7hnd7805sk: Starting multipart upload
2023/01/05 09:00:02 DEBUG : 0a1lv6uu4en1n1nc7hnd7805sk: Uploading segment 0/4370 size 4370
2023/01/05 09:00:03 DEBUG : git-log.txt: sha1 = f8b38254e37d92ee6fef928e25a482f4aa9f05b3 OK
2023/01/05 09:00:03 INFO  : git-log.txt: Copied (new)
2023/01/05 09:00:03 INFO  :
Transferred:        4.268 KiB / 4.268 KiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         8.2s

still not using server side copy

asdffdsa · January 5, 2023, 1:10am

not sure, perhaps try
--onedrive-server-side-across-configs
--server-side-across-configs

comicfans · January 5, 2023, 1:21am

neither works, no matter only 1 argument provided , nor both arguments provided .

asdffdsa · January 5, 2023, 1:25am

the docs make mention of using rclone move.
use rclone move to move the files between the crypt remotes

if not, then we have some good debug info for others to comment.

comicfans · January 5, 2023, 1:41am

You're right, move command do the trick. it shows

2023/01/05 09:35:22 DEBUG : git-log.txt: Need to transfer - File not found at Destination
2023/01/05 09:35:23 INFO  : git-log.txt: Moved (server-side)

I still not narrow down which arguments (onedrive-server-side-across-config/server-side-across-config/crypt-server-side-across-config) makes it work, but at least currently specified all these arguments makes server side move work.

but it still seems wired, move can using server side copy, but copy can not ? onedrive does not support server side copy ? or is this a rclone logic restriction ?

comicfans · January 5, 2023, 1:42am

And thank you very much for your help

asdffdsa · January 5, 2023, 1:44am

sure,

perhaps try all three flags with rclone copy

system · January 8, 2023, 1:44am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.