Pre-download move/sync when server does not support streaming upload

[jiru]

What is the problem you are having with rclone?

I'm migrating data from one remote to another. The dest remote does not support streaming uploads, so the default rclone action is to download the file then begin the upload. This is fine, but more often than not, files are only downloading then they upload, then repeat. It does not begin downloading the next file until the current file is done. Since my internet speed is better than the capabilities of both remotes combined, I was hoping for a flag that would allow me to pre-download up to a certain local cache size, so there can be a constant upload stream, thus cutting down on the migration time.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1
- os/version: debian 11.3 (64 bit)
- os/kernel: 5.15.102-1-pve (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

src: Google Drive (crypt)
dst: Jottacloud (union/crypt)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move src: dest: \
  --tpslimit 10 \
  --tpslimit-burst 10 \
  --exclude-from filter.txt \
  -vv \
  --checkers 10 \
  --transfers 4

filter.txt is just a list of folders i want to exclude, very simple, like /pictures/**

The rclone config contents with secrets removed.

src remote

[gdrive]
type = drive
client_id = myid
client_secret = mysecret
scope = drive
root_folder_id = rootid
token = {"access_token":"tokenhere","expiry":"2023-08-25T17:47:26.71093284Z"}

[src]
type = crypt
remote = gdrive:
filename_encryption = standard
directory_name_encryption = true
password = pass1
password2 = pass2

dest remote

[jotta01]
type = jottacloud
configVersion = 1
client_id = jottacli
client_secret =
tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
token = {"access_token":"tokenhere","expiry":"2023-08-25T18:12:27.34645542Z"}
device =
mountpoint =

[jotta02]
type = jottacloud
configVersion = 1
client_id = jottacli
client_secret =
tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
token = {"access_token":"tokenhere","expiry":"2023-08-25T18:11:24.073190584Z"}
device =
mountpoint =

[union]
type = union
upstreams = jotta01:vega jotta02:vega
create_policy = lus
search_policy = epall


[dest]
type = crypt
remote = vega-connect:
password = isqjC2jsy3Uf7R5h5y-qXmOvRNF9824aps40JjGCEEXOEsfTVqoPeB8RxBJ933oD
password2 = F1vMRUmo3SkVPgg2027G50Jm5IMAAnuBNjAkVvko-wExuwFj6vSr9ym4eO3irrG1
filename_encoding = base32

A log from the command with the -vv flag

Not applicable, just an observation of the timing.

Edits:
One approach was to first move the data locally (i have enough local buffer to do so) and start a move operation after a few minutes from local to the dest. But the src move create partial files. I could create a filter to exclude these file endings and avoid the use of --fast-list so the move from local to dest always finds new files to move. Thoughts on this approach, if a built-in flag does not exist?

[jiru]

I think I found a good solution, using this answer by ncw as inspiration.

I mounted the remote with appropriate settings, such as setting vfs mode to full and setting the max age to a large value. Equally important, I set the max cache size to the size of my disk.

Once mounted, in another session window, I use
rclone hashsum crc32 /path/to/mount --exclude-from filter.txt

This outputs the hash. The files are downloaded to the cache as a side effect. By supplying the same filter/excludes file i use in my final move command, only the files I want to move are cached and primed.

Last step is to start the move, using the local mount as the src.

The only thing im not sure how to configure is the order for the hashsum is not the same as the order of the move operation. So for a short while, the hashsum could be caching some files and the the move is working on other files. But at some point, im going to get the value of the cached files.

@ncw any recommendations on how to make sure im caching and moving the files in the same order to get the most effective use of it? @Animosity022 you too. You're pretty good with manipulating files and orders.