Copy file from Dropbox to Storj_S3 - without full download on HDD. Possible?

[Anze (Anže)]

Hello,

When i copy 500GB file from Dropbox to Storj, it gets file first fully downloaded inside rclone cache folder and then after is download completed starts uploading on Storj-S3.

My questions are:

• It's this a normal behavior?
• It's not possible to copy file from one remote to other and cache only current small chunks on local HDD?

Rclone version:
rclone v1.62.2

• os/version: ubuntu 22.04 (64 bit)
• os/kernel: 5.15.0-27-generic (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.20.2
• go/linking: static
• go/tags: none

Rclone.conf file:
[storj]
type = s3
provider = Storj
access_key_id = xxxxxxxx
secret_access_key = xxxxxxxx
endpoint = gateway.storjshare.io

[dropbox]
type = dropbox
client_id = xxxxxxxx
client_secret = xxxxxxxx
token = {"access_token":"xxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxx","expiry":"xxxxxxxx"}

[dropbox-crypt]
type = crypt
remote = dropbox:/Data Folder/
password = xxxxxxxx
password2 = xxxxxxxx
no_data_encryption = true

Mount commands:

• rclone mount storj:/bucket/ /mnt/storj/ --cache-dir /mnt/cache/ --vfs-cache-max-age 10m --vfs-cache-mode writes --vfs-cache-max-size 256M --s3-chunk-size 64M --s3-disable-checksum & disown
• rclone mount dropbox-crypt: /mnt/dropbox/ --cache-dir /mnt/cache/ --vfs-cache-max-age 10m --vfs-cache-mode full --vfs-cache-max-size 64M --vfs-read-chunk-size 64M --tpslimit 12 & disown

Copy command:
rclone copy /mnt/dropbox/testfile /mnt/storj/

Please tell me if i did something wrong. I was thinking that is file fully downloaded because of crypt option - but i see same "first download, then upload" even if i mount only dropbox without crypt and copy un-encrypted file.

Thank you!

[Animosity022]

Mounts are not needed and add waste in the process.

Just copy from remote to remote without the mount.

rclone copy storj:somefile dropbox-crypt:

Use --dry-run and make sure it's going where you want.

[Anze (Anže)]

Thank you. I didn't know that is rclone able to copy without mount - that will optimize many process

[minesheep]

Blockquote Mount commands:
rclone mount storj:/bucket/ /mnt/storj/ --cache-dir /mnt/cache/ --vfs-cache-max-age 10m --vfs-cache-mode writes --vfs-cache-max-size 256M --s3-chunk-size 64M --s3-disable-checksum & disown
rclone mount dropbox-crypt: /mnt/dropbox/ --cache-dir /mnt/cache/ --vfs-cache-max-age 10m --vfs-cache-mode full --vfs-cache-max-size 64M --vfs-read-chunk-size 64M --tpslimit 12 & disown

You used --vfs-cache-mode full and writes, these flags means rclone will cache files copied to mount location locally and upload later. If you remove --vfs-cache-mode full and --vfs-cache-mode writes rclone should download and upload in chunks. However uploading directly with rclone copy source: destination: has less overhead than using rclone mount command and then copying files between mounts

Edit: not 100% true see Animosity022 message below

[Animosity022]

@minesheep

these flags means rclone will cache files copied to mount location locally

Not exactly.

Full means you can still seek files and download chunks as needed. You only download a full file if the application reads the whole file.

Writes mean it'll write the whole file locally and upload later.