Best way to transfer to Backblaze B2 from mergerfs Remotes

What is the problem you are having with rclone?

Hey there! I've finally moved on to a more stable platform since the last time I was here (Now running x64 rclone 1.62.2 on an Ubuntu 22.04 Privileged LXC container for Proxmox 8), and have been enjoying it inmensely both with the Union remote and now with mergerfs, but recently it seems I've been running out of space quickly and I'd like to move to Backblaze B2 as a long term solution for my CryptFS remote.

Thing is, most of my information has been saved to several Consumer OneDrive accounts and a single Personal Google Drive, so I'd like to know which would be the fastest way to transfer about 1.7TB of information out of these remotes and into Backblaze.

So far, I've been able to transfer about 700GB with a 1000Mbps down/250Mbps up connection, but I'd like to make sure everything can be double-checked with the --checksum flag at the end and find a way to either pause or restart the transfer whenever one of the remotes gets rate-limited.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.2.16-3-pve (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive Personal, Google Drive Personal and Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sudo rclone mount ToasterDEV-OneDrive:/Union /mnt/cloud/ToasterDEV-OneDrive --config=/opt/rclone/rclone.conf --allow-other=true --attr-timeout=8700h --poll-interval=30s --umask=000 --user-agent=unionfs --syslog --checkers=24 --human-readable --low-level-retries=20 --retries=20 --retries-sleep=5s --metadata --multi-thread-streams=1024 -vv --fast-list --transfers=24 &

sudo rclone mount ToasterDEV-GDrive:/ /mnt/cloud/ToasterDEV-GDrive --config=/opt/rclone/rclone.conf --allow-other=true --attr-timeout=8700h --poll-interval=30s --umask=000 --user-agent=unionfs --syslog --checkers=24 --human-readable --low-level-retries=20 --retries=20 --retries-sleep=5s --metadata --multi-thread-streams=1024 -vv --fast-list --transfers=24 &

sudo rclone mount Lizita63-OneDrive:/Union /mnt/cloud/Lizita63-OneDrive --config=/opt/rclone/rclone.conf --allow-other=true --attr-timeout=8700h --poll-interval=30s --umask=000 --user-agent=unionfs --syslog --checkers=24 --human-readable --low-level-retries=20 --retries=20 --retries-sleep=5s --metadata --multi-thread-streams=1024 -vv --fast-list --transfers=24 &

sudo rclone mount Flanch1942-OneDrive:/Union /mnt/cloud/Flanch1942-OneDrive --config=/opt/rclone/rclone.conf --allow-other=true --attr-timeout=8700h --poll-interval=30s --umask=000 --user-agent=unionfs --syslog --checkers=24 --human-readable --low-level-retries=20 --retries=24 --retries-sleep=5s --metadata --multi-thread-streams=1024 -vv --fast-list --transfers=24 &

sudo rclone --config="/opt/rclone/rclone.conf" sync Local:/mnt/mergerfs Backblaze:/RCloneEncFS -vv --fast-list --checkers=100 --transfers=100 --multi-thread-streams=1024 --syslog --rc --rc-enable-metrics --rc-web-gui --rc-allow-origin --rc-web-gui-update --rc-web-gui-no-open-browser --rc-user=rclone --rc-pass=REDACTED --rc-addr=:5572 &

The rclone config contents with secrets removed.

[ToasterDEV-OneDrive]
type = onedrive
token = {"access_token":"REDACTED"}
drive_id = REDACTED
drive_type = personal
chunk_size = 250M

[ToasterDEV-GDrive]
type = drive
client_id = REDACTED
client_secret = REDACTED
scope = drive
root_folder_id = REDACTED
token = {"access_token":"REDACTED"}
chunk_size = 512Mi
team_drive =
use_trash = false
acknowledge_abuse = true

[Local]
type = local

[Union]
type = union
upstreams = ToasterDEV-OneDrive:Union ToasterDEV-GDrive: Lizita63-OneDrive:Union Flanch1942-OneDrive:Union
action_policy = lus
create_policy = lus
search_policy = lus
cache_time = 3600

[Chunker]
type = chunker
remote = Local:/mnt/mergerfs
hash_type = sha1all
chunk_size = 2Gi

[EncFS]
type = crypt
remote = Chunker:/
filename_encryption = obfuscate
directory_name_encryption = true
password = REDACTED
password2 = REDACTED

[Lizita63-OneDrive]
type = onedrive
token = {"access_token":"REDACTED"}
drive_id = REDACTED
drive_type = personal
chunk_size = 250M

[Lizita63-GDrive]
type = drive
client_id = REDACTED
client_secret = REDACTED
scope = drive
root_folder_id = REDACTED
token = {"access_token":"REDACTED"}
chunk_size = 512Mi
team_drive =
use_trash = false
acknowledge_abuse = true

[Flanch1942-OneDrive]
type = onedrive
token = {"access_token":"REDACTED"}
drive_id = 8559f24b32b9889d
drive_type = personal
chunk_size = 250M

[Flanch1942-GDrive]
type = drive
client_id = REDACTED
client_secret = REDACTED
scope = drive
root_folder_id = REDACTED
token = {"access_token":"REDACTED"}
chunk_size = 512Mi
team_drive =
use_trash = false
acknowledge_abuse = true

[MergerFS]
type = local
copy_links = true
links = true

[Backblaze]
type = b2
account = REDACTED
key = REDACTED
download_url = REDACTED
upload_cutoff = 5Gi
chunk_size = 5Gi
memory_pool_use_mmap = true

A log from the command with the -vv flag

Jun 24 16:13:50 UbuntuLXC rclone[5336]: OneDrive root 'Union': Checking for changes on remote
Jun 24 16:13:50 UbuntuLXC rclone[5336]: OneDrive root 'Union': Change notify listener failure: invalidRequest: [token.String] A value is required but was not present in the request.
Jun 24 16:13:53 UbuntuLXC rclone[5477]: OneDrive root 'Union': Checking for changes on remote

Firstly your mount commands are totally over engineered - maybe you have special purpose but flags like transfers, checkers, fast-list are not used by mount command at all. So it rather looks like you add flags randomly:) Unless needed e.g. to avoid throttling random stuff has no purpose and can actually cause problems.

The best way to transfer files between remotes is to run:

rclone sync source:DIR dest:DIR -vv - where source and destination are remotes - not already mounted remote. It will be much faster.

you can add --fast-list if gdrive is involved.

Sync can be stopped and restarted at your will. --checksum flag wont do anything as your remotes do note have common hashes.

Actually you still have a chance to sync with --checksum onedrive and B2. Onedrive should still have SHA1 (as B2) until end of June.

I see, I probably went a bit overboard as you said.

Just to be sure, given that the remotes are all combined into a mergerfs mount, would it be safe to just copy straight from the mountpoint's folder with rclone sync to Backblaze B2, let it run, and then go into the next remote?

As per the extra flags, for both Google Drive to B2 and OneDrive to B2, which options would lead to the fastest transfers between the remotes?

Thanks for the help!

Yes it should be safe. But will be much slower.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.