Chunker simultaneous upload of chunks

Feature
1

My Issue:
I am trying to upload large files, mainly backup images, which can easily be 250GB+, to my OneDrive storage space. There are two problems here:

A) This only applies to files >250GB, which is OneDrive's max file size. If I exceed the max file size, the file upload will fail.

B) OneDrive seems to put a limit on the upload throughput per file. This means uploading several files (I think I have been running 4 simultaneous transfers, the RClone default) runs quite fast (maxes out my link), but uploading a single file is quite slow. This means, for large, single files, E.G backup images, the backup is very slow.

A solution which solves both of these problems is to split the large files into several parts, and upload each part as a separate file. This means the upload will run much faster, as multiple files can be uploaded simultaneously, AND the max file size is not exceeded.

I have successfully split the files with 7-Zip on my local machine, prior to using RClone to then upload the files, but this is firstly quite manual, and secondly, and this is a bigger problem, it requires 2x the disk space: Space for the original file, and space for the split files, before they are uploaded.

I thought using RClone's 'chunker' would be a good way of solving the problem: More automated, does not require so much disk space (because the file is split as it is uploaded, and I was hoping chunker would be smart enough to upload multiple chunks simultaneously, improving the speed.

However, as far as I can see, chunker will only upload a single chunk at a time, limiting my upload speed severely. Is there some option I can use to solve my issue?

Alternatively, feel free to suggest any other solutions to the problems above given my circumstances. (mainly the disk space requirement and upload speed).

Appreciate any input.

Cloud Storage Provider: OneDrive Personal (1TB Subscription).

Full Command, as an example:

rclone copy C:\Users\User\Desktop\9gb.bin OneDrive_CloudStorage_Vault_Chunker: --progress

RClone Log:

2025/04/23 15:24:47 INFO  : 9gb.bin.rcchunk.001_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.001
2025/04/23 15:24:48 INFO  : 9gb.bin.rcchunk.002_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.002
2025/04/23 15:24:48 INFO  : 9gb.bin.rcchunk.003_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.003
2025/04/23 15:24:49 INFO  : 9gb.bin.rcchunk.004_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.004
2025/04/23 15:24:50 INFO  : 9gb.bin.rcchunk.005_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.005
2025/04/23 15:24:50 INFO  : 9gb.bin.rcchunk.006_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.006
2025/04/23 15:24:51 INFO  : 9gb.bin.rcchunk.007_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.007
2025/04/23 15:24:51 INFO  : 9gb.bin.rcchunk.008_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.008
2025/04/23 15:24:52 INFO  : 9gb.bin.rcchunk.009_6gc18n: Moved (server-side) to: 9gb.bin.rcchunk.009
2025/04/23 15:24:53 INFO  : 9gb.bin: Copied (new)
2025/04/23 15:24:53 INFO  : 
Transferred:   	    9.002 GiB / 9.002 GiB, 100%, 1.521 MiB/s, ETA 0s
Checks:                 9 / 9, 100%
Renamed:                9
Transferred:            1 / 1, 100%
Server Side Moves:      9 @ 9 GiB
Elapsed time:     57m16.9s

This was with -v set, I can provide a -vv log if needed

RClone Config:

[OneDrive_CloudStorage]
type = onedrive
client_id = XXX
client_secret = XXX
token = XXX
drive_id = XXX
drive_type = personal

[OneDrive_CloudStorage_Vault]
type = crypt
remote = OneDrive_CloudStorage:/rc_crypt
password = XXX
password2 = XXX

[OneDrive_CloudStorage_Vault_Chunker]
type = chunker
remote = OneDrive_CloudStorage_Vault:
chunk_size = 1Gi
name_format = *.rcchunk.###
description = Chunker overlay for OneDrive_CloudStorage_Vault remote.

RClone Version:

rclone v1.69.1
 - os/version: Microsoft Windows 10 Pro 22H2 22H2 (64 bit)
 - os/kernel: 10.0.19045.5737 (x86_64)
 - os/type: windows
 - os/arch: amd64
 - go/version: go1.24.0
 - go/linking: static
 - go/tags: cmount
2

welcome to the forum,

imho, backup files are very important and should not use too many tricks to workaround a simple issue.

  1. using 7zip to compress backup file.
  2. using chunker which is beta, should not be trusted.
  3. rclone crypt cannot verify the integrity of file transfers using checksums.

one solution, get 1TB from a trusted provider for reliable backups, such as wasabi, idrive or hetzner storagebox or any number of providers.

in a disaster recovery, need fast downloads, which wasabi certainly offers.
will never get that from onedrive personal.

what is the software program creating the images?

4

Thanks. Good points you mention. However:

  1. I am not using 7-Zip to actually compress the backup file. I was only using the 7-Zip 'split file' feature, to work around the OneDrive 250GB limit, and the speed limit per file. I was hoping to do away with 7-Zip by using Chunker instead.

  2. I agree using Chunker could be considered 'risky' as it is beta. However, as far as I know, the way Chunker splits files is relatively simple, and to quote the Rclone Chunker page:

As the split is trivial one could even manually concatenate data chunks together to obtain the original content.

  1. I am unsure exactly what you mean about crypt not verifying the integrity (sorry!), but it seems crypt is trusted by quite a few people to look after their data?

  2. I do have many of these backup files in cold storage on a local disk too, but storing in the cloud is a nice additional redundancy.

I know I could use an alternative storage provider, however I already have a couple of OneDrive accounts in use, and would like to use what I already have if possible, due to various other factors.

Also, I do use this storage for items other than backup files, for which my original question still applies, as to improving transfer speeds of large files through simultaneous uploads.

Thanks.

5

Forgot to mention: Some of the images are Windows VHD based images, and some are from Veeam Endpoint Agent.

However, as mentioned above, the real problem is: How to upload large files at good speed without having to manually split first, requiring 2x disk space.

6

now that i am thinking about it, might want to use rclone+restic.
but again, imho, much like crypt+chunker, an overly complicated contraption for critical backups files.

as for your feature request.
chunker: add support for multi-thread uploads #5041

https://forum.rclone.org/t/rclone-copy-entire-drive/25155
https://forum.rclone.org/t/rclone-compression-zip-rar-and-data-transfer/30942/6

ok, i use veeam backup and replication.
the good news is that veeam encrypts its own .vib|.vbk files, so no need for crypt remote.

and another workaround, which i do, is have a veeam backup repository in the cloud.
i rent the cheapest vm from hetzner, run a veeam linux backup repository, which is connected to a hetzner storagebox. so, no issues with large file sizes and file verification for file transfers.

i know this is a bit off topic, but i could explain in detail...
check the rclone docs and many posts in the forum. rclone crypt remotes cannot verify file transfers using hashes.
so not trustworthy for backups.

onedrive is well known for being very slow. wasabi has no problem saturating 1Gbps fiber connection. for uploads and more importantly, downloads.

7

I see you use Linux, you could just use split command to break apart the files into smaller sets before uploading to OneDrive.
As @asdffdsa mentioned, crypt does not do a byte check of the upload when uploading, but you can use cryptcheck to validate it after upload to minimize risk, or encrypt it locally before uploading to validate it was completed successfully. If you crypt before uploading you just need enough additional space for some segments.
But I agree on not recommending OneDrive to be the host of your backups, slow speeds, random throttling etc. can cause a huge headache trying to restore your data.

8

Thanks all for the great info.

I have had a brief look at the Restic website, and whilst it looks like a really nice piece of software, I don't think it is exactly what I am looking for. It seems to be a file-based backup program, whereas Veeam creates image backups, which as I understand, are more useful for backing up the actual OS. (Note, @jcastill, I use primarily Windows).

@asdffdsa, you pointed out that Veeam can encrypt its own backup files, meaning crypt is not needed. I am aware of this, however currently, I have just been using un-encrypted Veeam backups stored in a crypt, because I am also storing some general files on this OneDrive account. (Which I also need encrypted).

I do not claim to properly understand the file integrity verification process, but to quote the Rclone crypt page:

Hashes are not stored for crypt. However the data integrity is protected by an extremely strong crypto authenticator.

I assume this is what @jcastill is referring to with cryptcheck? It seems strange to me that if crypt cannot be trusted for backup file integrity, why would anyone use it for anything even slightly important?

I should also mention that this particular backup scheme is mostly for home use, so a slightly slow download speed when retrieving a backup file from OneDrive is not a disaster, albeit a bit inconvenient - It's not like I am restoring a mission-critical company server or anything.

9

correct. create the backup file using veeam, copy to file to cloud using rclone, restic, whatever.
restic will split the file into chunks and upload the chunks, which is one of your primary goals.

that has nothing to do with file transfer verification using hashes.

fwiw, for sure, that is not a viable solution for backups..
use veeam to encrypt the file, use rclone to copy to OneDrive_CloudStorage:veeam, that way rclone can use hashes.

anybody with rclone experience would know that crypt cannot verify file transfers using hash, and as a result, not suitable as a destination for critical backups like veeam.

imho, 99.9% of crypt remote are used to hide pirated, illegal media files and then rclone mount to stream it.

well, you mentioned that backing up the actual OS, so in a disaster, unless you have a local copy, would have to download a huge 100GiB .vbk on usb or local netshare and then boot veeam recovery image to restore

but luckily, i have posted in this forum and the veeam community forum, about a wonderful workaround trick for slow internet connections and veeam disaster recovery.

  1. boot the WINRE veeam recovery image on machine
  2. copy rclone.exe and rclone.conf to x: drive
  3. run rclone mount onedrive:veeam x:/veeam
    since this is a chunked download, no need for rclone vfs cache, no need for double storage.
  4. if the backup is of a virtual machine or a physical machine being restored as virtual, can start veeam instant recovery
    in five minutes, your window server will start to boot up, in ten minutes, will be able to login.
  5. if the backup is being restored to a physical server, run normal veeam recovery.
10

Thanks again for your response. I am certainly rethinking my whole setup at this point...

As to this point about file verification using hashes, and to get this completely clear: Despite crypt lacking support for file hash verification, does running cryptcheck after the upload has completed perform an equivalent verification of file integrity?

In fact, in this post (Integrity of rclone ENCRYPT - #5 by ncw), Nick says:

No, the crypt backend does a version of rclone cryptcheck while it is uploading files so it should notice corruptions at that point.

Sorry to keep going on about it, I just want to get it clear. I thought RClone crypt was a really nice solution for storing potentially sensitive files on a variety of cloud backends, but you are now causing me to doubt this somewhat... :slight_smile:
If so, I will have to do a lot of re-thinking.

11

my point is that with backup files, need to do whats best for the files, not trying to jerry-rig solution using the wrong, overly complex with too many parts.

  • chunker - beta
  • crypt - lack of checksum support.
  • onedrive - limits files size, flaky, very slow for uploads and even slower for downloads.
    imho, the safest, most reliable, and fastest solution is s3 provider such as wasabi, aws s3 or idrive

yes, rclone will have to re-read the entire local file to computer the hash.

well, the issue here is about critical backup files, not sensitive files.
i am very paranoid when dealing with backup files.

the rclone crypt is rock stable and has been so for many years.
but, given that veeam encrypts its own backup files, no reason to use crypt.

still not a real solution, as that is only for uploads, not downloads.

12

I can certainly understand your paranoia when considering important backups!

However, I am relieved to know that the crypt remote can be trusted to some extent, as I began to get worried that it was completly unreliable due to the lack of integrity checking, and only to be trusted for pirated, illegal media files.

still not a real solution, as that is only for uploads, not downloads.

I suppose if one was really concerned about this, you could download the files to a local location, and then run rclone cryptcheck downloaded_files/ cryptedremote:path_to_files/ before using the files, to make sure no corruption happened during the download? Unless I am missing some vital point here...

very slow for uploads and even slower for downloads.

Uploads can be slow, especially when uploading a single large file, but I just tested downloading 10 100MiB files, and got 200-300Mbps throughout. Yes, not saturating the link, but enough to make me happy. (And I am aware that throttling can be applied if downloading a much larger file, also non-parallel downloads are slower, but chunking avoids this).

13

I have been playing around with Restic over the last day or two, and have been really impressed. So @asdffdsa, thanks for introducing me to a great new piece of software.

I have setup a system, as you suggested, whereby I use Restic with an RClone backend (accessing the OneDrive remote) to upload my backup files. This works really nicely, and solves the problems listed above:

  1. Slow uploads of single files - Restic chunks files into packs which seem to upload simultaneously, as I get far better upload speeds to OneDrive than with a single file.

  2. Max filesizes on OneDrive - As above, Restic chunks the files so not an issue.

  3. Having to re-upload the entire image file after every backup run - since Restic performs de-duplication, I think this will significantly lower the amount of data that needs to be transferred in this situation.

  4. Possible concerns about the integrity of files in Rclone crypt - Restic encrypts repositories itself, so I do not have to have the Restic repo within an Rclone crypt, I can just put it in my OneDrive remote directly.

So within Rclone, I now have OneDrive_CloudStorage:\restic_repo\, which contains the Restic repository, and OneDrive_CloudStorage:\rc_crypt\, which is accessed by through a Crypt remote, OneDrive_CloudStorage_Vault:, which I use for storage of a few general files (generally accessed via rclone mount).

Overall, and so far, I am really happy with this solution, and it fits my needs well. I'm sorry if my backup strategy offends anybody due to being 'too complicated'. :slight_smile: