Backup 14TB to BackBlaze B2

[pedz (Perry Smith)]

If any of the following sounds insane, please tell me.

I have a NAS (TrueNAS) with 14TB of data on it that I want to backup to BackBlaze. The data is relatively static and does not change much. My prediction is that it will take about a month to do the upload.

The top level directory has about 45 subdirectories. Some are large (with a lot of data within the subdirectory) and some are small.

Which of these would you suggest doing?

1. I could do one rclone command to copy the entire NAS to one B2 bucket.
2. I could do 45 rclone commands, one for each subdirectory, to one B2 bucket.
3. I could create 45 buckets and do an rclone command per subdirectory into its own bucket.

Or there may be some alternatives. For example, there is nothing magic, actually, about the 45 top level directories. I could combine them into some small number of roughly equal sizes.

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

@pedz

Which of these would you suggest doing?

once a file is uploaded, rclone will not re-copy it.
so i would choose option 1 using rclone sync
just let it run.

[ncw (Nick Craig-Wood)]

I'd second @asdffdsa 's advice just sync the top level directory.

The only case I'd consider splitting it is if you have 100s of millions of files - a restart will take a long time in that case.

Adding --fast-list will speed up restarts if you have enough memory (roughly 1GB per 1 Million files).

If I was doing this I'd set --bwlimit - perhaps on a schedule so as not to interfere with other internet users.