General RClone questions and some specific to Backblaze B2

[Jeff]

Before I get too deep into this I want to make sure that my file counts should not cause any issues.
I’ll try to be as clear as I can and minimize the wall of text.

One of my sync commands will be on a folder that will have a bucket dedicated to it, currently it has 200GB, 260,000 files and 22,000 folders. I have concerns that this could possibly cause issues. Versioning is somewhat desirable on this folder and security is not a big concern so there is no need for file encryption.
I am using the following command.
rclone rsync --log-file C:\bulogs\folder1.txt c:\bigfolder ThisConfig:ThatBucket
In these folders there are the source images that get processed, converted to PNG and uploaded to our webserver. These do not change very often, but there are well over 250,000 images and I have no idea how subsequent sync commands will perform. we have a 50Mbps synchronous connection and at 14hrs I am just barely 15%. I know that latency is going to be a huge factor in my initial data uploads, it could also be the command I used, i.e. settings I can fine tune for better performance…

Another thing I have been unable to figure out is how to specify a target directory in a bucket so I can make a .BAT file to do certain folders less frequently.
rclone rsync --log-file C:\bulogs\folder1.txt c:\folderA ThisConfig:ThatBucket
rclone rsync --log-file C:\bulogs\folder1.txt c:\folderB ThisConfig:ThatBucket
The problem here is the files go into the root of the bucket.

I also have a folder of phone recordings, currently 300+GB and 800,000 files I need these to sync initially, and have to encrypt the contents but not file names. These files never change so future syncs I need to skip any sort of operation that verifies hashes and only copies files that do not exist on destination while still validating them and could use some guidance on a command line here…

Sorry for the wall, that is my 2 major issues at this point.
Thanks for any advice or input.

[ncw (Nick Craig-Wood)]

One of my sync commands will be on a folder that will have a bucket dedicated to it, currently it has 200GB, 260,000 files and 22,000 folders. I have concerns that this could possibly cause issues. Versioning is somewhat desirable on this folder and security is not a big concern so there is no need for file encryption.

I see no reason why that won’t work. It may need a bit of tuning though…

I am using the following command.
rclone rsync --log-file C:\bulogs\folder1.txt c:\bigfolder ThisConfig:ThatBucket
In these folders there are the source images that get processed, converted to PNG and uploaded to our webserver. These do not change very often, but there are well over 250,000 images and I have no idea how subsequent sync commands will perform. we have a 50Mbps synchronous connection and at 14hrs I am just barely 15%. I know that latency is going to be a huge factor in my initial data uploads, it could also be the command I used, i.e. settings I can fine tune for better performance…

One thing you could try is –fast-list. This will cost you less money at B2 and will use more memory. It may or may not run faster - you’ll have to try it! I suspect in this case it will run a lot faster.

You can also try tweaking --checkers and --transfers.

If you’ve got data which doesn’t change very often you can do an incremental update by first finding the files that have changed with rclone then supplying that list to rclone copy.

rclone lsf --files-only --max-age 1d C:\path > files.txt
rclone copy --files-from files.txt C:\path b2:bucket

That will be much more efficient. You’d want to do a full sync every now and again though.

Another thing I have been unable to figure out is how to specify a target directory in a bucket so I can make a .BAT file to do certain folders less frequently.
rclone rsync --log-file C:\bulogs\folder1.txt c:\folderA ThisConfig:ThatBucket
rclone rsync --log-file C:\bulogs\folder1.txt c:\folderB ThisConfig:ThatBucket
The problem here is the files go into the root of the bucket.

You need to put the name of the folder on the end, so

rclone rsync --log-file C:\bulogs\folder1.txt c:\folderA ThisConfig:ThatBucket\folderA

I also have a folder of phone recordings, currently 300+GB and 800,000 files I need these to sync initially, and have to encrypt the contents but not file names. These files never change so future syncs I need to skip any sort of operation that verifies hashes and only copies files that do not exist on destination while still validating them and could use some guidance on a command line here…

Rclone doesn’t verify hashes unless you pass in the -c flag. It will check modification times only so I think the default behaviour will work for you. You might want to use --fast-list here too.

However if you want to be sure you never overwrite anything you can use the --imutable flag which is for making an archive where the remote contents mustn’t change.