Before I get too deep into this I want to make sure that my file counts should not cause any issues.
I’ll try to be as clear as I can and minimize the wall of text.
One of my sync commands will be on a folder that will have a bucket dedicated to it, currently it has 200GB, 260,000 files and 22,000 folders. I have concerns that this could possibly cause issues. Versioning is somewhat desirable on this folder and security is not a big concern so there is no need for file encryption.
I am using the following command.
rclone rsync --log-file C:\bulogs\folder1.txt c:\bigfolder ThisConfig:ThatBucket
In these folders there are the source images that get processed, converted to PNG and uploaded to our webserver. These do not change very often, but there are well over 250,000 images and I have no idea how subsequent sync commands will perform. we have a 50Mbps synchronous connection and at 14hrs I am just barely 15%. I know that latency is going to be a huge factor in my initial data uploads, it could also be the command I used, i.e. settings I can fine tune for better performance…
Another thing I have been unable to figure out is how to specify a target directory in a bucket so I can make a .BAT file to do certain folders less frequently.
rclone rsync --log-file C:\bulogs\folder1.txt c:\folderA ThisConfig:ThatBucket
rclone rsync --log-file C:\bulogs\folder1.txt c:\folderB ThisConfig:ThatBucket
The problem here is the files go into the root of the bucket.
I also have a folder of phone recordings, currently 300+GB and 800,000 files I need these to sync initially, and have to encrypt the contents but not file names. These files never change so future syncs I need to skip any sort of operation that verifies hashes and only copies files that do not exist on destination while still validating them and could use some guidance on a command line here…
Sorry for the wall, that is my 2 major issues at this point.
Thanks for any advice or input.
I see no reason why that won’t work. It may need a bit of tuning though…
One thing you could try is –fast-list. This will cost you less money at B2 and will use more memory. It may or may not run faster - you’ll have to try it! I suspect in this case it will run a lot faster.
You can also try tweaking --checkers and --transfers.
If you’ve got data which doesn’t change very often you can do an incremental update by first finding the files that have changed with rclone then supplying that list to rclone copy.