rclone let’s you set the number of checkers and the number of transfer threads used to accommodate for lots of large files or lots of small files. For lots of small files, I think you’ll find it does reasonably well especially if you turn up the number of checkers.
Some things worth consdering:
- Do you need encrypted volumes? Currently encrypted volumes (crypt) won’t store a md5sum on amazon drive resulting in rclone only able to compare based on size which will likely cause you problems. There are ways to work around this and the developer is considering other work arounds like calcuating the MD5 or SHA on the fly. If you will not be using crypt then this all won’t be too much of an issue.
Amazon, for example, doesn’t store modification times like we’d want. So a sync compare will only be by file size (see #1) or by checksum which I think you would need considering the types of files you are storing (xml for example).
Since you have a small amount of data even a checksum won’t take that long I wouldn’t think since ACD has that pre-calulated and all it needs to do is calculate the checksums on the local side. 32GB is pretty small.
-
I only have experience with Amazon so I will defer this. Im very happy with it though. I can saturate my 150 Megabit ISP with it.
-
You probably will want to run with --checksum --checkers X --transfers Y adjusting the X and Y per testing. I personally use the defaults but I have a mix of very large files and small files (160,000 total in 8TB). You’d probably do well by adding checkers so many threads can check small files quickly in attempts to use as much bandwidth as possible.