Syncing many files (5M+)

[asdffdsa (jojothehumanmonkey)]

@ncw

wasabi?

that is a good suggestion. also, idrive works well with rclone.

i did some testing with wasabi with a bucket con tainting 1,000,000 files.
rclone sync --dry-run took just 33 seconds

Fastest way to check for changes in 2.5+ Million files?

for what it is worth, using wasabi, a s3 clone, known for hot storage for 1,000,000 files this command took 33 seconds. rclone sync d:\folder wasabi01:folder --size-only --transfers=64 --checkers=64 --dry-run --progress --stats-one-line --log-level=DEBUG --log-file=log.txt

[Netherwhal]

thats where I came from - but at 10TB+ and double the costs per TB, the price difference is very noticable.

minio is not very optimized compared to seaweedfs.

[asdffdsa (jojothehumanmonkey)]

@Netherwhal

the price difference is very noticable.

how are you calculdating that, as idrive pricing should be close to hetzner?

[Netherwhal]

comparable, but at 10TB there is a difference in total. Hetzner is something like 2-2.5 USD per TB. Idrive is 3.5+? B2 is 4-6?

[asdffdsa (jojothehumanmonkey)]

@Netherwhal

Idrive is 3.5+?

i could be wrong but for 1TB/month

idrive = $2.50
hetzner= $2.48 with the 10TB plan