Rclone has been banned from Amazon Drive

[xupetas]

@Philip_Konighofer

f you don't have a whole bunch of disks connected directly do the system, IOPS and latency becomes an issue. if you distribute among many small blades with just a few disks you need costly interconnects. therefore dedup becomes problematic. there are many expert opinions available on the web, stating google, aws, etc.. do not use dedup on large scale.

They don't use dedup on blob's. It's different from linear storage.

[Philip_Konighofer (Philip Könighofer)]

do have any reference?

[jnfingerle (Jan Niklas Fingerle)]

I explicitly talked about block level and your example only makes sense with unencrypted data, wich I wasn’t talking about. After that it’s plain science. Explain, how - in encrypted data! - bit patterns are more likely to occour in a block than other bit patterns, and you’re on with deduping. If you can’t, you’re not, because addressing is the copy will be at least as costly (in terms of space needed) as the real data. It’s just maths, arguing won’t help you there.

[GoneD]

I don’t know if i have corretly understand, but

If Rclone come back on Amazon, we are not possible to re-use encrypted files ???

Thanks for reply,

[jnfingerle (Jan Niklas Fingerle)]

No. This is just the meta-discussion how costly rclone users are for Amazon. It (the costlyness, not the discussion) may have impact on if Amazon reinstates rclone, but if rclone came back on this discussion wouldn’t have any further impact on you using rclone with Amazon.

[xupetas]

Simple math and using the YouTube example above.

You have 15 Ebytes of data, stored on 4 mbytes blocks. You are telling me that mathematically is impossible to have the exact same block repeated?

When you encrypt you just scramble your data using an algorithm. You don’t invent new characters or symbols. And there is a finite number of possibilities.

[xupetas]

Commercially? EMC datadomain

[xupetas]

I dint said that it has no impact! I said I is negligible compared to the bandwidth and CPU for the pull and gets that a rclone mount uses

[Philip_Konighofer (Philip Könighofer)]

@xupetas

EMC datadomain

cost per TB? per year?

[xupetas]

It’s a enterprise Datacenter product. It’s not a nas or a service, although they might lease the equipment.
The ones i know that exist here were bought and not leased

[Philip_Konighofer (Philip Könighofer)]

ok, so do the math. what does a TB cost you for your storage?

example: i bought it for X, it has Y TB. so?

[jnfingerle (Jan Niklas Fingerle)]

I didn’t talk about bandwith and CPU. So it’s a straw man.

I only pointed out, that de-duplication doesn’t work with encrypted data. You seem no longer to contest that. Great.

[xupetas]

Here we manage around 15PB of storage and use over 22 PB and there is a lot of room to spare. And yes the data is all encrypted

[Philip_Konighofer (Philip Könighofer)]

answer the question…

[xupetas]

You buy X PB of data and you get X PB of data. If you buy this equipment you can maximize your use to POSSIBLY Y PB of data. Is that so difficult do understand?

[xupetas]

Try dedup with encrypted data and zfs and then get back to me.

[Philip_Konighofer (Philip Könighofer)]

my point was that storage is not free (even for google) and you cannot get “almost infinite” storage with dedup alone. therefore storage costs money (even for google). right?

[Philip_Konighofer (Philip Könighofer)]

are you using compression with your zfs too?

[xupetas]

It’s peanuts compared to the other costs

[Philip_Konighofer (Philip Könighofer)]

it’s more than the average “poweruser” pays.