Project Copy, delete

Help and Support
1

Hi
I have 30TB Data at wasabi
It works fine.
I would like extra security and have made an agreement with scaleway to keep a copy of my data.

Here's what I want:
Data must be synchronized from Wasabi to Scaleway daily.
When data is deleted on Wasabi, it must be deleted on Scaleway 14 days later.

There will be approx. 200GB new data every day and 180GB is deleted every day (Wasabi)

Glaice at Scaleway
Server to Server copy (if possible)

Can it be done with Rclone

I am using the latest version of rclone
Jacob

2

hello and welcome to the forum,

rclone, nor any software, can do server to server between two different cloud providers.
there must be some computer to run rclone.

your best bet is to rent a virtual machine at scaleway and run rclone on that.
most providers do not charge ingress fees.

the exact rclone command would depend on the amount of files and the file sizes but this would be a good start
rclone sync wasabi: scaleway: --checksum --fast-list

3

Hi and thanks for your reply
my files are 90% 30mb files and 9% 16mb
I just tried your command and it works fine👍
Good idea to place it Scaleway, it does not look very expensive.
Do you think it is possible to make a solution so that all deleted files are only deleted after 30 days

4

i use wasabi, once a file is deleted it is gone forever.

might use versioning as a workaround
https://wasabi-support.zendesk.com/hc/en-us/articles/115003947151-If-I-delete-a-file-within-90-days-after-storing-it-how-is-that-handled-from-a-billing-and-file-recovery-perspective-

also, this might help
https://wasabi-support.zendesk.com/hc/en-us/articles/360046429571-How-to-download-ALL-versions-of-all-files-within-a-Bucket

so with wasabi, keep in mind wasabi that if you

  1. upload a file today
  2. delete it tomorrow
    you will be charge for 89 days of storage.
    tho there is a trick around that....
5

You can also use --backup-dir to store the old backups.

So you could $(date -I) in the backup dir and make a new directory for each backup then delete any older than 14 days.

This will require a very small amount of scripting so you only keep the last 14 days of backup dirs.

6

yes, that is what i always do, an example of that would be
rclone sync /path/to/data remote:data/backup --backup-dir=remote:data/archive/20210405.093036

also, a side question, as i use wasabi and aws s3 deep glacier.
why did you choose scaleway over aws as it is seems much move expensive to store data?
seems to only make sense if you plan to restore from scaleway glacier often.

7

Thanks for info on Wasabi💪
Here's what I want:
Data must be synchronized from Wasabi to Scaleway daily.
When data is deleted on Wasabi, it must be deleted on Scaleway 14 days later.

8

i think the price is good C14 Cold Storage €0.002/GB/month

9

Hi
I have read here in the forum that data in glacier (Scaleway) can not be moved.
I have tested, it works with Standard
Is it possible to give the file (to be deleted) a date and then just delete it after 14 days. (Without moving)

10

imho, that seems very expensive

aws deep glacier is $0.004/GB/month, which i think is €0.0034/GB/month

11

all files in the bucket?
or
just some individual files in the bucket?

on scaleway website, you would use a feature called lifecycle.

12

? that seems very expensive

13

It will always be a folder to be deleted

14

not sure what you mean?
you agree that scaleway is very expensive?

15

No
AWS €0.0034/GB/month
Scalw €0.002/GB/month

16

sorry, that price was to aws s3 glacier.

for aws s3 deep glacier, per GB
Amazon S3 Simple Storage Service Pricing - Amazon Web Services
$0.00099 per GB which should be €0.0008

17

S3 stores objects, has no concept of folders.
rclone has no way to know the creation date of a folder.
Organizing objects in the Amazon S3 console using folders - Amazon Simple Storage Service

and for files, rclone can only use the modification date, has no concept of creation date.
so if you want to delete a file 14 days after creation, rclone does not know the creation date.
tho depending on your use case, there might be a way around that.

scaleway, like most cloud providers, has a feature called lifecycle, that can delete objects after a period of time or transition files from standard storage to glacier storage.

18

ok it's cheap.:muscle: what about traffic and other expenses

19

Can it be made as the folders/file to be deleted at Scaleway they will be renamed or the folder/file for a "Tag" "Value" which I can then capture with Lifecycle Rule

20

as for other expenses, it is all on that weblink i shared.
aws has a very complex pricing scheme.
they charge for api calls, they charge to transition from glacier to standard, they are charge to egress.
so it all depends on the use-case.

for me,
i have a local server.
i keep a copy of recent backups and valuable data wasabi.
everything gets upload to aws s3 deep glacier
i archive some data to external usb hard disks and burn blu-ray discs and keep it off-site

in the three years of using aws s3, i have never downloaded any data and i do not expect to.