Possible to rename on Wasabi + crypt to avoid charges?

[stenwt]

What is the problem you are having with rclone?

When I rename a file, e.g.:
rclone moveto myremote:foo myremote:bar , rclone copies the file and deletes the original it, rather than simply changing the name on the remote. This is a problem on Wasabi, because deleted files continue to be charged for 3 months after the delete.

One thing I have noticed is, once, I accidentally used "move" instead of moveto. After the first move, moveto

What is your rclone version (output from rclone version)

rclone v1.55.1

• os/type: linux
• os/arch: arm
• go/version: go1.16.3
• go/linking: static
• go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

I've tried on armv7 linux, aarch64 linux, amd64 linux

Which cloud storage system are you using? (eg Google Drive)

Wasabi S3 + crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone moveto remote:foo remote:bar

The rclone config contents with secrets removed.

[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = mykey
secret_access_key = mysecret
endpoint = s3.wasabisys.com

[wasabic]
type = crypt
remote = wasabi:mybucket
filename_encryption = standard
directory_name_encryption = true
password = 12345
password2 = 54321

A log from the command with the -vv flag

2021/05/09 03:03:53 DEBUG : Using config file from "/home/sten/.config/rclone/rclone.conf"
2021/05/09 03:03:53 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "moveto" "-P" "-vv" "wasabic:foo" "wasabic:bar"]
2021/05/09 03:03:53 DEBUG : Creating backend with remote "wasabic:foo"
2021/05/09 03:03:53 DEBUG : Creating backend with remote "wasabi:mybucket/pqbmgfonf5pmtfcajbrpqlcbmpkmqp9j5ssh44tj86paj1bh1vvg"
2021/05/09 03:03:54 DEBUG : fs cache: adding new entry for parent of "wasabi:mybucket/pqbmgfonf5pmtfcajbrpqlcbmpkmqp9j5ssh44tj86paj1bh1vvg", "wasabi:mybucket"
2021/05/09 03:03:54 DEBUG : Creating backend with remote "wasabic:"
2021-05-09 03:03:54 DEBUG : foo: Need to transfer - File not found at Destination
2021-05-09 03:06:08 INFO  : foo: Copied (server-side copy) to: bar
2021-05-09 03:06:09 INFO  : foo: Deleted
Transferred:        4.052G / 4.052 GBytes, 100%, 30.837 MBytes/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:      2m15.3s
2021/05/09 03:06:09 INFO  :
Transferred:        4.052G / 4.052 GBytes, 100%, 30.837 MBytes/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:      2m15.3s

2021/05/09 03:06:09 DEBUG : 4 go routines active

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

i use wasabi, which as a s3 clone, does not support server-side-move operations.

which is documented here

about the 90 days, if you use wasabi to store veeam backups files, contact wasabi and they will reduce the charge period to 30 days.

[jwink3101 (Justin Winokur)]

@stenwt

This is a problem on Wasabi, because deleted files continue to be charged for 3 months after the delete.

My understanding isn't that you pay 3 more months; it's that you pay up to three months. So if your file was there for two months and then you delete it, you pay for an additional month. If your file was there 5 months and you delete it, it gets deleted.

Doesn't change your question but worth noting

[asdffdsa (jojothehumanmonkey)]

hi, yes that is correct

[jwink3101 (Justin Winokur)]

@asdffdsa

does not support server-side-move operations.

You are 100% correct about the S3 spec but, FWIW, Wasabi has a custom move API. It does not make sense, in my opinion for rclone to natively support it in S3 but OP could submit a feature request a new wasabi remote type.

[asdffdsa (jojothehumanmonkey)]

yes, i am aware of that and somewhere in the past, i raised that issue in the forum.

once i contacted wasabi about it, asking for them to offer better support and working with rclone for that move api and updating the endpoints used in rclone to match the endpoints now offered by wasabi.
they were not responsive.

tho they were helpful with this nightmare.

[jwink3101 (Justin Winokur)]

That is unfortunate. Personally, the 90 day thing is one of the major reasons I chose B2 over Wasabi (and it is $0.99/Tb/month less). I have heard Wasabi is faster but since my use case is just backups, it hardly matters to me.

B2 also offers shipped drives if I ever need it.

[asdffdsa (jojothehumanmonkey)]

as i upload veeam backups, it is a 30 day thing and if the use-case is backups, the 90 or 30 day rentention period is not really an issue.

i like wasabi for the very fast downloads at no charge.
how does b2 compare in terms of download speeds?

and as for storage price,
i keep the latest backups in wasabi and need to know i can saturate my 1Gbps fiber optic connection to download after disaster, ransomware or what not.
i move older stuff to aws s3 deep glacier for $1.01/TB/month.
so overall my costs are very low...

[stenwt]

I asked about reducing my retention, apparently that's only available for customers storing >50TB with a pay-ahead plan rather than pay-as-you-go. I'm neither of those.

[asdffdsa (jojothehumanmonkey)]

if you upload some veeam backups files, then tell wasabi, they will look for the files and then change the retention period to 30 days.

perhaps @jwink3101 suggestion about b2 might be the way to go.

[stenwt]

I actually just migrated B2 -> Wasabi after getting crushed with API charges from B2. I'm also enjoying the faster uploads and downloads. I may just have to lump it, unless/until the aforementioned custom move api is implemented.

[jwink3101 (Justin Winokur)]

@asdffdsa

if the use-case is backups

This really depends on the backup strategy. If you use rclone directly with --backup-dir then it can add up. But I do see your point.

@asdffdsa

downloads at no charge

I forgot about that when I said it was cheaper. That does change the calculus

@asdffdsa

how does b2 compare in terms of download speeds?

I don't really know. I don't have much to compare to and I have 200/10 connection so I am usually the limiting factor.

---

In general, I use B2 because I closely follow the company and use their personal backup service too. They are far from perfect but I haven't had any issues to speak of. They also don't have anything like a minimum object size or time. Just feels more straight-forward. (except for egress)

Anyway, This has been an interesting discussion and I appreciate it!

[jwink3101 (Justin Winokur)]

@stenwt

after getting crushed with API charges from B2

Was it with rclone? Are you using --fast-list? (Not sure it matters anymore or not. I think it does for some actions).

Either way, best of luck!

[asdffdsa (jojothehumanmonkey)]

ditto

i have been using wasabi to stream my media collection to a local jellyfin on a raspberry pi 4.
so far, no complaints from wasabi....

the great thing is that there are so many cloud providers to choose from and rclone works with them all!
computational life is good!

[ncw (Nick Craig-Wood)]

@jwink3101

@asdffdsa

does not support server-side-move operations.

You are 100% correct about the S3 spec but, FWIW, Wasabi has a custom move API . It does not make sense, in my opinion for rclone to natively support it in S3 but OP could submit a feature request a new wasabi remote type.

Hmm, custom move protocol...

It probably wouldn't be too difficult to implement (handwaving) if someone wanted to have a go?