As a user who is considering using rclone with Wasabi, would it be feasible for rclone to add support for Wasabi's MOVE API (an extension to the standard S3 API)?
I was going to post in this thread, but it is locked to new posts:
While AWS' official S3 API doesn't have a server-side move, Wasabi has extended the API to provide one. They do this because all new objects in Wasabi will be charged for a minimum of 90 days of storage -- and the copy and delete method to move or rename objects doesn't play nicely with that billing practice. Here is a link to their API documentation:
It is certainly possible. Rclone uses the AWS S3 SDK so to implement it we'd need to find out how to make that call via the SDK which might be fairly tricky.
Other than that it would be easy to implement a Move API for the wasabi provider.
hi leerspace,
i am the one whom you qouted, asking about wasabi and server-side-copy vs. move.
i had no idea that wasabi had an extended api.
i would love to have a move option, so when using --backup-dir, files can be moved versus copy.
i am going to look into this and perhaps you can to.
i understand from ncw, that rclone uses the aws s3 sdk, so it does not seem likely that rclone will enable it.
i also use cloudberry explorer and s3 browser, i am sure that both of them do not implement server side move.
i must be a super-geek because just the very idea of server-side-move has me way too excited!
I suspect it's not possible to do this using the AWS S3 SDK, and that it never will be unless Amazon adds an official move endpoint to the S3 API. At least for the only other project I'm aware of adding support for this custom endpoint, Duplicacy, they wrote their own lightweight implementation against the raw API.
I'm interested, but realistically probably won't be able to give this the time it deserves (e.g., setting up a dev environment for rclone, learning how the project is structured, actually doing the dev, then addressing all the issues likely to be found) for at least several months.
I've emailed their support for how they envision S3 clients to support their custom extension to the standard S3 API and if they have any tips/plans/resources to make this easier; but I'm not sure that will get anywhere based on their response so far.
Do you prefer using these forums for tracking feature requests or would it help for me a create a github issue to track this enhancement?
FWIW, starting to look into this has given me the impression that I'm trying to use a storage provider with a Frankenstein API that isn't adequately supported by the storage provider; so at this rate I may end up moving back to Backblaze B2 and using rclone with that. It's a bit discouraging to think that every client I may want to use would need to be customized to not incur additional charges if I move or rename a file with a vanilla S3 client less than 3 months after uploading the file.
I think the copy-and-delete method being used to move/rename files should all be happening server-side. It's just that if you move/rename files less than 90 days after creating them, then my understanding is that you're essentially going to be paying twice for those files (until the 90 day clock on the original object is hit) when using Wasabi.
about the 90 day clock, if you ask wasabi they will reduce it to just a 30 day clock,
i use veeam backup and replication and backup those files to wasabi using rclone.
if you email wasabi, tell them you use veeam, they will set the change the 'clock' from 90 days to 30 days.
check out this post
" On the topic the 90 day minimum storage retention policy, we are in the process of rolling out a 30 day minimum storage policy for customers where the optimal backup strategy will benefit from it. Any Veeam user that is a Wasabi customer can request a switch to the 30 day policy via a note to support@wasabi.com."
If anyone else comes across this thread, moves in rclone (and likely other vanilla s3 clients) don't seem to cause double-charging due to how Wasabi handles copies (details in the linked github issue). This seems to be a non-issue; and I'm not sure what benefit there is to using the MOVE API.