"Delete" (Mask/Hide) From Read-Only Backend w/Union

[hax0rwax0r]

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I'm trying to "delete" (mask/hide) a file that is on a read-only Google Drive backend that is also using union with SMB backend that is read/write. I know it is not possible to actually delete a file on a read-only filesystem, thus I'm really looking to mask or hide the file from the directory structure that union creates.

It seems that @raymond-h asked a similar question back in 2020 about this functionality but didn't get a response or comment that was related to additional functionality similar to "unionfs whiteout," or I am misinterpreting what was being stated/asked for:

Union file deletion using whiteouts

Hello. In the current beta (beta release 1.51.0-124-ge569977c), when you try to delete a file from a union, it will try to delete the file from whichever upstream contains the file, even if the upstream is read-only. In the current stable release it will even delete files from upstreams that are explicitly stated in the docs to be read-only. It would be great to support something similar to unionfs's whiteout feature, where a file is created in a read-write upstream that notes to unionfs that …

Is there a known way to configure rclone that I'm simply overlooking that would emulate this functionality? Or perhaps a more obvious solution to what I'm trying to do here, even if it isn't rclone functionality specific?

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2
- os/version: centos 7.9.2009 (64 bit)
- os/kernel: 6.4.11-1.el7.elrepo.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

# mount | grep media
union_media: on /mnt/media type fuse.rclone (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

# rm -rf /mnt/media/TV/Show\ Name/Season\ 01/Old\ File\ To\ Delete.mkv
rm: cannot remove ‘/mnt/media/TV/Show Name/Season 01/Old File To Delete.mkv’: Operation not permitted

# rclone delete union_media:TV/Show\ Name/Season\ 01/Old\ File\ To\ Delete.mkv
2023/10/21 22:59:05 ERROR : Old File To Delete.mkv: Couldn't delete: permission denied
2023/10/21 22:59:05 ERROR : Attempt 1/3 failed with 2 errors and: failed to delete 1 files
2023/10/21 22:59:06 ERROR : Old File To Delete.mkv: Couldn't delete: permission denied
2023/10/21 22:59:06 ERROR : Attempt 2/3 failed with 2 errors and: failed to delete 1 files
2023/10/21 22:59:06 ERROR : Old File To Delete.mkv: Couldn't delete: permission denied
2023/10/21 22:59:06 ERROR : Attempt 3/3 failed with 2 errors and: failed to delete 1 files
2023/10/21 22:59:06 Failed to delete with 2 errors: last error was: failed to delete 1 files

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[encrypted_media]
type = alias
remote = gdrive_media:

[gdrive_base]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive =

[gdrive_media]
type = crypt
remote = gdrive_base:Media
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

[google_photos]
type = google photos
read_only = true
token = XXX

[qnap1_base]
type = smb
host = XXX
user = XXX
pass = XXX

[qnap1_media]
type = alias
remote = qnap1_base:Media

[union_media]
type = union
upstreams = gdrive_media:/:ro qnap1_media:/

A log from the command that you were trying to run with the -vv flag

2023/10/21 23:04:16 DEBUG : /: Lookup: name="TV"
2023/10/21 23:04:16 DEBUG : /: >Lookup: node=TV/, err=<nil>
2023/10/21 23:04:16 DEBUG : TV/: Attr:
2023/10/21 23:04:16 DEBUG : TV/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2023/10/21 23:04:16 DEBUG : TV/: Lookup: name="Show Name"
2023/10/21 23:04:19 DEBUG : TV/: >Lookup: node=TV/Show Name/, err=<nil>
2023/10/21 23:04:19 DEBUG : TV/Show Name/: Attr:
2023/10/21 23:04:19 DEBUG : TV/Show Name/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2023/10/21 23:04:19 DEBUG : TV/Show Name/: Lookup: name="Season 01"
2023/10/21 23:04:19 DEBUG : TV/Show Name/: >Lookup: node=TV/Show Name/Season 01/, err=<nil>
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/: Attr:
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/: Lookup: name="Old File To Delete.mkv"
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/: >Lookup: node=TV/Show Name/Season 01/Old File To Delete.mkv, err=<nil>
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/Old File To Delete.mkv: Attr:
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/Old File To Delete.mkv: >Attr: a=valid=1s ino=0 size=1234567890 mode=-rw-rw-r--, err=<nil>
2023/10/21 23:04:19 DEBUG : /: Lookup: name="TV"
2023/10/21 23:04:19 DEBUG : /: >Lookup: node=TV/, err=<nil>
2023/10/21 23:04:19 DEBUG : TV/: Attr:
2023/10/21 23:04:19 DEBUG : TV/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/: Remove: name="Old File To Delete.mkv"
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/Old File To Delete.mkv: Remove:
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/Old File To Delete.mkv: File.Remove file error: permission denied
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/Old File To Delete.mkv: >Remove: err=permission denied
2023/10/21 23:04:19 DEBUG : TV/Show Name/Season 01/: >Remove: err=operation not permitted

[kapitainsky]

I do not think you can achieve it with rclone union but as it is something you say is supported by unionfs then obvious solution seems to use unionfs instead of rclone.

You can still use rclone to mount your gdrive_media and then combine it with SMB mount using unionfs.

Also worth to investigate mergerfs which allows even more customizations.

[hax0rwax0r]

Thanks for your reply --

I thought for sure that there was something I read a few months back that basically said don't use UnionFS (or alike) with rclone because reasons. That is how and when I discovered the union and combine backends for rclone. I could be totally wrong in my recollection, or maybe there was some half truth to it. To that point, the reason could have been quite simply why reinvent the wheel since rclone has union built in.

After re-reading what I had asked, I guess it sounded like I was saying that UnionFS had the whiteout functionality as fact like I knew something about it. For the record, I was purely referencing what the OP in the post I linked was stating and using it to describe what I was looking to do.

Per your recommendation, as far as I can tell, mergerfs doesn't support the functionality I am looking for. Their GitHub page states: "mergerfs does not support the copy-on-write (CoW) or whiteout behaviors found in aufs and overlayfs."

This sent me down the path of aufs and overlayfs. My antique CentOS 7.x server, even with running the newer 6.4.11 kernel via the third party kernel-ml package, does not appear to have aufs support. That might be something I need to pursue compiling and adding in myself if I think it's worthwhile and will work in my setup.

I tried overlayfs as that was already supported and Docker was already using it on my system today. I was writing up a nice step-by-step guide with all the commands to post as a follow-up reply here for future folks to find if they stumbled upon my post. Unfortunately, one of the last steps was to tell overlayfs to layer the lower and upper file systems together and thus I hit my dead end. Apparently when using a NFSv4 mount for the local "upper" (writable/local) file storage presented from my QNAP, I got the following error in dmesg:

[5268261.009770] overlayfs: upper fs does not support tmpfile.
[5268261.012391] overlayfs: upper fs does not support RENAME_WHITEOUT.
[5268261.012407] overlayfs: failed to set xattr on upper
[5268261.012409] overlayfs: upper fs missing required features.

I'm going to fiddle around with it more and see if there's a solution I can come up with. If all else fails, I bet I can make it work with an iSCSI LUN presented from my QNAP and treat it like a local block storage device.

I think the important takeaways at this moment, given this is the rclone forum and my issue seems certainly outside the scope of rclone, are:

1. I didn't miss something in the documentation that would have given me the functionality I was looking for with using rclone only.
2. It sounds like third party mergerfs/overlayfs/aufs/unionfs/etc. aren't necessarily problematic when used with rclone (might be a big assumption on my part).

I'll fiddle around with it and if I find a solution, I'll do my best to not be DenverCoder9, even if not rclone specific at this point, in case someone finds themselves on this post in the future.