Sick Rage post process to UnionFS volume actually goes directly to rclone mount volume?

Help and Support
1

I have an rclone mount /mnt/remote/storage and a UnionFS mount /mnt/storage (-o cow /mnt/local/storage=RW:/mnt/remote/storage=RW). Sick Rage shows are configured with /mnt/storage/FOO as their directory.

When I post process (initiated from NZBget/nzbToMedia or manually from Sick Rage) a download, Sick Rage log reports that it is trying to move a file from /mnt/local/nzbget/completed/TV Shows/FOO/... to /mnt/storage/FOO/...

I am expecting this to be a local operation, resulting in the file being saved to /mnt/local/storage/TV Shows/FOO/..., so that I can then move it with rclone move /mnt/local/storage/ remote: (because it seems rclone mount is not very stable for writes)

BUT, the rclone mount log is showing that Sick Rage is attempting to write the file directly to the mount. This is slower than a local write, and has proven to be unstable.

How is this possible? In the terminal if I do cp /mnt/local/nzbget/completed/TV Shows/FOO/... /mnt/storage/TV Shows/FOO/..., it is a local operation as expected, it’s quick, and my other rclone move script will eventually move it to remote: as expected.

Why and how is Sick Rage post processing writing directly to the rclone mount, when all it knows is /mnt/storage?

2

You made merged folder writable to both local and remote.
you need to change ;
/mnt/storage (-o cow /mnt/local/storage=RW:/mnt/remote/storage=RO)

3

Why would that make a difference? I want /mnt/remote/storage to be RW so that Sick Rage can delete files.

When I test this RW/RW unionfs mount from the terminal, it seems to work as expected. Writes to /mng/storage go to /mnt/local/storage (and are eventually moved via rclone move to /mnt/remote/storage).

Only the move triggered by Sick Rage during post processing, which says it is moving to /mnt/storage/..., actually goes directly to the mount.

I even looked at the Sick Rage source code and I see it is using shutil.move to to the move. When I execute this in an interactive Python shell, it works as expected also.

4

Just tested RW/RO and it seems you are right. This makes Sick Rage move to local storage. I guess then I need to do rclone sync /mnt/storage remote: to sync deletes to remote storage, too. Strange that moving files to /mnt/storage from terminal and Python interactive shell seems to work as expected?

5

How would that work? It’s kinda messing with my mind thinking about it…you’re going to be reading from the drive you’re sync’ing to. mnt/storage is your UnionFS mount which points to both local and remote storage, but you’re syncing with the same remote storage :slight_smile: I’d love to know how that works out for you.

6

I do the same thing. In essence it takes and applies the modifications you made on the union to the bottom layer. Remember when you delete something in a union and that file exists in the bottom layer, it doesn’t actually delete the file in that layer. It only uses a white out file so it isn’t seen. When you run this it will make the changes to the bottom layer based on what the union looks like at that point of time.

What nice about it is it I manually delete a file for the top layer only (outside of the union) it will preserve that file on the bottom layer. I use this technique to keep a local copy of recent data and archive the larger set to the bottom layer. My to respond layer is not exactly like my bottom layer on purpose. The top layer is a subset of my cloud layer. :slight_smile:

/data/Media <= union
/data/Media1 <= RW layer
/data/Media2 <= Cloud layer (marked RO in unionfs but mounted RW)

This is my processing:
CLOUD_UNION=/data/Media
CLOUD_FSDIR=/data/Media2

echo +---------------------------------------+
echo  Delete empty directories from $CLOUD_FSDIR
echo +---------------------------------------+

ls -d $CLOUD_FSDIR/* > /dev/null 2>&1 && \
ls -f $CLOUD_FSDIR/.imhere > /dev/null 2>&1 && \
 rclone \
   -v \
   --exclude=/.imhere \
   --exclude=/.unionfs/** \
 rmdirs $CLOUD_FSDIR 2>&1 | egrep -v "$CRYPTLOGFILTER"

echo +---------------------------------------+
echo  Apply change deltas to cloud volume $CLOUD_FSDIR
echo +---------------------------------------+
ls -d /data/Media1/* > /dev/null 2>&1 && \
ls -f $CLOUD_FSDIR/.imhere > /dev/null 2>&1 && \
 rclone \
   -v \
   --size-only \
   --exclude=/.imhere \
   --exclude=/.unionfs/** \
   --exclude=/Videos/Kids-*/** \
 copy /data/Media1 $CLOUD_FSDIR 2>&1 | egrep -v "$CRYPTLOGFILTER"

echo +---------------------------------------+
echo Age/remove things that have been HIDDEN on $CLOUD_UNION
echo +---------------------------------------+
ls -d $CLOUD_UNION/* > /dev/null 2>&1 && \
ls -f $CLOUD_FSDIR/.imhere > /dev/null 2>&1 && \
 rclone \
   -v \
   --size-only \
   --exclude=/.imhere \
   --exclude=/.unionfs/** \
   --exclude=/Videos/Kids-*/** \
 sync $CLOUD_UNION $CLOUD_FSDIR 2>&1 | egrep -v "$CRYPTLOGFILTER"

echo +---------------------------------------+
echo Doing checksums for NFOs
echo +---------------------------------------+
ls -d /data/Media1/* > /dev/null 2>&1 && \
ls -f $CLOUD_FSDIR/.imhere > /dev/null 2>&1 && \
 rclone \
   -v \
   --checksum \
   --include=*.nfo \
 copy /data/Media1/Videos $CLOUD_FSDIR/Videos 2>&1  | egrep -v "$CRYPTLOGFILTER"