Rclone Union issue with crypt

darkavenger · April 15, 2020, 9:25am

Hello,

I've recently decided to create a crypt drive alongside with my unencrypted rclone drive and to slowly start moving files to the crypt one.

The issue that I'm experiencing is that if I add an episode mid-season to a TV show that's already uploaded to the normal rclone drive is uploaded to the crypt, the new episode it not visible in Union. It seems like if a folder is present in both the normal rclone drive and the crypt it only allows you to use 1 and not both. The issues that's causing is that, Plex won't see the episodes I add to crypt unless the whole TV show is moved to crypt.

Is there any way around that, so that even if I have files in a folder called 'Test' both in normal rclone drive and a crypt it shows the combined files from both?

Edit: using version rclone v1.50.2

Thanks.

ncw · April 15, 2020, 3:06pm

Can you try the latest beta which has a re-worked union backend and see how that behaves?

darkavenger · April 15, 2020, 3:26pm

Will do and report back, thanks for the help.

darkavenger · April 16, 2020, 8:41am

Hi @ncw,

I've tried with the new beta (rclone v1.51.0-163-ge2bf9145-beta) but I still get the same issue. While I was investigating I noticed, that the issue is not happening for all files but it seems to be happening at random.

Here is an example below:

Crypt drive have the latest file:

Crypt1144×130 16.7 KB
Union doesn't see that file, so plex shows it as unavailable:
Union1145×610 171 KB

I've checked the permissions and they are the same. I've also tried with a different show and it worked fine, so I'm not sure why it breaks for some but not others.

Edit: I've also found that if I manually rename the missing file on Crypt and bring it back to original name, the file appears. Would that give any clues?

ncw · April 16, 2020, 3:26pm

That probably means that it was missing the change notify message that told the mount that a new file arrived.

Can you paste your mount settings? I expect if you were to wait as long as --dir-cache-time then the file would appear.

You could also try doing a vfs refresh with the rc https://rclone.org/rc/#vfs/refresh and see if that makes the file appear

If those things work then it narrows down the problem

darkavenger · April 16, 2020, 4:07pm

Hi,

Here are my mount settings (anything wrong with them? I basically copy-pasted them for all mounts). I haven't tried the vfs refresh as I haven't got it to happen again, but will try it when it happens.

Rclone:

#[Unit]
Description=Mount and cache Google drive to /media/Plex
After=syslog.target local-fs.target network.target
[Service]
Environment=RCLONEHOME=/home/XxX/.config/rclone
Environment=MOUNTTO=/home/XxX/Downloads/gdrive
Environment=LOGS=/home/XxX/logs
Environment=UPLOADS=/home/XxX/uploads
Type=simple
User=XxX
ExecStartPre=/bin/mkdir -p ${MOUNTTO}
ExecStartPre=/bin/mkdir -p ${LOGS}
ExecStartPre=/bin/mkdir -p ${UPLOADS}
ExecStart=/usr/bin/rclone mount \
  --log-file ${LOGS}/rclone.log \
  --log-level INFO \
  --dir-cache-time 96h \
  --poll-interval 15s \
  --buffer-size 128M \
  --timeout 1h \
  --umask 002 \
  --allow-other \
  --vfs-cache-max-age 72h \
  --vfs-cache-mode writes \
  --vfs-cache-max-size 100G \
  --vfs-read-chunk-size 128M \
  --vfs-read-chunk-size-limit 2G \
  --allow-other \
  --config ${RCLONEHOME}/rclone.conf \
gdrive: ${MOUNTTO}
ExecStop=/bin/fusermount -u -z ${MOUNTTO}
ExecStop=/bin/rmdir ${MOUNTTO}
Restart=always
[Install]
WantedBy=multi-user.target

Crypt:

#[Unit]
Description=Mount and cache Google drive to /media/Plex
After=syslog.target local-fs.target network.target
[Service]
Environment=RCLONEHOME=/home/XxX/.config/rclone
Environment=MOUNTTO=/home/XxX/Downloads/gcrypt
Environment=LOGS=/home/XxX/logs
Environment=UPLOADS=/home/XxX/uploadscry
Type=simple
User=XxX
ExecStartPre=/bin/mkdir -p ${MOUNTTO}
ExecStartPre=/bin/mkdir -p ${LOGS}
ExecStartPre=/bin/mkdir -p ${UPLOADS}
ExecStart=/usr/bin/rclone mount \
  --log-file ${LOGS}/rcrypt.log \
  --log-level INFO \
  --dir-cache-time 96h \
  --poll-interval 15s \
  --buffer-size 128M \
  --timeout 1h \
  --umask 002 \
  --allow-other \
  --vfs-cache-max-age 72h \
  --vfs-cache-mode writes \
  --vfs-cache-max-size 100G \
  --vfs-read-chunk-size 128M \
  --vfs-read-chunk-size-limit 2G \
  --allow-other \
  --config ${RCLONEHOME}/rclone.conf \
gcrypt: ${MOUNTTO}
ExecStop=/bin/fusermount -u -z ${MOUNTTO}
ExecStop=/bin/rmdir ${MOUNTTO}
Restart=always
[Install]
WantedBy=multi-user.target

Union:

#[Unit]
Description=Mount and cache Google drive to /media/Plex
After=syslog.target local-fs.target network.target
[Service]
Environment=RCLONEHOME=/home/XxX/.config/rclone
Environment=MOUNTTO=/home/XxX/Downloads/gunion
Environment=LOGS=/home/XxX/logs
Environment=UPLOADS=/home/XxX/uploadsuni
Type=simple
User=XxX
ExecStartPre=/bin/mkdir -p ${MOUNTTO}
ExecStartPre=/bin/mkdir -p ${LOGS}
ExecStartPre=/bin/mkdir -p ${UPLOADS}
ExecStart=/usr/bin/rclone mount \
  --log-file ${LOGS}/runion.log \
  --log-level INFO \
  --dir-cache-time 96h \
  --poll-interval 15s \
  --buffer-size 128M \
  --timeout 1h \
  --umask 002 \
  --allow-other \
  --vfs-cache-max-age 72h \
  --vfs-cache-mode writes \
  --vfs-cache-max-size 100G \
  --vfs-read-chunk-size 128M \
  --vfs-read-chunk-size-limit 2G \
  --allow-other \
  --config ${RCLONEHOME}/rclone.conf \
gunion: ${MOUNTTO}
ExecStop=/bin/fusermount -u -z ${MOUNTTO}
ExecStop=/bin/rmdir ${MOUNTTO}
Restart=always
[Install]
WantedBy=multi-user.target

ncw · April 16, 2020, 4:12pm

Thanks! You have a long --dir-cache-time which is fine as long as the change notify mechanism is working, so I'm wondering if that is broken.

If doing a vfs/refresh fixes it then that is almost certainly the problem and I'll dig in further.

darkavenger · April 16, 2020, 4:54pm

I was missing the --rc command on rclone and union (for some reason couldn't add to the crypt).

I did it and restarted the services. Did a refresh by using rclone rc vfs/refresh but it didn't fix the issue:

It's weird as I tried to refresh that specific directory on the union but it said it's not found:

Edit: I was wondering if I'm supposed to have the vfs settings on my union mount or I should remove that?

Thanks.

ncw · April 16, 2020, 9:10pm

You have to give the path relative to the root of the mount not the path on disk - I think that is the problem.

Can you paste your config with the secrets XXXed out? I just want to see how you have set up your remotes.

darkavenger · April 16, 2020, 9:26pm

The XxX is just my username, I haven't removed anything else from the config.

I was looking wondering if I remove the vfs from the mount settings for the union, if there would be any performance impact and if it would solve it, as it seems like a caching issue as you said.

Thanks for the help so far, appreciate it.

ncw · April 19, 2020, 9:14am

I means your rclone config file - I don't think you've pasted that yet?

You can try but I don't think it will make any difference.

darkavenger · April 19, 2020, 4:03pm

Here is the config (if I removed too much things let me know, will send them in a private message):

[gdrive]
type = drive
scope = drive
token = {"access_token":"XXXXXXXXXXXXXXXXXXXXXXXXXXX","token_type":"Bearer","refresh_token":"XXXXXXXXXXXXXXXXXXXXXXXXXXX","expiry":"2020-04-19T18:51:35.971808404+02:00"}
client_secret = XXXXXXXXXXXXXXX
client_id = XXXXXXXXXXXXXXXXXXXXXXXXXXX
root_folder_id = XXXXXXXXXXXXXX

[gcache]
type = cache
remote = gdrive:
chunk_size = 5M
info_age = 1h0m0s
chunk_total_size = 100G

[gcrypt]
type = crypt
remote = gdrive:All/Encrypt
filename_encryption = standard
directory_name_encryption = true
password = XXXXXXXXXXXXXXXXXXXXXXXXXXX
password2 = XXXXXXXXXXXXXXXXXXXXXXXXXXX

[gunion]
type = union
remotes = gdrive:All gcrypt:

I'm stuck, I think I will just stick with the unencrypted one for now instead of using union. It should be okay for now until I copy over everything until the end of the year to the crypt one.

On the union mount when I try to rename a file or rename it I get a 'Input/Output' error (works if I directly rename it from the rclone drive itself), I get a similar error when I try to send a file to that 'Trash'. In the folder in the terminal it shows me (btw the red part it only my username (non root user):

In the log it shows 'permissions denied' but I do have access as the username is the same as on the files, not sure why:

ncw · April 20, 2020, 2:37pm

Thanks!

That is quite complicated that config...

I think you are being bitten by directory cache invalidation somewhere...

The error in the log "Permission denied" is probably being generated by the union backend itself. I suspect when you try to rename stuff that is on gdrive:All it will give that error but it will be ok on gcrypt:.

The "no such file or directory" in your listing is strange, that indicates union has got confused somehow.

darkavenger · April 20, 2020, 3:53pm

Yes, I can confirm that renaming items on the crypt causes no such issue, so only renaming on crypt works at the moment when I use Union. What do you reckon is the best way to simplify my config?

Thanks.

ncw · April 21, 2020, 3:35pm

I think this is probably a bug... I haven't managed to replicate it though. Any chance you could give me instructions to reproduce? As precisely as possible preferably with shell commands! Thanks!

There is one thing that jumps out to me is that your crypted files and non crypted files overlap. I'm not sure that is causing a problem or not, but I would fix it so they don't overlap.

darkavenger · April 22, 2020, 9:34am

Hello,

Will do that as soon as I can, just to check, is it possible this is caused by the following (mentioned in the rclone config for union)

The last remote is used to write to.

From what it seems when using Union, the 1st location doesn't have sufficient permissions to create/delete files, the 2nd one does. Is it possible that's the cause of the issues?

I'll post the rest whenever I can. Thanks for the help.

ncw · April 22, 2020, 12:49pm

Yes that is an accurate summary. I'm assuming that you can write to the 1st location? If not then suffix it with :ro and union won't attempt to write to it.

darkavenger · April 23, 2020, 3:01pm

Hi,

I managed to sort my issue. When using mergerfs I don't get the same issue and can edit files in both rclone and crypt. Also when transferring files from local to crypt and merging with an unencrypted file, they are appearing immediately. I'll still try to get you the steps I used for union if you want it (whenever I get the time I will sit and write it down), so that we can get that issue sorted.

Thanks for the help.

ncw · April 23, 2020, 6:56pm

Great

That would be great, thanks.

system · June 23, 2020, 2:56pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.