Rclone VFS and MergerFS Setup on Ultraseedbox - 2020 06

Help and Support
1

Linux Newbie for about the last 4weeks.

I recently dipped my toes into the world of seedboxes and have moved onto Ultraseedbox in the last week. With all the normal app setup out of the way after testing and good success. I felt ready for the next task of getting my rclone to work with my google drive.

I followed the guidance on USB = https://kb.ultraseedbox.com/display/DOC/rclone+-+Rclone+VFS+and+MergerFS+Setup = which worked great (up to a point)

ATM I am stuck at pretty much the same place as this post = Rclone VFS and MergerFS Setup on Ultraseedbox. Reading that post it became clear that the --allow--other was causing the issue.

I've searched and searched this forum, Google, etc. and tried for many hours to get this to work but definately outside my depth, hence why I need your guys help. @Animosity022 I know you are the beast in this area and would like if you could have a look.

rclone version:
rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp )

Running this command from this guide = https://kb.ultraseedbox.com/display/DOC/rclone+-+Rclone+VFS+and+MergerFS+Setup

systemctl --user enable --now rclone-vfs && systemctl --user enable --now mergerfs

I get this error:

systemctl --user enable --now rclone-vfs && systemctl --user enable --now mergerfs
Job for rclone-vfs.service failed because the control process exited with error code.
See "systemctl --user status rclone-vfs.service" and "journalctl --user -xe" for details.

What I know from the previous post ( Rclone VFS and MergerFS Setup on Ultraseedbox) is that the root, seems to cause the issue in my rclone-vfs.service file. From what I could understand is that --allow--other needs to be amended.

Text Inside of my rclone-vfs.service file (I have added more to this file with Log info, which still doesnt seem to give me any file after running the command = systemctl --user enable --now rclone-vfs && systemctl --user enable --now mergerfs

rclone-vfs.service file

[Unit]
Description=RClone VFS Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
KillMode=none
Environment=GOMAXPROCS=2
ExecStart=/home17/goflix/bin/rclone mount gdrive: /home17/goflix/Stuff/Mount
--allow-other
--buffer-size 512M
--config /home17/goflix/.config/rclone/rclone.conf
--use-mmap
--dir-cache-time 168h
--log-level INFO
--log-file /home17/goflix/scripts/rclone.log
--timeout 1h
--umask 002
--poll-interval=15s
--vfs-cache-mode writes
--vfs-read-chunk-size 64M
--vfs-read-chunk-size-limit 2048M
--tpslimit 10
--tpslimit-burst 10
ExecStop=/bin/fusermount -uz /home17/goflix/Stuff/Mount
Restart=on-failure

[Install]
WantedBy=default.target

Few other things to mention

  1. Not able to see my gdrive files or folders in MergerFS directory = /home17/MergerFS

  2. When running the below command, I am not able to see my service running (however see all the users on USB service running = not sure if this should be happening but not able to see mine)

mount | grep rclone

2

You'd want to remove that. Allow Other is used for a single server solution and that means anyone else on the server can see your files, which is why it's off on the seedbox by default and commented out of /etc/fuse.conf.

I use allow other because I have my own server at home and have no other users.

Just try starting rclone first before you try to start mergerfs as well and validate that's working.

If you check the /home17/goflix/scripts/rclone.log file, it should complain I'd imagine about allow-other. If not, post the contents of that file.

3

Hey man ,

Thanks for getting back so quick on this, please see below the information from the log file.

Log files;

2020/06/06 15:14:16 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:14:16 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:14:16 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:14:16 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:14:17 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:14:17 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:14:17 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:14:17 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:14:17 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:14:17 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:16:06 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:16:06 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:16:06 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:16:06 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:16:06 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:16:06 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:16:06 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:16:06 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 15:16:07 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 15:16:07 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 16:44:31 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 16:44:31 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 16:44:32 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 16:44:32 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 16:44:32 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 16:44:32 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 16:44:32 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 16:44:32 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory
2020/06/06 16:44:32 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 10
2020/06/06 16:44:32 Fatal error: Can not open: /home17/goflix/Stuff/Mount: open /home17/goflix/Stuff/Mount: no such file or directory

Soooo I made a newbie mistake in that my Mount folder inside of the Stuff directory was actually called Rclone. I have amended that to Mount. Which caused the following command to work (i think)

systemctl --user enable --now rclone-vfs && systemctl --user enable --now mergerfs

To kick out = assume this is working right now.

systemctl --user enable --now rclone-vfs && systemctl --user enable --now mergerfs
Created symlink /home17/goflix/.config/systemd/user/default.target.wants/mergerfs.service → /home17/goflix/.config/systemd/user/mergerfs.service.

Checked the MergerFS and that is now a Mirror of my shared drive on Google drive (Very Happy!!!)

Assume I am good to start looking at the script portion now to drive the uploading?

How would you suggest I do a test to ensure it works right?

4

Nice!

So I keep things simple, I'd just test with a small text or something.

For me:

/gmedia is my merger mount
/GD is rclone
/local is my local disk

So my simple test, I copy a file, see it is in my local disk, one rclone to move it and validate it's on my rclone. Takes 1 minute for polling to pick it up by default or I have mine set to 15 seconds.

felix@gemini:/gmedia$ cp /etc/hosts .
felix@gemini:/gmedia$ ls -al hosts
-rw-r--r-- 1 felix felix 413 Jun  6 11:46 hosts
felix@gemini:/gmedia$ ls -al /local/hosts
-rw-r--r-- 1 felix felix 413 Jun  6 11:46 /local/hosts
felix@gemini:/gmedia$ ls /GD/hosts
ls: cannot access '/GD/hosts': No such file or directory
felix@gemini:/gmedia$ rclone move /local/hosts gcrypt:
felix@gemini:/gmedia$ date
Sat 06 Jun 2020 11:46:42 AM EDT
felix@gemini:/gmedia$ ls /GD/hosts
ls: cannot access '/GD/hosts': No such file or directory
felix@gemini:/gmedia$ ls /GD/hosts
/GD/hosts
felix@gemini:/gmedia$ date
Sat 06 Jun 2020 11:46:58 AM EDT
5

o/

@DanMan,you can check out this link right here to set the uploading part of the script (from the new docs which was working on): https://docs.usbx.me/books/rclone/page/rclone-vfs-and-mergerfs-setup#bkmrk-systemd-uploader

Just make sure you change the paths to what you have. Also to test it out, in addition to what animosity said, if you want to test out the systemd timer you can just copy from the ExecStart= to second to the last flag (which I belive is --drive-stop-on-upload-limit) and paste it in your terminal. Below's what you'd want to paste to test out (replacing paths to whatever you have.

/homexx/yyyyy/bin/rclone move /homexx/yyyyy/Stuff/Local/ gdrive: \
    --config=/homexx/yyyyy/.config/rclone/rclone.conf \
    --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36' \
    --drive-chunk-size 64M \
    --tpslimit 5 \
    --drive-acknowledge-abuse=true \
    -vvv \
    --delete-empty-src-dirs \
    --fast-list \
    --bwlimit=8M \
    --use-mmap \
    --transfers=2 \
    --checkers=4 \
    --drive-stop-on-upload-limit

If you got any questions, feel free to tag me here or ask on the USB Discord Server... or not. Whatever you like mate lol

@Animosity022, thanks for the heads up for the --allow-other flag which was on our service files. On our setup, clients cannot access other users home slots other than their own and it is commented out in fuse.conf as you mentioned. But just to be safe, I've removed it from the service files, in addition to adding the new --drive-stop-on-upload-limit introduced from 1.51 for the upload scripts.

Cheers on your work, both rclone and the homescripts.

1 Like
6

You seem to be an actual human being or a superb AI so I made it so you can post links :slight_smile:

1 Like
7

@Animosity022 dunno man I'm just a doge LOL. Cheers.

8

Hey guys (@Animosity022 and @Xan) = #Legends!

Appreciate all the support here!

I am a little lost on steps to check the upload, @Xan will catch you on the Discord USB.

Just to check, I assume if I just take the standard upload script it will run it once a day, right? If thats the case I don't think I see a reason to have this quicker than that (unless any more insights can be shared).

Beyond that it seems the files move from MergerFS into the Stuff/Local when copied to MergerFS. Ie only the ones not yet uploaded to Google Drive. Assume the Script will do the rest tonight and move this to the cloud, assuming the standard script does the upload without any timing tweaks.

Thanks in advance guys.

9

Yep. If you use the systemd uploader/timer by default, it'll execute daily +-30 minutes at CET/CEST. You can also change it too if you'd like. The Arch Linux wiki has some nifty examples on how to do so: https://wiki.archlinux.org/index.php/Systemd/Timers#Examples

I'd suggest sticking to whatever defaults we have. You're free to change any of the ones on the systemd files if you'd like but I suggest not to. The ones on the docs works great in terms of streaming, uploading files within the Google Drive Upload limits and such. It's gonna be down to your peering basically.

It's not fun if you execute it like every 30 minutes with high values such as more than 5 transfers and removed the bwlimit which trashes the IO and we'll get complaints from other users in your disk

(unless you're in the dedi disk plan or on NVMe, to which case, go ham...within the ram limits)

:smile:

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.