Dropbox mount - Servicefile assistance :)

What is the problem you are having with rclone?

I dont actual have a problem - Im moving my data from Googledrive to Dropbox, and im curious if my servicefile is missing details so I dont hit rate limits for Plexmedia server.
I read something in the forum about Dropbox with tpslimits and tpslimit-burst which I wasnt using for my Googledrive - it this needed for Dropbox?

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-153-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.5
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox ( Data not moved yet, but preparing for my Plexmedia server. )

Here is my rclonemount.service file:

[Unit]
Description=RClone Service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/plex/.config/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount Googlecrypt: /home/plex/mnt \
--allow-other \
--dir-cache-time 1000h \
--umask 002 \
--bwlimit-file 50M
ExecStop=/bin/fusermount -uz /home/plex/mnt
Restart=on-abort
User=plex
Group=plex

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[Dropbox]
type = dropbox
client_id = xxxxx
client_secret = xxxxxxx
token = {"access_token":"xxxxx","expiry":"2023-07-13T18:10:32.049833+02:00"}

[Dropboxcrypt]
type = crypt
remote = Dropbox:Private
password = xxxxxx

A log from the command with the -vv flag

No log yet - system not online :) 

Thanks :slight_smile:

Morph

2 Likes

Thank you so much @kapitainsky

Will read it through :slight_smile:

@kapitainsky

Im a bit confused - im trying to implement a new setup via the link you provided me

In this setup Ani writes - that he has removed Mergerfs .

Under "My workflow" he write: "That was the reason to remove mergerfs and the upload script...."

But further down he writes he uses mergerfs . Under "My Rclone looks like: Rclone.conf"... :

They are all mounted via systemd scripts, rclone is mounted first followed by the mergerfs mount. ?

But his setup don't show the mergerfs.service file ? I don't see anything by with mergerfs under https://github.com/animosity22/homescripts/tree/master/systemd ?

Im just trying to understand the setup since im moving from a Google drive setup with 1 mount point , mergerfs and upload script.

As I understand from Ani in this guide, Dropbox use another API than Google drive, so not to overload the api hits, he made several mount points .

He seems to download to a folder called /data which then copy to /media/tv or /media/movies , but where does mergerfs appear in this setup? :slight_smile:

I want to move away from the old upload script and try use rclone to upload automatically :slight_smile:

In my setup I have 1 drive only 1 TB SSD:
Rutorrent download all the files to /home/plex/rutorrent/download . Once downloaded Sonarr/Radarr copies the file to a /home/plex/move folder . Sonarr/Radarr root folders are set to /home/plex/media/ .
The script upload the files after 15 minuts. So if Sonarr/Radarr has copied it looks something like /home/plex/move/Movie/title 2023/ or /home/plex/move/TV-shows/Title/season etc. Once uploaded - plex see the data in /home/plex/mnt/

My rclonemount.service:

[Unit]
Description=RClone Service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/plex/.config/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount Googlecrypt: /home/plex/mnt \
--allow-other \
--dir-cache-time 1000h \
--umask 002 \
--bwlimit-file 32M
ExecStop=/bin/fusermount -uz /home/plex/mnt
Restart=on-abort
User=plex
Group=plex

[Install]
WantedBy=default.target

My mergerfs.service :

[Unit]
Description=mergerfs mount
Requires=rclonemount.service
After=rclonemount.service
RequiresMountsFor=/home/plex/mnt

[Service]
Type=forking
ExecStart=/usr/bin/mergerfs /home/plex/move:/home/plex/mnt /home/plex/media -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full
KillMode=process
Restart=on-failure

Rclone-upload script:

#!/bin/bash
EXCLUDE=/home/plex/scripts/exclude-file.txt
LOGFILE=/home/plex/logs/rclone-upload.log
FROM=/home/plex/move
TO=Googlecrypt:/

if pidof -o %PPID -x "rclone-upload.sh"; then
   exit 1
fi
# CHECK FOR FILES IN FROM FOLDER THAT ARE OLDER THAN 15 MINUTES
if find $FROM* -type f -mmin +15 | read
  then
  start=$(date +'%s')
  echo "$(date "+%d.%m.%Y %T")  Rclone upload started" | tee -a $LOGFILE
  # MOVE FILES OLDER THAN 15 MINUTES
  rclone move "$FROM" "$TO" -v --drive-chunk-size=32M --delete-empty-src-dirs --transfers=5 --bwlimit 25M --checkers=5 --progress --min-age 15m --log-file=$LOGFILE
  echo "$(date "+%d.%m.%Y %T") Rclone upload finished in $(($(date +'%s') - $start)) seconds" | tee -a $LOGFILE
fi
exit

What am I missing here regarding his mergerfs? :slight_smile:

Kind Regards

Morphy

Ok I appreciate that you read Ani github setup page (it is good source IMO) - but I have no idea what it is overall and what are details of all setup design decisions. I do not use plex, neither mergerfs but I find his dropbox serviced mount file very comprehensive and instructive. It covers all aspects of well thought solution and contains comments to clarify what for and why particular options are used. It is not one fit all but gives you great start to setup dropbox mount. So it perfectly fit into your original question.

@Animosity022 - would you like to shed some light on @Morphy questions?

1 Like

@Animosity022 all help would be appreciated :slight_smile:

I cant get my setup to work with cache and rclone upload, dont know what im missing here.
Guessing that im missing a mergerfs systemd , but which setup I dont know :frowning:

Thanks

Morphy

Got it sorted - working :slight_smile:

Your guide just confused me a bit when you wrote about Mergerfs not being used, and further down you are using it :slight_smile: .

All up and running - thanks .

/Morphy

1 Like

I was on holiday so just getting back. I don't use mergerfs, probably a typo somewhere that I need to clean up.

1 Like

Thanks @Animosity022 and @kapitainsky

When I start to transfer from Gdrive to Dropbox , are there any speciel settings I need to be aware of so I dont hit api limits? —tpslimit=12 and —tpslimit-burst=0
—transfers=4 —checkers=4 is that ok? Or can you recommmend a better setting for a faster transfer?

I have added the app with own client info and pass

I need to transfer aprox 20TB from Gdrive to Dropbox. Thinking about a Hetzner VPS .

Thank you :slight_smile:

Hmm im a bit worried , why can my copy process only see 5.7TB out of 20TB on my Googledrive?

Transferred: 2.244 GiB / 5.726 TiB, 0%, 9.958 MiB/s, ETA 6d23h25m
Transferred: 11 / 10027, 0%
Elapsed time: 3m54.4s

Im using this command:

rclone copy --drive-chunk-size=32M --transfers=8 --checkers=8 --tpslimit=12 --ignore-existing --progress

This is good start - if you see errors than lower transfers.

Because rclone only sees 10000 files ahead..

--max-backlog int Maximum number of objects in sync or check backlog (default 10000)

you can change it but it will use more memory

1 Like

Thank you @kapitainsky :slight_smile:

Just curious - are they any speed limitations or should I just go full speed?

your "full speed" is a limit. Only limitation I am aware of is that you can "only" download 10TB per day from gdrive.

You can add e.g. --max-transfer 9T to your transfer. But good luck if you manage to copy so fast that you have to worry about it.

1 Like

Thank you again :slight_smile: . Im hitting 123MB/s via Hetzner VPS so far. :slight_smile:

@kapitainsky

Hmm seems to have major problems creating crypt.

First I have made my drive connection , second the crypt:

[Dropbox]
type = dropbox
client_id = xxxxx
client_secret = xxxxx
token = {"access_token":"xxxxxxxxW","token_type":"bearer","refresh_token":"xxxxxxxxxxxx","expiry":"2023-07-16T20:51:48.36098+02:00"}

[dcrypt-ani]
type = crypt
remote = Dropbox:Private/Animations
password = xxxx

[dcrypt-film]
type = crypt
remote = Dropbox:Private/Film
password = xxxxxx

If I run rclone lsd and both crypts - I get the same error:

Rclone lsd dcrypt-film:


plex@webservices**:**~/.config/rclone**$ rclone -vv lsd dcrypt-film:

2023/07/16 15:24:07 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "-vv" "lsd" "dcrypt-film:"]

2023/07/16 15:24:07 DEBUG : Creating backend with remote "dcrypt-film:"

2023/07/16 15:24:07 DEBUG : Using config file from "/home/plex/.config/rclone/rclone.conf"

2023/07/16 15:24:07 DEBUG : Creating backend with remote "Dropbox:Private/Film"

2023/07/16 15:24:07 ERROR : : error listing: directory not found

2023/07/16 15:24:07 DEBUG : 7 go routines active

2023/07/16 15:24:07 INFO : Dropbox root 'Private/Film': Committing uploads - please wait...

2023/07/16 15:24:07 Failed to lsd with 2 errors: last error was: directory not found

My crypt folder in Dropbox is Private under that one Film , but I can't seem to decrypt?

If I only add Private to the crypt , its working fine?


plex@webservices:~/.config/rclone$ rclone lsd -vv dcrypt-film:
2023/07/16 15:26:45 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "lsd" "-vv" "dcrypt-film:"]
2023/07/16 15:26:45 DEBUG : Creating backend with remote "dcrypt-film:"
2023/07/16 15:26:45 DEBUG : Using config file from "/home/plex/.config/rclone/rclone.conf"
2023/07/16 15:26:45 DEBUG : Creating backend with remote "Dropbox:Private"
          -1 2000-01-01 00:00:00        -1 Animations
          -1 2000-01-01 00:00:00        -1 Film

2023/07/16 15:26:45 DEBUG : 7 go routines active
2023/07/16 15:26:45 INFO  : Dropbox root 'Private': Committing uploads - please wait...

What am I doing wrong?

When creating the crypt I write:

Dropbox:Private/Film 

But I can't get it to list the content of the folder? :confused:

You are doing all correct. Trying to access data in Dropbox:Private/Film and there is no Film folder there.

You already uploaded some data. Just use the same config. Or you want to change something?

Or you want now another crypt in Private/Films? So you have to create new folder.

you can see your folders by running:

rclone tree Dropbox: --dirs-only

Im trying to "copy" Ani setup , but just with my own folders .
As I understand correctly - The more connections/mounts - the less chance I have to hit the API limits. Ani separates his apps in Dropbox. He use one for uploads, one for movies and tv . (and here is probably the part I don't understand :frowning: )

As I see it, he creates:

DB = Connection to Dropbox
dcrypt=
Remote: dbcrypt:media ( in my case its Private )

But then he have two more

DBMovies : Connection to Dropbox
dbcrypt-movies
Remote DBMovies:Movies

DBTV : Connection to Dropbox
dbcrypt-tv
DBTV:TV

If you look at the service files for his mounts , he has one for movies and one for tv. But I don't see the mount for [dbcrypt] anywhere in his service files? Which probably is the part I don't quite understand.

As I see it - He has a remote and a decrypt, and then again the same for Movies and TV ?

Im so confused :slight_smile:

He has

[DBMovies]
type = dropbox

[dbcrypt-movies]
type = crypt
remote = DBMovies:Movies

and

[DBTV]
type = dropbox

[dbcrypt-tv]
type = crypt
remote = DBTV:TV

you copy everything to:

[Dropbox]
type = dropbox

[Dropboxcrypt]
type = crypt
remote = Dropbox:Private

so this is what you have to use.

If you want two crypts you have to create them from the start e.g.:

[Dropbox]
type = dropbox

[DropboxCryptFilm]
type = crypt
remote = Dropbox:Private/Film

[DropboxCryptAnimations]
type = crypt
remote = Dropbox:Private/Animations

and then copy from google films to films and animations to animation

1 Like

Ok I will try again :slight_smile:

Thank you so much for your help @kapitainsky

There is another possible approach with only one crypt folder to use more clientID for dropbox.

[Dropbox1]
type = dropbox

[Dropbox2]
type = dropbox

[DropboxCryptFilms]
type = crypt
remote = Dropbox1:Private

[DropboxCryptAnimations]
type = crypt
remote = Dropbox2:Private

but now you mount it like this:

rclone mount DropboxCryptFilms: mountpointFilms --exclude /Animations/**
rclone mount DropboxCryptAnimations: mountpointAnimations --exclude /Films/**