Syncing External Changes in remote to Rclone mount

sungold · February 5, 2023, 2:58pm

What is the problem you are having with rclone?

Hi I'm running a onedrive remote and noticed if I make a change using the web client of onedrive like deleting a file, it doesn't get synced to my mount. I'm using the mount for Plex and with mergerfs combining a local and cloud folder, but I think that's irrelevant to the issue. I've followed the onedrive docs to use my own client id and other steps listed to prevent my API requests getting throttled. Also if you have any suggestions to improve my setup, please share them.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-58-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive for Buisness

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

[Unit]
Description=RClone Mount Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
TimeoutStopSec=60

ExecStart=/usr/bin/rclone mount \
  --config /home/sungold/.config/rclone/rclone.conf \
  --log-level=INFO --log-file=/home/sungold/rclone/rclone-mount.log \
  --user-agent "ISV|rclone.org|rclone/v1.55.1" \
  --umask=022 --gid=1000 --uid=1000 \
  --allow-other \
  --poll-interval=20s \
  --dir-cache-time=1000h \
  --cache-dir=/home/sungold/rclone/cache \
  --vfs-cache-mode=full \
  --vfs-cache-max-size=150G \
  --vfs-cache-max-age=720h \
  --vfs-read-ahead=100M \
  --use-mmap \
  onedrive: /home/sungold/rclone/cloud

ExecStop=/bin/fusermount -uz /home/sungold/rclone/cloud
Restart=on-failure

[Install]
WantedBy=multi-user.target

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
client_id = XXXXXXXXXXXXXXXXXXXXXXXXX
client_secret = XXXXXXXXXXXXXXXXXXX
auth_url = https://login.microsoftonline.com/XXXXXXXXXXX/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/XXXXXXXXXXX/oauth2/v2.0/token
no_versions = true
token = {"access_token":"XXXXXX"}
drive_id = XXXXXXXXXX
drive_type = business

A log from the command with the `-vv` flag

2023/02/05 14:54:56 DEBUG : rclone: Version "v1.61.1" starting with parameters ["/usr/bin/rclone" "mount" "--config" "/home/sungold/.config/rclone/rclone.conf" "--log-file=/home/sungold/rclone/rclone-mount.log" "--user-agent" "ISV|rclone.org|rclone/v1.55.1" "--umask=022" "--gid=1000" "--uid=1000" "--allow-other" "--poll-interval=20s" "--dir-cache-time=1000h" "--cache-dir=/home/sungold/rclone/cache" "--vfs-cache-mode=full" "--vfs-cache-max-size=150G" "--vfs-cache-max-age=720h" "--vfs-read-ahead=100M" "--use-mmap" "-vv" "onedrive:" "/home/sungold/rclone/cloud"]
2023/02/05 14:54:56 DEBUG : Creating backend with remote "onedrive:"
2023/02/05 14:54:56 DEBUG : Using config file from "/home/sungold/.config/rclone/rclone.conf"
2023/02/05 14:54:59 DEBUG : OneDrive root '': Next delta token is: XXXXXXXXXXXX
2023/02/05 14:54:59 DEBUG : vfs cache: root is "/home/sungold/rclone/cache"
2023/02/05 14:54:59 DEBUG : vfs cache: data root is "/home/sungold/rclone/cache/vfs/onedrive"
2023/02/05 14:54:59 DEBUG : vfs cache: metadata root is "/home/sungold/rclone/cache/vfsMeta/onedrive"
2023/02/05 14:54:59 DEBUG : Creating backend with remote "/home/sungold/rclone/cache/vfs/onedrive/"
2023/02/05 14:54:59 DEBUG : fs cache: renaming cache item "/home/sungold/rclone/cache/vfs/onedrive/" to be canonical "/home/sungold/rclone/cache/vfs/onedrive"
2023/02/05 14:54:59 DEBUG : Creating backend with remote "/home/sungold/rclone/cache/vfsMeta/onedrive/"
2023/02/05 14:54:59 DEBUG : fs cache: renaming cache item "/home/sungold/rclone/cache/vfsMeta/onedrive/" to be canonical "/home/sungold/rclone/cache/vfsMeta/onedrive"
2023/02/05 14:54:59 DEBUG : OneDrive root '': Mounting on "/home/sungold/rclone/cloud"
2023/02/05 14:54:59 DEBUG : vfs cache RemoveNotInUse (maxAge=2592000000000000, emptyOnly=false): item media/movies/Tron Legacy (2010) [1080p] {5.1}/Tron.Legacy.BluRay.1080p.x264.5.1.Judas.mp4 not removed, freed 0 bytes
2023/02/05 14:54:59 DEBUG : vfs cache RemoveNotInUse (maxAge=2592000000000000, emptyOnly=false): item mountcheck not removed, freed 0 bytes
2023/02/05 14:54:59 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 0, to upload 0, uploading 0, total size 34.998Mi (was 34.998Mi)
2023/02/05 14:54:59 DEBUG : : Root: 
2023/02/05 14:54:59 DEBUG : : >Root: node=/, err=<nil>

Running the mount again with -vv made the deleted file change sync.

Animosity022 · February 5, 2023, 3:05pm

Let it run a bit longer as it has to poll for new changes as it is not instant.

sungold · February 5, 2023, 3:10pm

I deleted a file using the OneDrive Web Client about 13 hours ago and it was still present when I creating this post. I restarted the service with -vv flag to get logs and that made the change finally sync. I believe my logs are not helpful in this case, I'm going to reply with the correct logs.

Animosity022 · February 5, 2023, 3:12pm

Should be pretty easy to test.

Just run through a simple pattern.

Stop the mount, start in debug.

rlcone copy somefile.txt onedrive:
rclone lsl onedrive:somefile.txt -vv

Wait a few minutes and share the log.

sungold · February 5, 2023, 3:33pm

I first made a file using touch /home/sungold/rclone/cloud/testfile1 and deleted it 30 seconds later from OneDrive web client. After waiting 4-5 mins, I checked /home/sungold/rclone/cloud using ls and found the file present.
So I made another file using touch /home/sungold/rclone/cloud/testfile2 and before deleting it, I ran:

sungold@poseidon:~$ ls rclone/cloud/
media  mountcheck  testfile1  testfile2

So this time I again deleted the file from onedrive web and this time both of the testfiles had been deleted.

sungold@poseidon:~$ ls rclone/cloud/
media  mountcheck

I believe my mergerfs setup may indeed be the cause of problem, so here's it's command.

[Unit]
Description=MergerFS Mount
Requires=rclone-normal.service
After=rclone-normal.service

[Service]
Type=forking
ExecStart=/usr/bin/mergerfs /home/sungold/rclone/local:/home/sungold/rclone/cloud /home/sungold/rclone/media -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full
ExecStop=/usr/bin/fusermount -uz /home/sungold/rclone/media
KillMode=none
Restart=on-failure

[Install]
WantedBy=multi-user.target

system · March 7, 2023, 3:34pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.