OneDrive Rclone mount not updating from remote

What is the problem you are having with rclone?

I use systemd service to mount my OneDrive, uploaded new files to it and they showed up in the mounted folder. But after some time, the new files disappeared in the folder. I entered rclone ls test: saw that the uploaded files are still in the remote without problems

I reboot the vps, those previously uploaded files now show up in the mounted folder. But if I upload new files to the mount, after a while, they would disappear again. The new added files won't synchronize to the mount for some reasons

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.15.0-1044-oracle (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.21.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Oracle Cloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Here's the content in my /etc/systemd/system/rclone.service

[Unit]
Description=Rclone Mount Service 
After=network-online.target

[Service] 
Type=notify
ExecStart=/usr/bin/rclone mount \
--config=/root/.config/rclone/rclone.conf \
--log-level=INFO \
--log-file=/home/ubuntu/rclone/log/rclone-mount.log \
--user-agent=root \
--umask=022 \
--gid=0 \
--uid=0 \
--allow-other \
--timeout=1h \
--poll-interval=15s \
--dir-cache-time=1000h \
--cache-dir=/home/ubuntu/rclone/cache \
--vfs-cache-mode=full \
--vfs-cache-max-size=4G \
--vfs-cache-max-age=12h \
test: /home/ubuntu/rclone/cloud
ExecStop=/bin/fusermount -uz /home/ubuntu/rclone/cloud 
Restart=on-abort 
RestartSec=5 
StartLimitInterval=60s 
StartLimitBurst=3 
 
[Install] 
WantedBy=multi-user.target

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[onedrive]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal

[test]
type = crypt
remote = onedrive:/test1
password = XXX

A log from the command that you were trying to run with the -vv flag

I checked the info log, there are lots of repetitions of the below lines

2023/10/06 01:13:26 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 0, to upload 0, uploading 0, total size 7 (was 7)
2023/10/06 01:13:27 INFO  : OneDrive root 'test1': Change notify listener failure: invalidRequest: [token.String] A value is required but was not present in the request.
2023/10/06 01:13:42 INFO  : OneDrive root 'test1': Change notify listener failure: invalidRequest: [token.String] A value is required but was not present in the request.
2023/10/06 01:13:57 INFO  : OneDrive root 'test1': Change notify listener failure: invalidRequest: [token.String] A value is required but was not present in the request.

to get a deeper look, can use --dump=headers or --dump=headers,responses

How do I use --dump=headers , do I include this line in the rclone systemd service, and then check back its output log ?

rclone lsd test: --dump=headers

give it a try.....

After I enter rclone lsd test: --dump=headers in Putty, here's the full response (I change my file name to file-1, folder-1 etc.)

Edit: There were a lot of space when I paste the lines, I remove the space and repaste the lines here for easy reading

2023/10/06 16:43:08 NOTICE: Automatically setting -vv as --dump is enabled
2023/10/06 16:43:08 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "lsd" "test:" "--dump=headers"]
2023/10/06 16:43:08 DEBUG : Creating backend with remote "test:"
2023/10/06 16:43:08 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/10/06 16:43:08 DEBUG : Creating backend with remote "onedrive:/test1"
2023/10/06 16:43:08 DEBUG : You have specified to dump information. Please be no ted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2023/10/06 16:43:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/10/06 16:43:08 DEBUG : HTTP REQUEST (req 0x400011d700)
2023/10/06 16:43:08 DEBUG : GET /v1.0/drives/df27da9fc9b40c72/root HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
Accept-Encoding: gzip

2023/10/06 16:43:08 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/10/06 16:43:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/10/06 16:43:09 DEBUG : HTTP RESPONSE (req 0x400011d700)
2023/10/06 16:43:09 DEBUG : HTTP/2.0 200 OK
Cache-Control: private
Client-Request-Id: dff273a8-e5ee-4a79-8530-0417a5af5e1c
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Fri, 06 Oct 2023 16:43:09 GMT
Odata-Version: 4.0
Request-Id: dff273a8-e5ee-4a79-8530-0417a5af5e1c
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"West Central US","Slice":"E","Ring":"1","ScaleUnit":"001","RoleInstance":"CY4PEPF0000FB15"}}

2023/10/06 16:43:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/10/06 16:43:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/10/06 16:43:09 DEBUG : HTTP REQUEST (req 0x4000993000)
2023/10/06 16:43:09 DEBUG : GET /v1.0/drives/df27da9fc9b40c72/items/DF27DA9FC9B40C72!101:/test1: HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
Accept-Encoding: gzip

2023/10/06 16:43:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/10/06 16:43:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/10/06 16:43:09 DEBUG : HTTP RESPONSE (req 0x4000993000)
2023/10/06 16:43:09 DEBUG : HTTP/2.0 200 OK
Cache-Control: private
Client-Request-Id: 8e2b5002-6534-4c2c-95f8-7595e1522c29
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Fri, 06 Oct 2023 16:43:09 GMT
Odata-Version: 4.0
Request-Id: 8e2b5002-6534-4c2c-95f8-7595e1522c29
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"West Central US","Slice":"E","Ring":"1","ScaleUnit":"001","RoleInstance":"CY4PEPF0000FB15"}}

2023/10/06 16:43:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/10/06 16:43:09 DEBUG : fs cache: renaming cache item "onedrive:/test1" to be canonical "onedrive:test1"
2023/10/06 16:43:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/10/06 16:43:09 DEBUG : HTTP REQUEST (req 0x400011de00)
2023/10/06 16:43:09 DEBUG : GET /v1.0/drives/df27da9fc9b40c72/items/DF27DA9FC9B40C72!227/children?$top=1000 HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.64.0
Authorization: XXXX
Accept-Encoding: gzip

2023/10/06 16:43:09 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/10/06 16:43:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/10/06 16:43:09 DEBUG : HTTP RESPONSE (req 0x400011de00)
2023/10/06 16:43:09 DEBUG : HTTP/2.0 200 OK
Cache-Control: no-store
Client-Request-Id: 74b14e93-363d-4d3b-a57e-0c253826d177
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Fri, 06 Oct 2023 16:43:09 GMT
Odata-Version: 4.0
Request-Id: 74b14e93-363d-4d3b-a57e-0c253826d177
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"West Central US","Slice":"E","Ring":"1","ScaleUnit":"001","RoleInstance":"CY4PEPF0000FB15"}}

2023/10/06 16:43:09 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/10/06 16:43:09 DEBUG : folder-4: Skipping undecryptable dir name: not a multiple of blocksize
2023/10/06 16:43:09 DEBUG : file-3.rar: Skipping undecryptable file name: illegal base32 data at input byte 2
2023/10/06 16:43:09 DEBUG : file-1.txt: Skipping undecryptable file name: illegal base32 data at input byte 3
2023/10/06 16:43:09 DEBUG : file-2.txt: Skipping undecryptable file name: illegal base32 data at input byte 3
          -1 2023-10-06 00:04:28         1 folder-1
          -1 2023-10-05 21:45:27         1 folder-2
          -1 2023-10-03 20:54:04         1 folder-3
2023/10/06 16:43:09 DEBUG : 7 go routines active

In terms of INFO log, there's not much difference comparing to the previous one

listener failure: invalidRequest: [token.String] A value is required but was not present in the request.
i do not know why that happens, i did not quick test, could not replicate.
adding --dump can give you a deeper look into what rclone is sending/receiving, so add it to your mount command.

not sure what that means?
can you post some set of commands to document that, how to replicate.

fwiw, for debugging, i find it easier to not use systemd. to use .sh on command line.

I would recommend you try 1.65 beta release. ListR support via delta API is now merged on that revision and fixed a lot of my cache issues plus it made the file list way more reliable. I used to have this issue and had to do a rc call for vs/forget and vfs/refresh every now and resync all the files without remounting, but since using the new method I have not experienced this issue.

1 Like

that is interesting,
i did a quick test, could not replicate. and in the forum, only found a single mention of token.String

Sure, here's my workflow:

  1. use systemd service to mount my remote as described previously
  2. create a local folder in my vps, use mergerfs to merge the local folder and the mounted folder into a new folder. (This new folder is the path of my Plex and relevant apps)
  3. setup a cron job to auto-upload files from the local folder to the mounted folder

Here's what I mean by 'the new files disappeared in the mounted folder'

If I create a new file in the local folder, it gets uploaded into the mounted folder, and I can see it inside the mounted folder within like 2 hours after the upload. Then If I check it again on the 3rd hour, it's not in the mounted folder anymore. I thought it was deleted in the first place, but when I used rclone ls test: the file is still there, so it looks like the mounted folder does not sync with my cloud remote for some reason

thanks for that info.

now, might test what @jcastill suggested

Within the 3 steps I mentioned above, I don't think the issue belongs to step 2 and step 3, as they're quite straightforward commands

So I think it's the 1st step which causes the problem. Some settings in the systemd service are probably not correct, cache settings maybe? But currently they looks alright to me

Hey, thanks for sharing this!

Do I simply keep my workflow and just set up everything again in rclone 1.65 beta version ?

at this point, should test @jcastill suggestion

if not that, then i would start from scratch, create a new remote, using client id/secret.
might workaround that token.String A value is required but was not present in the request.

Sure, I'll try @jcastill suggestion first

By the way, do I need to uninstall the current rclone v1.64 first, and how do I do this ?

Sorry, didn't mean the token.String error directly. But overall OneDrive cache going crazy. Based on my hunch if the mount ever gets throttled, it will eventually go out of sync as the polling will be throttled as well and miss the sync information. Based on my (still limited) testing it seems Microsoft is more relaxed on delta API calls so it won't happen.

1 Like

Simply run rclone selfupdate --beta and it should update. rclone selfupdate

rclone is a single file, portable, no install needed. download the latest beta into a dir and run.
https://beta.rclone.org/

or could install beta, using https://rclone.org/downloads/#script-download-and-install

@asdffdsa @jcastill

Thanks and I will keep you guys updated, when I switch to beta version and test it

1 Like

Since I have plan to use Google Drive, I test this by creating a new remote. All mount settings in systemd file are the same, except this time I added client id/secret through the remote creating process.

Now there's no syncing issues, all uploaded files are updated in time in the mounted folder. I think I'll stick to this setup now. Although I haven't yet tested rclone beta, it might help if someone were to use onedrive; Or else, adding client id/secret to the remote works. Thank you for these solutions! @asdffdsa @jcastill

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.