Impossible to reach the contents, at random moments of the day. rclone 100%

Help and Support
1

What is the problem you are having with rclone?

Hi guys, I've had a problem for a while, but only now did I decide to write here.

At some random moments of the day, rclone goes to 100% of the CPU and any content becomes unreachable.
The only way to resume the correct functionality of the operations is to force the task manager to close and it will work again.

I have shared with you here the debug log of the exact moment something happened and it became impossible to continue watching the content with plex.
You can also see the moment when I restarted and correct functioning resumed.

I hope someone can help me.

The mount command has been modified in all sorts of ways. I also tried deleting everything and leaving only the basic things, like vfs-cache full, and log.
It doesn't solve the problem.

Run the command 'rclone version' and share the full output of the command.

C:\rclone>rclone version
rclone v1.68.1
- os/version: Microsoft Windows Server 2019 Standard Evaluation 1809 (64 bit)
- os/kernel: 10.0.17763.5458 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

mount --vfs-cache-mode=full --vfs-cache-max-age=4h --vfs-cache-max-size=24G --drive-pacer-min-sleep=10ms --drive-pacer-burst=200 --cache-dir c:\rclone\temp cloud-crypt: H: --log-file="C:\rclone\log\cloud.log" --log-level DEBUG --onedrive-av-override

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[cloud]
type = onedrive
token = {"access_token":"#############
drive_id = ######
drive_type = business
av_override = true

[cloud-crypt]
type = crypt
remote = cloud:contenuti
filename_encryption = obfuscate
directory_name_encryption = false
password = #####
password2 = #####

LOG DEUBG

https://pastebin.com/raw/DU02ucKz
2

What are you running it on? Specs of the server?

3

The logs show lots of throttling too, which will cause problems.

I don't recall the tps limits for OneDrive, but a search on the forums and you'll find some values.

2024/10/14 22:41:04 DEBUG : pacer: low level retry 1/10 (error <nil>)
2024/10/14 22:41:04 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/10/14 22:41:04 DEBUG : pacer: low level retry 2/10 (error <nil>)
2024/10/14 22:41:04 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/10/14 22:41:04 DEBUG : pacer: low level retry 3/10 (error <nil>)
2024/10/14 22:41:04 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2024/10/14 22:41:04 DEBUG : pacer: low level retry 4/10 (error <nil>)
2024/10/14 22:41:04 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2024/10/14 22:41:04 DEBUG : pacer: low level retry 5/10 (error <nil>)
2024/10/14 22:41:04 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2024/10/14 22:41:05 DEBUG : pacer: low level retry 6/10 (error <nil>)
2024/10/14 22:41:05 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2024/10/14 22:41:05 DEBUG : pacer: low level retry 7/10 (error <nil>)
2024/10/14 22:41:05 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2024/10/14 22:41:06 DEBUG : pacer: low level retry 8/10 (error <nil>)
2024/10/14 22:41:06 DEBUG : pacer: Rate limited, increasing sleep to 2s
2024/10/14 22:41:07 DEBUG : pacer: low level retry 9/10 (error <nil>)
2024/10/14 22:41:09 DEBUG : pacer: low level retry 10/10 (error <nil>)
4

pacer: Rate limited, increasing sleep to 2s
rclone is getting throttled by onedrive. which is a common issue with onedrive and much discussed in the forum.

check the rclone docs
The default Client ID and Key are shared by all rclone users when performing requests.
You may choose to create and use your own Client ID, in case the default one does not work well for you.
For example, you might see throttling.

check the rclone docs
This can be very useful for rclone mount to control the behaviour of applications using it.

that flag is for gdrive, not onedrive.

5

Not very scientific as I kept simply lowering values until I reached stable behaviour for rustic backup to onedrive.

Configure your own client_it - this is must.

Try these flags:

--tpslimit 10 --tpslimit-burst 0
--onedrive-delta

Depending on what are you using it for slightly different values might be better - try.

In general OneDrive has limits - if you are heavy user then consider something more appropriate for your workflow. From Microsoft you can get Azure Storage - very different performance level than OneDrive. But of course you have to pay for it accordingly.

6

I would add trying this beta release that has some fix for 503 errors pacing:

Been working great on my setup. @kapitainsky recommendations are pretty solid as well, if you still see the pacer errors popping up lower the tpslimit down. I've found that 3 is the sweet number that makes Microsoft stop complaining for the most part.

7

Good reminder about fixed 503 retry-after. Works for me too.

It has not been merged into main yet... I keep an eye on it to make sure it lands in v1.69

1 Like
8

I'm not very familiar with limitations.

But as I said in the main post, it's quite random.
It happens especially when I see for more than 2 hours, but not always.

9

I added your suggestions to the boot command.

mount --vfs-cache-mode=full --vfs-cache-max-age=4h --vfs-cache-max-size=12G --drive-pacer-burst=200 --cache-dir c:\rclone\temp cloud-crypt: H: --log-file="C:\rclone\log\cloud.log" --log-level NOTICE --onedrive-av-override --tpslimit 10 --tpslimit-burst 0 --onedrive-delta

10

Thanks for the suggestion.
I updated to beta version 1.69, hopefully that fixes the issues.

I activate the notice log in the command, unfortunately debugging in recent days has created over 500mb of text files for me and it becomes difficult to open it.
However, if I notice any anomaly in the base log, it will be an alarm bell to switch to debug and return here for further support.

11

This is another Google drive flag. Remove it. It does nothing:)

OneDrive is polling remote so I would also add:

--vfs-refresh --dir-cache-time 9999h

Also try:

--vfs-fast-fingerprint
12

why use all that time?

13

It is directories structure and files listing only - if all in cache than browsing such mount is a breeze. --vfs-refresh makes sure that all is refreshed after mount starts.

It resides in memory anyway so only lives as long as your mount. There is no point to invalidate this cache for polling remote ever - so 9999h means pretty much keep it "forever".

1 Like
14

You can use RC and enable debug logging when you are facing the issue. I rely on that when troubleshooting issues that are not easily to replicate and take some setup or have not figured out what triggers a problem.

15

I've been trying for two days.
It doesn't seem to crash during the day. But I can't test well, because now I'm encountering another problem.
Streaming is very slow, it's slow to start, sometimes I have to try multiple times to get it to start.

my mount command.

mount --vfs-cache-mode=full --vfs-cache-max-age=4h --vfs-cache-max-size=12G --vfs-refresh --dir-cache-time 9999h --vfs-fast-fingerprint --cache-dir c:\rclone\temp cloud-crypt: H: --log-file="C:\rclone\log\cloud.log" --log-level NOTICE --onedrive-av-override --tpslimit 10 --tpslimit-burst 0 --onedrive-delta

Advice?

16

i suggested that you create your own client id, and then test without --tpslimit 10 --tpslimit-burst 0
did you ever try that?

the goal is to have a small log to document the issue.
so, for testing, i suggest the following script or do the same manually on the command line.

:: kill the mount
taskkill /f /t /im rclone.exe
:: delete the log
del c:\rclone\log\cloud.log
:: delete the cache
del c:\rclone\temp /q /s
:: start your mount command, must use `--log-level DEBUG`, not `--log-level NOTICE`
rclone mount ...

start to play a file, make sure it is direct play, no transcoding.
as soon as the video starts to play, kill the mount
now, you have a small log that will document the issue.

17

I removed these parameters and did the test. Below you will find the debug log.
--tpslimit 10 --tpslimit-burst 0

At first glance there don't seem to be any problems... in your opinion?

https://pastebin.com/raw/iVRcVYEf

18

well, confirm, did the have the issue of slow start to playback or not?

Forbidden (#403)
Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.
19

I'm sorry, I don't know where to upload the log.
Let's hope they publish it soon.

I tested it on a very high-performance device, so I had no problems starting up.
it went "quite" fast, there is still a lot of improvement to be done in starting playback.

so I think that during the day I will continue to try without tpslimit.
Even though I was advised to add it...

20

well, in that case, no point in posting a log

not sure about that
onedrive is not built for streaming, the forum is full of posts about that.

maybe find another provider?
what is the total size of all media files, can use rclone size ?