Cache backend very slow with a lot of "chunk retry storage" messages

Kunio · July 26, 2018, 7:50pm

I’m pretty new to rclone so apologies if I’m making any obvious mistakes.

The problem is that the cache backend is very slow for me, while direct (unwrapped) remote downloads are very fast. In the debug output I see a lot of “chunk retry storage: 0, 1, 2, …” messages which causes network traffic to stall.

Downloads via the (empty) cache are never above 5MB/s, while those direct are always at least 20MB/s.

I’m not using crypt, nor mount for these tests. Just plain copy commands.

Rclone version: 1.42. Tested both in Windows and Arch Linux.
Bandwidth: tested 1 Gbps dedicated server and 200 Mbps line at work.
Cloud provider: Google Drive

Rclone.conf:

[gdrive]
type = drive
client_id = 
client_secret = 
scope = drive
root_folder_id = 
service_account_file = 
token = {"access_token":"...","token_type":"Bearer","refresh_token":"...","expiry":"2018-07-26T21:44:38.185665219+02:00"}

[gcache]
type = cache
remote = gdrive:
plex_url = 
plex_username = 
plex_password = 
chunk_size = 
info_age = 
chunk_total_size = 2G

[gdrive-test]
type = alias
remote = gdrive:test

[gcache-test]
type = alias
remote = gcache:test

Direct download:

$ rclone copy gdrive-test:vietnam.mkv . -v
2018/07/26 21:33:45 DEBUG : rclone: Version "v1.42" starting with parameters ["rclone" "-v" "copy" "gdrive-test:vietnam.mkv" "." "-v"]
2018/07/26 21:33:45 DEBUG : Using config file from "/home/kunio/.config/rclone/rclone.conf"
2018/07/26 21:33:45 DEBUG : pacer: Rate limited, sleeping for 1.991937468s (1 consecutive low level retries)
2018/07/26 21:33:45 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:33:45 DEBUG : pacer: Rate limited, sleeping for 2.885713156s (2 consecutive low level retries)
2018/07/26 21:33:45 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:33:50 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2018/07/26 21:33:52 DEBUG : vietnam.mkv: Couldn't find file - need to transfer
2018/07/26 21:33:56 INFO  : vietnam.mkv: Copied (new)
2018/07/26 21:33:56 INFO  :
Transferred:   390.325 MBytes (34.849 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       11.2s

2018/07/26 21:33:56 DEBUG : 6 go routines active
2018/07/26 21:33:56 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone" "-v" "copy" "gdrive-test:vietnam.mkv" "." "-v"]

Download via empty cache:

The full output was too long for this post, so I put it up on Pastebin: https://pastebin.com/beWVFjQk
However, here’s an excerpt:

2018/07/26 21:04:13 DEBUG : test/vietnam.mkv: 256901120: chunk retry storage: 0
2018/07/26 21:04:13 DEBUG : worker-3 <vietnam.mkv>: downloaded chunk 256901120
2018/07/26 21:04:13 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 0
2018/07/26 21:04:14 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 1
2018/07/26 21:04:14 DEBUG : pacer: Rate limited, sleeping for 1.243364331s (1 consecutive low level retries)
2018/07/26 21:04:14 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:14 DEBUG : pacer: Rate limited, sleeping for 2.34956455s (2 consecutive low level retries)
2018/07/26 21:04:14 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:14 DEBUG : pacer: Rate limited, sleeping for 4.657515688s (3 consecutive low level retries)
2018/07/26 21:04:14 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:14 DEBUG : pacer: Rate limited, sleeping for 8.836516241s (4 consecutive low level retries)
2018/07/26 21:04:14 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:14 DEBUG : pacer: Rate limited, sleeping for 16.7914385s (5 consecutive low level retries)
2018/07/26 21:04:14 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:14 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 2
2018/07/26 21:04:15 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 3
2018/07/26 21:04:15 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 4
2018/07/26 21:04:16 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2018/07/26 21:04:16 DEBUG : worker-0 <vietnam.mkv>: downloaded chunk 272629760
2018/07/26 21:04:16 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 5
2018/07/26 21:04:16 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 6
2018/07/26 21:04:17 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 7
2018/07/26 21:04:17 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 8
2018/07/26 21:04:18 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 9
2018/07/26 21:04:18 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 10
2018/07/26 21:04:19 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 11
2018/07/26 21:04:19 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 12
2018/07/26 21:04:20 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 13
2018/07/26 21:04:20 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 14
2018/07/26 21:04:21 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 15
2018/07/26 21:04:21 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 16
2018/07/26 21:04:22 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 17
2018/07/26 21:04:22 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 18
2018/07/26 21:04:23 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 19
2018/07/26 21:04:23 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 20
2018/07/26 21:04:24 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 21
2018/07/26 21:04:24 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 22
2018/07/26 21:04:25 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 23
2018/07/26 21:04:25 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 24
2018/07/26 21:04:26 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 25
2018/07/26 21:04:26 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 26
2018/07/26 21:04:27 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 27
2018/07/26 21:04:27 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 28
2018/07/26 21:04:28 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 29
2018/07/26 21:04:28 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 30
2018/07/26 21:04:29 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 31
2018/07/26 21:04:29 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 32
2018/07/26 21:04:30 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 33
2018/07/26 21:04:30 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 34
2018/07/26 21:04:31 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 35
2018/07/26 21:04:31 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 36
2018/07/26 21:04:32 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 37
2018/07/26 21:04:32 DEBUG : test/vietnam.mkv: 267386880: chunk retry storage: 38
2018/07/26 21:04:33 DEBUG : worker-1 <vietnam.mkv>: downloaded chunk 277872640
2018/07/26 21:04:33 DEBUG : worker-2 <vietnam.mkv>: downloaded chunk 267386880
2018/07/26 21:04:33 DEBUG : worker-3 <vietnam.mkv>: downloaded chunk 283115520
2018/07/26 21:04:33 DEBUG : test/vietnam.mkv: 288358400: chunk retry storage: 0
2018/07/26 21:04:33 DEBUG : pacer: Rate limited, sleeping for 1.0964908s (1 consecutive low level retries)
2018/07/26 21:04:33 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:33 DEBUG : pacer: Rate limited, sleeping for 2.408862967s (2 consecutive low level retries)
2018/07/26 21:04:33 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:33 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2018/07/26 21:04:33 DEBUG : worker-2 <vietnam.mkv>: downloaded chunk 298844160
2018/07/26 21:04:33 DEBUG : worker-0 <vietnam.mkv>: downloaded chunk 288358400
2018/07/26 21:04:33 DEBUG : worker-1 <vietnam.mkv>: downloaded chunk 293601280
2018/07/26 21:04:33 DEBUG : test/vietnam.mkv: 304087040: chunk retry storage: 0
2018/07/26 21:04:34 DEBUG : test/vietnam.mkv: 304087040: chunk retry storage: 1
2018/07/26 21:04:34 DEBUG : pacer: Rate limited, sleeping for 1.843628418s (1 consecutive low level retries)
2018/07/26 21:04:34 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:34 DEBUG : pacer: Rate limited, sleeping for 2.819349828s (2 consecutive low level retries)
2018/07/26 21:04:34 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:34 DEBUG : pacer: Rate limited, sleeping for 4.944461911s (3 consecutive low level retries)
2018/07/26 21:04:34 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:34 DEBUG : pacer: Rate limited, sleeping for 8.802814445s (4 consecutive low level retries)
2018/07/26 21:04:34 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:34 DEBUG : pacer: Rate limited, sleeping for 16.413970945s (5 consecutive low level retries)
2018/07/26 21:04:34 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/07/26 21:04:34 DEBUG : test/vietnam.mkv: 304087040: chunk retry storage: 2
2018/07/26 21:04:35 DEBUG : test/vietnam.mkv: 304087040: chunk retry storage: 3
2018/07/26 21:04:35 DEBUG : test/vietnam.mkv: 304087040: chunk retry storage: 4
2018/07/26 21:04:36 DEBUG : test/vietnam.mkv: 304087040: chunk retry storage: 5
2018/07/26 21:04:36 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2018/07/26 21:04:36 DEBUG : worker-2 <vietnam.mkv>: downloaded chunk 309329920


2018/07/26 21:05:21 INFO  :
Transferred:   390.325 MBytes (4.179 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:     1m33.4s

Animosity022 · July 26, 2018, 7:58pm

You generally wouldn’t want to use the cache for rclone copy as the use case for that is more plex/emby/media streaming related.

For GD, I tend to drop my transactions per second so I don’t get any rate limits so I use:

--tpslimit 3 --transfers 3

You can just do a direct rclone copy to your GD with that without adding any cache or anything in between.

The cache errors on the chunks looks like maybe you changed the chunk size? I’d delete the chunks directory and up the chunks a bit bigger too. 5M is really tiny. The cache itself is slower as it handles the chunking and spawn multiple workers. It’s not currently faster than plexdrive imo.

I personally use:

-vfs-read-chunk-size 32M --vfs-read-chunk-size-limit 2G

over the cache as the performance is better than plexdrive for me (ymmv) but it should be in the same ballpark.

The # of 403s is quite high and I wonder if that’s due to the low chunk size as those are coming quite fast and it is causing you to slow down while rclone backs off.

Kunio · July 26, 2018, 8:33pm

Thanks for your very fast response Animosity!

The reason I’m using rclone copy right now is to make the test case as simple as possible. Originally I set up a drive -> cache -> crypt remote that I mounted with rclone. However trying to play back the above file did not even start. I did have VFS cache writes enabled, but didn’t touch chunked reading. To be clear, you are using VFS cache on top of the cache backend right?

I changed the cache chunk size to 32M and deleted the whole cache folder. This result is a large improvement, though still a number of those chunk retry storage messages. Do you know what these mean?

https://pastebin.com/dufpFzmf

2018/07/26 22:12:59 INFO  :
Transferred:   390.325 MBytes (13.304 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       29.3s

Limiting the number of transactions per second is even faster, and is starting to near the speed of uncached remote. But it still bumps into the rate limit and produces the same chunk retry storage messages.

https://pastebin.com/D1n8Y0vs

2018/07/26 22:20:24 INFO  :
Transferred:   390.325 MBytes (20.822 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       18.7s

Next I will move the cache off my root to a larger partition, increase the cache size and try a larger file for better speed measurements.

Animosity022 · July 26, 2018, 9:28pm

Are you using your own rclone key ? The 403s for rate limiting on start are confusing as you shouldn’t see those.

No, the vfs-read-chunk-size config and cache configs require you to pick one or the other. You normally run them both with dir-cache to cache the directory/filenames. The issue prior to vfs-read-chunk-size is that it would grab the entire file on an open instead of chunking it up.

The chunk retries are when a chunk doesn’t come back fast enough. It basically requests a chunk for each worker and it loops waiting for the chunk to come back and each one of those reties is the waiting for it to come back.

Kunio · July 26, 2018, 10:18pm

No, I’m not using my own API key for Google Drive. I had considered it at the start of debugging this speed issue, but I read from ncw somewhere in the forum that his key probably has higher rate limits anyway.

If I understand correctly, the difference between VFS chunked reading (if enabled) and reading via the cache backend, is that the cache backend will keep the chunks on disk for later if they are needed again. Oh, and VFS cache chunks can increase in size.

So basically, your setup will always stream from GDrive? Are you doing it this way for Plex transcoding?

Thanks for clearing up what those retry messages mean. So when I’m seeing chunk retry storage: 30, does that mean that worker has been waiting on GDrive for a chunk for 30 ticks? (1 tick looks like 500ms)

Can and should that wait time be decreased? It seems to me that it sometimes waits a very long time for a chunk (15 ticks or 7.5 seconds in the last example).

Also, it seems that no other workers are downloading chunks while one is retrying for a chunk.

Animosity022 · July 27, 2018, 1:00am

So it can be messed with but you’d have to grab the code/compile it and make the changes. I’m kind of baffled how if you are using your own key that you are getting 403s with a single rclone copy command.

Copying from my mount I can get a nice rate:

rsync --progress Unsane\ \(2018\).mkv /data
Unsane (2018).mkv
  1,608,908,800   6%   66.32MB/s    0:05:58

Yes, that is the difference as my use case is playing media. The odds of someone playing the same show/movie and keeping a few chunks local really doesn’t matter for me.

His key probably has higher rates and a lot more people using it.

Kunio · July 27, 2018, 2:37pm

I think you may have misread, I did NOT have my own key before.

However, I just made my own key now, and the results are impressive. I haven’t hit any rate limits yet and the retry storage messages are significantly decreased! Before they would count up to around 15, and now no more than 2!

I had also moved the cache to another partition and increased its size, but those did not have any noticable effect.

https://pastebin.com/nr74hX19

2018/07/27 16:20:52 INFO  :
Transferred:   390.325 MBytes (31.561 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       12.3s

It also looks like further increasing the chunk size is beneficial. At 64M chunks it’s now surpassing single transfers direct from the remote:

https://pastebin.com/G3a0zqk5

2018/07/27 16:31:08 INFO  :
Transferred:   390.325 MBytes (42.987 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:          9s

Animosity022 · July 27, 2018, 2:41pm

Oh yes, sorry I did misread that.

Depending on your internet speeds, finding the ‘right’ chunk size would be beneficial as well.

You may want to try a mount with just vfs-chunks as well to gauge that performance:

/usr/bin/rclone mount gcrypt: /GD --allow-other --dir-cache-time 48h --vfs-read-chunk-size 64M --vfs-read-chunk-size-limit 2G --umask 002 --bind 192.168.1.30 --log-level INFO --log-file /home/felix/logs/rclone.log

I use that currently for me.

Kunio · July 29, 2018, 5:20pm

Here’s a quick update. I’ve tried a mount with VFS cache based on your settings and I’m pretty much able to max out my home connection at about ~70 Mbps for reads which is sufficient for streaming media.

The most important things to get the best performance were again increasing the chunk size and using my own Client ID/key. I haven’t toyed with the chunk-size-limit yet. Here’s the mount command I used in Windows for now:

rclone mount gcrypt-media-nocache: W: ^
 --tpslimit 3 ^
 --vfs-cache-mode writes ^
 --vfs-read-chunk-size 32M ^
 --log-level DEBUG

Start time before playback is about 5 seconds, likewise for skipping in the video.

I also learned that I couldn’t use MPC-HC or PotPlayer to play the videos. I think they may try to scan the whole file before starting playback. VLC did work fine however.

Writing to the mount is a disaster in Windows. It hangs the explorer, blocking even local storage browsing, until the upload to the remote is finished. But that’s a topic for another thread and needs more investigative work first.

As an aside, do you handle writes through rclone?

Animosity022 · July 29, 2018, 6:37pm

Sadly, I’m not a Windows user as I’m a Linux/Mac user so on Linux I write locally and upload using unionfs/mergers type solution which combines mount points for you.

I’d defer to some other Windows users to see what they do as a few searches didn’t yield much for me.

Kunio · July 29, 2018, 7:15pm

Thanks, and no problem. I’m not too attached to Windows as my initial plan was to run rclone on my Linux NAS and share it via SMB.

What kind of mount do you use for your writes with mergerfs? A cache wrapped remote with the offline uploading feature?

Animosity022 · July 29, 2018, 8:06pm

On the linux side, you can basically ‘combine’ multiple paths into a single mount point.

So I take a local disk as my read/write and combine that with the rclone mount.

Any writes happen to the local mount and get uploaded at night via script to my GD.

kieranu · December 21, 2018, 2:38am

This is kind of related. I have a Windows mount that seems to be slow for Plex updates. This is the only activity on this server and accounts for most of the activity on my network at the moment. I don’t have any active uploads or downloads happening currently. The uploads to gsuite which would trigger Plex updates are happening on another server on another network. However, I am still consistently seeing 6GB upload and 200GB download traffic on the network with this Windows mount per day. In contrast another mount for Plex on Linux to the same file system only has about 20GB overall traffic per day. I’m wondering if you might know why this would be the case.

The relevant mount in both cases is media-cypt:
Linux: rclone --daemon --log-file=/home/hd1/kieranu/rclone/rclone.log --vfs-cache-mode=writes --cache-dir=/home/hd1/kieranu/rclone --allow-non-empty --allow-other --cache-tmp-wait-time=10m --cache-tmp-upload-path=/home/hd1/kieranu/rclone/cache-tmp-upload --cache-workers=12 --cache-db-purge --cache-db-wait-time=5s --drive-chunk-size=256M --cache-chunk-path=/home/hd1/kieranu/rclone/cache-backend mount media-crypt: /home/hd1/kieranu/media/crypt

Windows: rclone mount --rc --log-level=INFO --log-file=“D:\rclone\rclone.log” --config=“C:\Users\user.config\rclone\rclone.conf” --allow-non-empty --allow-other --drive-chunk-size=32M --dir-cache-time=72h --cache-chunk-path=D:\rclone --cache-db-path=D:\rclone --cache-db-purge --cache-workers=12 --cache-tmp-upload-path=D:\rclone --cache-tmp-wait-time=1m media-crypt: Z:

rclone.conf on windows:
[VConsulting]
type = drive
client_id =
client_secret =
service_account_file =
token =
[plextv]
type = cache
remote = VConsulting:TV
plex_url = http://127.0.0.1:32400
plex_username =
plex_password =
chunk_size = 10M
info_age = 24h
chunk_total_size = 10G
plex_token =

[media-crypt]
type = crypt
remote = media-cache:
filename_encryption = standard
directory_name_encryption = true
password = …
password2 =

[media-cache]
type = cache
remote = VConsulting:media
plex_url = http://127.0.0.1:32400
plex_username =
plex_password =
chunk_size = 10M
info_age = 12h
chunk_total_size = 10G
plex_token =

rclone.conf on linux:
[VConsulting]
type = drive
client_id =
client_secret =
scope = drive
root_folder_id =
service_account_file =
token =
[plextv]
type = cache
remote = VConsulting:TV
plex_url = http://127.0.0.1:32400
plex_username =
plex_password =
chunk_size = 10M
info_age = 24h
chunk_total_size = 1G
plex_token =

[media-cache]
type = cache
remote = VConsulting:media
plex_url =
plex_username =
plex_password =
chunk_size = 10M
info_age = 6h
chunk_total_size = 1G
plex_token =

[media-crypt]
type = crypt
remote = media-cache:
filename_encryption = standard
directory_name_encryption = true
password =

Animosity022 · December 21, 2018, 3:01am

I’m not sure how it’s related to a chunk retry storage message unless you are seeing those.

I’d check your plex settings and see what you have configured for scheduled maintenance as it’s most likely that.

kieranu · December 21, 2018, 4:06am

Yeah, maybe. Its the closest thing that came up in a search so perhaps I should start a new topic.

Both Plex servers have same setting for Library, “Run a partial scan when changes are detected”. No other options selected (Scan library automatically, periodically, etc) and fewer options are actually enabled for metadata analysis etc on the Windows server

So I’m wondering if something about the cache is making it think a folder has changed and triggering a rescan on the windows mount when it hasn’t actually changed in gsuite.

Animosity022 · December 21, 2018, 4:34am

A scan doesn’t download anything though so that shouldn’t normally cause an issue. If it thinks it is a new file, it would have to analyze it.

Is someone sync’ing?

Are there any scheduled tasks in Plex?

Do the libraries have thumbnail generation or previews on?

Local media assets on?

Something from Plex is triggering I would guess. Just not sure what.

kieranu · December 21, 2018, 9:59am

Local Media Assets are are turned on (but this is also true on Linux server), I’ve now turned them off as test
There are some scheduled tasks but fewer than on the Linux server
Neither generates video preview thumbnails
Yeah, I also not sure what.
That it generates so much more traffic on Windows than on Linux is indeed a conundrum. One reason I switched off the Windows server in the first place but when I got a better local internet connection I thought it would rebuild library and sort itself out. Weeks later …

Also, I switched to pure VFS at one point because it is purely for updating library in Plex (no uploads to gsuite except perhaps subtitles / metadata) and this meant playback was quicker (less buffering to start) but found I got API errors from too many reads on a file so reverted to cache. Again found this phenomenon on my Linux server. Linux server is quick to update library when new files are detected in file system and uses minimum data (about 20GB all up per day max) whereas Windows seems to spin its wheels, takes longer to go through updating the library, and burns through 200GB of bandwidth per day

Animosity022 · December 21, 2018, 12:42pm

What’s your rclone version show?

kieranu · December 21, 2018, 2:35pm

Version showing is 1.44

Animosity022 · December 21, 2018, 2:38pm

I’d move to 1.45 / latest but that really won’t do much with the usage as that’s all driven by plex.

API errors wouldn’t hit VFS or cache really as those are all use things. Are you using your own API key?