Rclone getting slower over time?

Help and Support
#1

What is the problem you are having with rclone?

Over the last 2 or 3 months, i kinda feel that my rclone has slowed down with basic operations. I know this sounds really generic, but i cannot make out hard facts to prove it, as i don't have a before situation.
So for example, i have a gdrive share mounted with rclone. When i want to create a directory, it literally takes 2-3 seconds. Or when i move files or folders on the share, it takes a good 10 seconds for each operation. Furthermore i also think that scanning folders and directories has slowed down, especially for sonarr and plex library scanners. But again, i cannot show you any hard evidence.

Maybe you could give me some insight as to how rclone mount behaves in your setups?
The only thing i tried so far is restarting the mount which seems to help for some time i guess. I also played around with the dir cache time which seems to work nicely when browsing the directories but doesn't make a difference with the problems i described.

Unfortunately, i am a bit limited by my system, as i am running off a VPS which has distributed services (docker swarm or something like that) accessing rclone. However i am able to see, that rclone mount will increase it's memory and even CPU usage and does not let go of it, even in idle phases. It reaches up to 1,5GB reserved memory, even with nmap activated. In the past, memory freeing has worked better for my mount, so this is something that stands out from my point of view.

What is your rclone version (output from rclone version)

rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 16.04.6 LTS (GNU/Linux 3.10.0-1062.18.1.el7.x86_64 x86_64)

Which cloud storage system are you using? (eg Google Drive)

Gdrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --allow-other --use-mmap --allow-non-empty --gid 911 --uid 911 --umask 000 --timeout 1h --buffer-size 32M --dir-cache-time 720h --cache-info-age 800h --drive-chunk-size 64M --fast-list --log-level INFO --log-file '/mnt/shared/config/rclone/log/rclone.log' --vfs-read-chunk-size 64M --vfs-read-chunk-size-limit 1024M --vfs-cache-mode writes --vfs-cache-max-age 5m --vfs-cache-max-size 150G --cache-dir '/mnt/shared/fusemounts/.rclonecache/' --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" gdrive_media_vfs: /mnt/shared/fusemounts/gdrive

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

i checked the debug logs yesterday and wasn't able to find anything suspicious. I just deleted them yesterday, because it was over 10GB already. Tell me if you need some logs anyway.

Thanks!

edit: Here is an example log of how long it takes sonarr to scan individual items (timestamp)

20-5-7 22:19:21.7|Debug|MetadataService|Episode image already exists: /media//Season 4/The.Cosby.Show.S04E24.SDTV.x264.AAC-Gone.Fishin'-Sonarr-thumb.jpg
20-5-7 22:19:21.7|Debug|XbmcMetadata|Generating Episode Metadata for: /media//Season 5/The.Cosby.Show.S05E01.SDTV.x264.AAC-Together.Again.and.Again-Sonarr.mkv
20-5-7 22:19:22.4|Debug|MetadataService|Episode image already exists: /media//Season 5/The.Cosby.Show.S05E01.SDTV.x264.AAC-Together.Again.and.Again-Sonarr-thumb.jpg
20-5-7 22:19:22.4|Debug|XbmcMetadata|Generating Episode Metadata for: /media//Season 5/The.Cosby.Show.S05E02.SDTV.x264.AAC-The.Physical-Sonarr.mkv
20-5-7 22:19:23.1|Debug|MetadataService|Episode image already exists: /media//Season 5/The.Cosby.Show.S05E02.SDTV.x264.AAC-The.Physical-Sonarr-thumb.jpg
20-5-7 22:19:23.1|Debug|XbmcMetadata|Generating Episode Metadata for: /media//Season 5/The.Cosby.Show.S05E03.SDTV.x264.AAC-Rudy's.All-Nighter-Sonarr.mkv
20-5-7 22:19:23.8|Debug|MetadataService|Episode image already exists: /media//Season 5/The.Cosby.Show.S05E03.SDTV.x264.AAC-Rudy's.All-Nighter-Sonarr-thumb.jpg
20-5-7 22:19:23.8|Debug|XbmcMetadata|Generating Episode Metadata for: /media//Season 5/The.Cosby.Show.S05E04.SDTV.x264.AAC-Move.It-Sonarr.mkv
20-5-7 22:19:24.5|Debug|MetadataService|Episode image already exists: /media//Season 5/The.Cosby.Show.S05E04.SDTV.x264.AAC-Move.It-Sonarr-thumb.jpg
20-5-7 22:19:24.5|Debug|XbmcMetadata|Generating Episode Metadata for: /media//Season 5/The.Cosby.Show.S05E05.SDTV.x264.AAC-Out.of.Brooklyn-Sonarr.mkv
20-5-7 22:19:25.4|Debug|MetadataService|Episode image already exists: /media//Season 5/The.Cosby.Show.S05E05.SDTV.x264.AAC-Out.of.Brooklyn-Sonarr-thumb.jpg
20-5-7 22:19:25.4|Debug|XbmcMetadata|Generating Episode Metadata for: /media//Season 5/The.Cosby.Show.S05E06-07.SDTV.x264.AAC-The.Birth-Sonarr.mkv
20-5-7 22:19:26.1|Debug|MetadataService|Episode image already exists: /media//Season 5/The.Cosby.Show.S05E06-07.SDTV.x264.AAC-The.Birth-Sonarr-thumb.jpg
20-5-7 22:19:26.1|Debug|XbmcMetadata|Generating Episode Metadata for: /media//Season 5/The.Cosby.Show.S05E08.SDTV.x264.AAC-Cyranoise.de.Bergington-Sonarr.mkv
20-5-7 22:19:26.1|Debug|Api|[GET] /api/system/status?apikey=(removed) 200.OK (4 ms)
20-5-7 22:19:26.1|Debug|Api|[GET] /api/episode?seriesId=43&apikey=(removed) 200.OK (2098 ms)
#2

hello,

have you done this? if not, please do so and test again.
https://rclone.org/drive/#making-your-own-client-id

have you contacted your vps provider about it?

#3

sure, sure. The setup has basically been working for over 1 year now.

I haven't talked to them yet, as i wanted check in here first because it is application specific.

#4

i did not see any errors in that log file.

if
for over one year,
you are using the same rclone version
and
same rclone command
and
you are using a "distributed services (docker swarm or something like that)"
then you try should contact the vps host.

#5

i did not say that i used the same rclone version. Maybe that is where you are mistaken.

#6

i was not mistaken, i was asking a question.
that if you nothing in your setup has changed in that last few months, then the odds are it has something to do with the vps provider.
if something has changed, you should revert back to a time when everything was working.
revert rclone and config files and whatever else you can.

i have seen posters with strange issues related to certain kernel versions.

also one thing i do when having a problem that is hard to define is i simplify my rclone command to the abolute minimum and test, adding flags back in one at a time.

#7

Debug log and config with keys removed would help.

It looks like you're using cache. Try without it.

#8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.