1. copying a file to the mount slows down the whole system (the whole system is on 100% SSD (specs below))
copying a file to the mount (with or without caching writes) slows down the whole server
allthough CPU usage is barely near 2%
I guess its IOs or IOPS but I couldn't verify that
unrar of 15GB takes a few seconds
start copy of the file to the mount
unrar another 15GB would take up to 3 hours
cancel the copy
unrar finishes the process in a few seconds
2. slow write speed to disk (reading is fine and plex works with multiple users at the same time)
importing a file to the mount is slow AF
(writes are cached)
I recently moved everything to a more powerfull server but rclone is the only program that won't work like it did on the old server
Even if I copy a file manually to the rclone mount cp src dst the speed is somewhere in between 8 to 15mbits
rclone sync or rclone copy utilizes almost the whole bandwidth and does not slow down the other processes
initially I've transfered the config 1:1 to the new server, then I noticed the slow write speeds.
I then reconfigured everything -> same issue
purged the rclone installation and started from scratch -> same issue
purged again and configured a cache remote ->
now it was a bit different:
copying a file would start with 30 to 40 MB/s and drop every second
but after 15 seconds (arround 400MB) it stopped for about 7 seconds (showed a speed of 0.00)
then it continued with like 900-1200kbits
but strangely not for every file
every file had this issues but at different time stamps
at one file it stopped after 5 sec, another after 20 the next didn't stop at all
but instead was at 900kbits from the start,
the next would again start at 30MBps stop after 15s and continue with ~1MBps
played arround with different settings like chunk size, cache retention, workers,... (don't remember everything -> same issue everytime
I noticed that I didn't have logging enabled -> enabled it
tried to reproduce the above issues -> failed -> now all file transfers would run with 0.9 to 1.8Mbits and I canceled after ~30 secs
disabled logging -> now the behaviour was again like in step 3
3. rclone stopped utilizing the cache folder
before I reconfigured everything I noticed that the cache folder was not used, instead (I guess) it uploaded directly to the drive
the cache folder was set with the mount command
4. server side move/copy seems to fail
I sometimes copy stuff to an "import" folder on the drive and then import from there into sonarr/radarr
until now it worked great, took like 3 seconds and just moved the file
now on the new server it seems to copy the file locally, which takes forever
What is your rclone version (output from rclone version )
go version: go1.12.9
the old server was running 1.42.x so I've updated rclone on the old server to see if I could reproduce the issues there
no luck, worked just fine, no issues at all
(the 2 servers have their own API creds)
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Which cloud storage system are you using? (eg Google Drive)
Google Team Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp )
My mount command is(was): /usr/bin/rclone mount -v --config=/root/.config/rclone/rclone.conf --allow-other --cache-writes --cache-dir=/media/rclonecache gdrive: /mnt/gdrive
I've enabled the log file and the behaviour changed (like described above) /usr/bin/rclone mount -vv --config=/root/.config/rclone/rclone.conf --allow-other --cache-writes --cache-dir=/media/rclonecache --log-file=/var/log/rclone/rclone.log gdrive: /mnt/gdrive
I don't see anything obvious that jumps out as I had a look through the logs as I'm guessing something gets caught up on the cache backend. I personally do not write to the mount as you might have better luck removing the cache backend from the equation as it works ok for the read part, but the backend doesn't have an active maintainer so it hasn't gotten much love recently.
I use mergerfs and an upload script to move files locally to my cloud mount. You could use the same as well and it would remove the writes from the equation.
thank you for looking into the logs
I've also tried it without caching the writes (or is that default behaviour if not set in the command?)
I've heard about mergerfs but never looked into it.
but I don't like the fact of needing another tool/script, cause what I want to accomplish, already worked on the previous server (with same OS and same rclone version).
would you mid sharing your setup?
I'm thinking: mount google drive read only and merge it with a local folder
so all writes would be local (if I understood mergerfs' wiki right)
then have a script move the local files to the drive, therefore the paths would never change.
but how would that merged mount behave during the move?
The file would be visible twice during the sync process, right?
i have set it to writes
this is my current mount command usr/bin/rclone mount -vv --config=/root/.config/rclone/rclone.conf --allow-other --cache-writes --cache-dir=/media/rclonecache --vfs-cache-mode writes --vfs-read-chunk-size 16M --log-file=/var/log/rclone/rclone.log gdrive: /mnt/gdrive
I think you misunderstood me here
I was just speculating about the swap
the server has 12GB and currently 2.3GB are used, the rest is free
so it should not need a swap
there are no leaks, as far as I can tell
That will take some time. Quite a lot of data already queued to be uploaded.
In the meanwhile, could you try doing a rename after disabling "Use Hardlinks instead of Copy" in the Settings? I am not using mergerfs and currently upload directly via the mount so this is disabled for me.