What is the problem you are having with rclone?
Very slow moves of files
Run the command 'rclone version' and share the full output of the command.
rclone v1.61.1
- os/version: debian 11.6 (64 bit)
- os/kernel: 5.15.39-1-pve (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none
#RCLONE SERVICE CONFIG
[Unit]
Description=RClone Service
PartOf=gmedia.service
RequiresMountsFor=/gmedia
[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/USER/.config/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount gdrive_linux_crypt: /gmedia/ \
--allow-other \
--cache-dir=/docker/vfs_cache \
--attr-timeout 1000h \
--buffer-size 256M \
--dir-cache-time 1000h \
--log-level INFO \
--log-file /var/log/rclone/rclone-linux.log \
--vfs-cache-mode full \
--vfs-cache-max-size 200G \
--vfs-fast-fingerprint \
--vfs-read-chunk-size 32M \
--vfs-write-back 5m \
--vfs-cache-max-age 24h \
--disable-http2 \
--tpslimit 12 \
--tpslimit-burst 0
ExecStop=/bin/fusermount -uz /gmedia
Restart=on-failure
User=USER
Group=GROUP
[Install]
WantedBy=multi-user.target
Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
--> Yes
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
Just moving a file from local storage to the rclone mount
mv filename /gmedia/Location/destinationfilename
The rclone config contents with secrets removed.
[gdrive_linux]
type = drive
client_id = redacted
client_secret = redacted
scope = drive
token = redacted
A log from the command with the -vv
flag
I don't have any log entries for what I was doing as it was just a move using mv
So let me explain my setup here. I don't run plex on the same box as my docker containers. I use proxmox and I have 2 Hosts. 1 host has my plex LXC on it and then my other host has my docker lxc on it.
I used to download everything to one location and then I would NFS share that location with plex and use mergerfs on both plex as well as my docker server so plex could see the files after sonarr/radarr moved the files to their temporary location until midnight rolled around when my upload script would fire off and move everything up to the drive.
After reading about how vfs-cache-mode full works I liked the idea of things just going up to the drive as quickly as possible instead of at the end of the day.
So my sonarr has been backlogged severely this week and I've been trying to figure out why I thought it might be disk usage contention between it and another VM so I stopped what I thought would be causing the problem with that other VM. No change
So then today I manually moved a file for Sonarr to see for myself what was going on. As it was just a mv it should've been pretty instant I thought since I was moving it from where it was stored at to the drive and the vfs-cache is located on the same mount point as where the file was at so I figured the system would just move it like moving anything else normally and it'd be quick.
The file I moved was 10GB in size and it took over 10 mins for it to end up in the vfs-cache. I confirmed that while I did move it to /gdrive I looked in the cache dir and there it is just chilling in there.
I've checked my systemd scripts against what Animosity has (thank you for sharing with all of us) and everything looks fine. Yeah I have mine set to write back after 5m because I want plex to pick up the file once it sees it.
Thank you all for your help here and I look forward to finding a solution here. If I need to make the /gmedia mount within the /docker mount point thats fine I don't mind doing that but I just figured since the end location was on the same mount as the original location it would move it nice and fast.