What is the problem you are having with rclone?
I have quite a particular case, causing high memory usage on RClone (+1.7GB RAM) using mount with Google Drive remote
The usecase is the following one:
- A program can get a file to download and send it to the client, or receive it from client.
- If the client needs the file, the path is known and the file is fetched from mountpoint and sent to the user.
- If user makes an upload, a temporary file is created, preallocated, filled with the data and then moved to it's final place - everything on the mountpoint.
- Files are about 2-4 MiB in size
- Mounted unit currently has ~600 GiB of files of that kind.
The issues I'm having are:
Slow path access due to large folders that need to be listed from the remote - already fixed using
--fast-listand doing a periodic
vfs/refreshremote command, specially after the unit has been mounted.
Slow IO on writing - probably because of locks while doing the prealloc - filling - move operation. I have solved this doing a local cache using mergerfs and an every-minute rclone move.
At the beggining I was pointing the server directly to RClone unit, but that caused slowness on uploads and lot's of IO wait processes.
To speed up this part, I'm doing the
rclone moveto the mountpoint, so I can make use of directory cache and speed up the move a lot (from 200KiB/s moving directly to the remote vs 2MiB/s when moving through FUSE mount)
- Excesive memory usage - after mounting the unit and refresh the duirectory cache, the RClone processes take like 400 MiB of RAM. But after the file dancing starts, it easy hits 1.8 GiB of memory. Also, stoping all related programs and processes - leaving the mountpoint with no traffic - doesn´t help, as the memory is not returned.
Can I tweak the mount command to avoid that excessive memory usage?
I currently have this command:
/usr/bin/rclone mount gdrive: /mnt/remote\ --allow-non-empty \ --buffer-size 16M \ --dir-cache-time 48h \ --fast-list \ --drive-use-trash=false \ --use-mmap \ --rc \ --no-modtime \ --write-back-cache \ --vfs-cache-max-age 1m \ --vfs-cache-mode writes \ --vfs-read-chunk-size 64M \ --vfs-cache-poll-interval 2m0s \ --vfs-read-chunk-size-limit 256M \ --vfs-write-wait 2s \ --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"