Trying to achieve continuous transfer of files for rclone move, especially rclone picking up newly added files of the source folder. I've been using sonarr&radarr and previously made sonarr to move files to remote (Onedrive) directly but this resulted in very high iowait and thus unnecessary system slowdown.
I have tried using mergerfs to merge local and remote then moving by scheduled cron job, but some of my files are huge and take several hours to transfer and this causes new downloaded files to queue up. For example when a single 4k remux file has started to move, it uses maximum 100mbps and takes about two hours to upload and during that 2 hours other files cannot move and I cannot utilize 1gbps connection
I think this will be solved by starting a rclone move job that picks up new files that gets added to the source folder. Is there any way to achieve this, either by using rclone commands or scripts?
Run the command 'rclone version' and share the full output of the command.
os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.4.0-100-generic (x86_64)
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Paste command here
The rclone config contents with secrets removed.
type = onedrive
drive_type = business
I might not have been clear with my problem. I currently use --vfs-cache-mode full mount based on your mount service
ExecStart=/usr/bin/rclone mount disk0: /home/spark/media/ \
--dir-cache-time 10s \
--log-file /home/spark/logs/rclone.log \
--log-level INFO \
--vfs-cache-mode full \
--vfs-cache-max-size 30G \
--vfs-cache-max-age 5000h \
The problem is that if I point sonarr/radarr directly to the mount and let them transfer the files, I end up with excessive iowait and system slowdown. If I use your method of using mergerfs, the sonarr importing part goes smoothly but if the scheduled rclone move happens to start when a single 4k remux file is downloaded, it queues up the future downloads and I would like to speed up the process as if it was a mount. (Also when handling large files rclone move seem to be much reliable than mount especially with Onedrive hash mismatch errors)