Issue:
I am noticing a data transfer slow-down as time progresses. I am using rclone to move files to my encrypted Google Drive on a daily basis. This happens via script, initiated by cron.
Version:
rclone v1.51.0-134-g6765303d-beta
- os/arch: linux/amd64
- go version: go1.14.1
Upload script cron runs:
/home/path/scripts/rclone-upload.sh
#!/bin/bash
lock_file="$HOME/scripts/rclone-upload.lock"
trap 'rm -f "$lock_file"; exit 0' SIGINT SIGTERM
if [ -e "$lock_file" ]
then
echo "Rclone upload script is already running."
exit
else
rm "$HOME"/scripts/rclone-upload.log
touch "$lock_file"
"$HOME"/bin/rclone move "$HOME"/MergerFS/local.store/ "$HOME"/MergerFS/rclone.mount/ --config="$HOME"/.config/rclone/rclone.conf --drive-chunk-size 64M --tpslimit 5 --drive-acknowledge-abuse=true -vvv --delete-empty-src-dirs --fast-list --bwlimit=12M --use-mmap --transfers=2 --checkers=4 --log-file "$HOME"/scripts/rclone-upload.log
rm -f "$lock_file"
trap - SIGINT SIGTERM
exit
fi
rclone config:
/home/path/.config/rclone/rclone.conf
[gdrive]
type = drive
client_id = [redacted]
client_secret = [redacted]
scope = drive
token = [redacted]
root_folder_id = [redacted]
[gcache]
type = cache
remote = gdrive:/gdrive
plex_url = [redacted]
plex_username = [redacted]
plex_password = [redacted]
chunk_size = 16M
info_age = 2d
chunk_total_size = 20G
plex_token = [redacted]
[gcrypt]
type = crypt
remote = gcache:/crypt
filename_encryption = standard
directory_name_encryption = true
password = [redacted]
password2 = [redacted]
rclone service:
/home/path/.config/systemd/user/rclone-vfs.service
[Unit]
Description=RClone VFS Service
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
KillMode=none
Environment=GOMAXPROCS=2
ExecStart=/home/path/bin/rclone mount gcrypt: /home/path/MergerFS/rclone.mount \
--allow-other \
--user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36' \
--config /home/path/.config/rclone/rclone.conf \
--use-mmap \
--dir-cache-time 168h \
--timeout 1h \
--umask 002 \
--poll-interval=1m \
--vfs-cache-mode writes \
--vfs-read-chunk-size 64M \
--vfs-read-chunk-size-limit 2048M \
--tpslimit 10 \
--tpslimit-burst 10 \
--log-level INFO \
--log-file /home/path/.config/rclone/rclone.log
ExecStop=/bin/fusermount -uz /home/path/MergerFS/rclone.mount
Restart=on-failure
[Install]
WantedBy=default.target
mergerfs service:
/home/path/.config/systemd/user/mergerfs.service
[Unit]
Description = MergerFS Mount
After=rclone-vfs.service
RequiresMountsFor=/home/path/MergerFS/local.store
RequiresMountsFor=/home/path/MergerFS/rclone.mount
[Service]
Type=forking
KillMode=none
ExecStart=/home/path/bin/mergerfs /home/path/MergerFS/local.store:/home/path/MergerFS/rclone.mount /home/path/Mount -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true,threads=8
ExecStop=/bin/fusermount -uz /home/path/Mount
Restart=on-failure
[Install]
WantedBy=default.target
Slow-down examples from log:
2020/04/17 10:31:03 INFO :
Transferred: 40.069G / 77.939 GBytes, 51%, 11.211 MBytes/s, ETA 57m38s
Checks: 102 / 104, 98%
Deleted: 51
Transferred: 51 / 103, 50%
Elapsed time: 1h0m59.8s
2020/04/17 11:31:03 INFO :
Transferred: 45.177G / 77.939 GBytes, 58%, 6.372 MBytes/s, ETA 1h27m44s
Checks: 114 / 116, 98%
Deleted: 57
Transferred: 57 / 103, 55%
Elapsed time: 2h0m59.8s
2020/04/17 12:31:03 INFO :
Transferred: 45.177G / 77.939 GBytes, 58%, 4.260 MBytes/s, ETA 2h11m15s
Checks: 114 / 116, 98%
Deleted: 57
Transferred: 57 / 103, 55%
Elapsed time: 3h0m59.8s
2020/04/17 13:31:03 INFO :
Transferred: 45.177G / 77.939 GBytes, 58%, 3.199 MBytes/s, ETA 2h54m45s
Checks: 114 / 116, 98%
Deleted: 57
Transferred: 57 / 103, 55%
Elapsed time: 4h0m59.8s
2020/04/17 14:31:03 INFO :
Transferred: 45.177G / 77.939 GBytes, 58%, 2.562 MBytes/s, ETA 3h38m16s
Checks: 114 / 116, 98%
Deleted: 57
Transferred: 57 / 103, 55%
Elapsed time: 5h0m59.8s
2020/04/17 15:31:03 INFO :
Transferred: 45.177G / 77.939 GBytes, 58%, 2.136 MBytes/s, ETA 4h21m47s
Checks: 114 / 116, 98%
Deleted: 57
Transferred: 57 / 103, 55%
Elapsed time: 6h0m59.8s
Logs will occasionally begin like this:
2020/04/17 09:30:02 DEBUG : rclone: Version "v1.51.0-134-g6765303d-beta" starting with parameters ["/home/path/bin/rclone" "move" "/home/path/MergerFS/local.store/" "/home/path/MergerFS/rclone.mount/" "--config=/home/path/.config/rclone/rclone.conf" "--drive-chunk-size" "64M" "--tpslimit" "5" "--drive-acknowledge-abuse=true" "-vvv" "--delete-empty-src-dirs" "--fast-list" "--bwlimit=12M" "--use-mmap" "--transfers=2" "--checkers=4" "--log-file" "/home/path/scripts/rclone-upload.log"]
2020/04/17 09:30:02 DEBUG : Using config file from "/home/path/.config/rclone/rclone.conf"
2020/04/17 09:30:02 INFO : Starting bandwidth limiter at 12MBytes/s
2020/04/17 09:30:02 INFO : Starting HTTP transaction limiter: max 5 transactions/s with burst 1
2020/04/17 09:30:03 DEBUG : Local file system at /home/path/MergerFS/rclone.mount/: Using server side directory move
2020/04/17 09:30:03 INFO : Local file system at /home/path/MergerFS/rclone.mount/: Server side directory move failed - fallback to file moves: can't copy directory - destination already exists
2020/04/17 09:30:03 DEBUG : Local file system at /home/path/MergerFS/rclone.mount/: Waiting for checks to finish
2020/04/17 09:30:03 DEBUG : Local file system at /home/path/MergerFS/rclone.mount/: Waiting for transfers to finish
2020/04/17 09:30:07 DEBUG : [redacted_01].avi: Can't move: rename /home/path/MergerFS/local.store/[redacted_01].avi /home/path/MergerFS/rclone.mount/[redacted_01].avi: invalid cross-device link: trying copy
2020/04/17 09:30:07 DEBUG : [redacted_01].avi: Can't move, switching to copy
2020/04/17 09:30:07 DEBUG : [redacted_02].avi: Can't move: rename /home/path/MergerFS/local.store/[redacted_02].avi /home/path/MergerFS/rclone.mount/[redacted_02].avi: invalid cross-device link: trying copy
2020/04/17 09:30:07 DEBUG : [redacted_02].avi: Can't move, switching to copy
2020/04/17 09:30:07 DEBUG : preAllocate: got error on fallocate, trying combination 1/2: operation not supported
2020/04/17 09:30:07 DEBUG : preAllocate: got error on fallocate, trying combination 2/2: operation not supported
Questions:
- Currently, the upload happens through the mount, which uses the cache. Would it be better if I created a second cache and crypt combo that points to the same crypt directory in my gdrive?
- I feel like I am missing some understanding of the various buffer and chunk size settings, as though I have them set incorrectly or sub-optimally for uploading files over the mount at a constant rate. Do the settings make sense where I have them?
- These services and configs use settings recommended by my hosting provider. Should I be worried about apps like Sonarr and Radarr interacting with the mergerfs mount while the command is moving files out of it?
Thanks for any help in advance.