Transfer speed gradually slows - rclone cron job, with mergerFS setup

Issue:

I am noticing a data transfer slow-down as time progresses. I am using rclone to move files to my encrypted Google Drive on a daily basis. This happens via script, initiated by cron.

Version:

rclone v1.51.0-134-g6765303d-beta

  • os/arch: linux/amd64
  • go version: go1.14.1

Upload script cron runs:

/home/path/scripts/rclone-upload.sh

#!/bin/bash

lock_file="$HOME/scripts/rclone-upload.lock"

trap 'rm -f "$lock_file"; exit 0' SIGINT SIGTERM
if [ -e "$lock_file" ]
then
    echo "Rclone upload script is already running."
    exit
else
    rm "$HOME"/scripts/rclone-upload.log
    touch "$lock_file"
    "$HOME"/bin/rclone move "$HOME"/MergerFS/local.store/ "$HOME"/MergerFS/rclone.mount/ --config="$HOME"/.config/rclone/rclone.conf --drive-chunk-size 64M --tpslimit 5 --drive-acknowledge-abuse=true -vvv --delete-empty-src-dirs --fast-list --bwlimit=12M --use-mmap --transfers=2 --checkers=4 --log-file "$HOME"/scripts/rclone-upload.log
    rm -f "$lock_file"
    trap - SIGINT SIGTERM
    exit
fi

rclone config:

/home/path/.config/rclone/rclone.conf

[gdrive]
type = drive
client_id = [redacted]
client_secret = [redacted]
scope = drive
token = [redacted]
root_folder_id = [redacted]

[gcache]
type = cache
remote = gdrive:/gdrive
plex_url = [redacted]
plex_username = [redacted]
plex_password = [redacted]
chunk_size = 16M
info_age = 2d
chunk_total_size = 20G
plex_token = [redacted]

[gcrypt]
type = crypt
remote = gcache:/crypt
filename_encryption = standard
directory_name_encryption = true
password = [redacted]
password2 = [redacted]

rclone service:

/home/path/.config/systemd/user/rclone-vfs.service

[Unit]
Description=RClone VFS Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
KillMode=none
Environment=GOMAXPROCS=2

ExecStart=/home/path/bin/rclone mount gcrypt: /home/path/MergerFS/rclone.mount \
  --allow-other \
  --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36' \
  --config /home/path/.config/rclone/rclone.conf \
  --use-mmap \
  --dir-cache-time 168h \
  --timeout 1h \
  --umask 002 \
  --poll-interval=1m \
  --vfs-cache-mode writes \
  --vfs-read-chunk-size 64M \
  --vfs-read-chunk-size-limit 2048M \
  --tpslimit 10 \
  --tpslimit-burst 10 \
  --log-level INFO \
  --log-file /home/path/.config/rclone/rclone.log
ExecStop=/bin/fusermount -uz /home/path/MergerFS/rclone.mount
Restart=on-failure

[Install]
WantedBy=default.target

mergerfs service:

/home/path/.config/systemd/user/mergerfs.service

[Unit]
Description = MergerFS Mount
After=rclone-vfs.service
RequiresMountsFor=/home/path/MergerFS/local.store
RequiresMountsFor=/home/path/MergerFS/rclone.mount

[Service]
Type=forking
KillMode=none
ExecStart=/home/path/bin/mergerfs /home/path/MergerFS/local.store:/home/path/MergerFS/rclone.mount /home/path/Mount -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true,threads=8
ExecStop=/bin/fusermount -uz /home/path/Mount
Restart=on-failure

[Install]
WantedBy=default.target

Slow-down examples from log:

2020/04/17 10:31:03 INFO  : 
Transferred:   	   40.069G / 77.939 GBytes, 51%, 11.211 MBytes/s, ETA 57m38s
Checks:               102 / 104, 98%
Deleted:               51
Transferred:           51 / 103, 50%
Elapsed time:    1h0m59.8s

2020/04/17 11:31:03 INFO  : 
Transferred:   	   45.177G / 77.939 GBytes, 58%, 6.372 MBytes/s, ETA 1h27m44s
Checks:               114 / 116, 98%
Deleted:               57
Transferred:           57 / 103, 55%
Elapsed time:    2h0m59.8s

2020/04/17 12:31:03 INFO  : 
Transferred:   	   45.177G / 77.939 GBytes, 58%, 4.260 MBytes/s, ETA 2h11m15s
Checks:               114 / 116, 98%
Deleted:               57
Transferred:           57 / 103, 55%
Elapsed time:    3h0m59.8s

2020/04/17 13:31:03 INFO  : 
Transferred:   	   45.177G / 77.939 GBytes, 58%, 3.199 MBytes/s, ETA 2h54m45s
Checks:               114 / 116, 98%
Deleted:               57
Transferred:           57 / 103, 55%
Elapsed time:    4h0m59.8s

2020/04/17 14:31:03 INFO  : 
Transferred:   	   45.177G / 77.939 GBytes, 58%, 2.562 MBytes/s, ETA 3h38m16s
Checks:               114 / 116, 98%
Deleted:               57
Transferred:           57 / 103, 55%
Elapsed time:    5h0m59.8s

2020/04/17 15:31:03 INFO  : 
Transferred:   	   45.177G / 77.939 GBytes, 58%, 2.136 MBytes/s, ETA 4h21m47s
Checks:               114 / 116, 98%
Deleted:               57
Transferred:           57 / 103, 55%
Elapsed time:    6h0m59.8s

Logs will occasionally begin like this:

2020/04/17 09:30:02 DEBUG : rclone: Version "v1.51.0-134-g6765303d-beta" starting with parameters ["/home/path/bin/rclone" "move" "/home/path/MergerFS/local.store/" "/home/path/MergerFS/rclone.mount/" "--config=/home/path/.config/rclone/rclone.conf" "--drive-chunk-size" "64M" "--tpslimit" "5" "--drive-acknowledge-abuse=true" "-vvv" "--delete-empty-src-dirs" "--fast-list" "--bwlimit=12M" "--use-mmap" "--transfers=2" "--checkers=4" "--log-file" "/home/path/scripts/rclone-upload.log"]
2020/04/17 09:30:02 DEBUG : Using config file from "/home/path/.config/rclone/rclone.conf"
2020/04/17 09:30:02 INFO  : Starting bandwidth limiter at 12MBytes/s
2020/04/17 09:30:02 INFO  : Starting HTTP transaction limiter: max 5 transactions/s with burst 1
2020/04/17 09:30:03 DEBUG : Local file system at /home/path/MergerFS/rclone.mount/: Using server side directory move
2020/04/17 09:30:03 INFO  : Local file system at /home/path/MergerFS/rclone.mount/: Server side directory move failed - fallback to file moves: can't copy directory - destination already exists
2020/04/17 09:30:03 DEBUG : Local file system at /home/path/MergerFS/rclone.mount/: Waiting for checks to finish
2020/04/17 09:30:03 DEBUG : Local file system at /home/path/MergerFS/rclone.mount/: Waiting for transfers to finish
2020/04/17 09:30:07 DEBUG : [redacted_01].avi: Can't move: rename /home/path/MergerFS/local.store/[redacted_01].avi /home/path/MergerFS/rclone.mount/[redacted_01].avi: invalid cross-device link: trying copy
2020/04/17 09:30:07 DEBUG : [redacted_01].avi: Can't move, switching to copy
2020/04/17 09:30:07 DEBUG : [redacted_02].avi: Can't move: rename /home/path/MergerFS/local.store/[redacted_02].avi /home/path/MergerFS/rclone.mount/[redacted_02].avi: invalid cross-device link: trying copy
2020/04/17 09:30:07 DEBUG : [redacted_02].avi: Can't move, switching to copy
2020/04/17 09:30:07 DEBUG : preAllocate: got error on fallocate, trying combination 1/2: operation not supported
2020/04/17 09:30:07 DEBUG : preAllocate: got error on fallocate, trying combination 2/2: operation not supported

Questions:

  • Currently, the upload happens through the mount, which uses the cache. Would it be better if I created a second cache and crypt combo that points to the same crypt directory in my gdrive?
  • I feel like I am missing some understanding of the various buffer and chunk size settings, as though I have them set incorrectly or sub-optimally for uploading files over the mount at a constant rate. Do the settings make sense where I have them?
  • These services and configs use settings recommended by my hosting provider. Should I be worried about apps like Sonarr and Radarr interacting with the mergerfs mount while the command is moving files out of it?

Thanks for any help in advance.

For uploads, then will run quicker most likely if you create a remote which doesn't include cache, then use rclone copy to copy straight to that remote.

--drive-chunk-size will help a lot at a cost of more memory, especially with your limited number of tps as uploading one chunk will only count as one transaction.

They will use up transactions and so affect the upload in that way if that is what you mean?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.