Rclone SFTP DL Server view media

What is the problem you are having with rclone?

I have a two server setup, one plex and one download server.
Plex Server is using rclone with SFTP connection + merge into the folders to see earlier what files exist before they get moved to cloud.
But somehow rclone mount didnt notice changes. i tried also to do a cronjob with this command
rclone rc vfs/refresh --rc-addr=127.0.0.1:5571
but there appears no changes only when folders changes but not files.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.1.0-1014-oem (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The rclone config contents with secrets removed.

[Unit]
Description=RClone Service Dropbox Movies
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/root/.config/rclone/rclone.conf
RestartSec=5
ExecStart=/usr/bin/rclone mount dlserver:/media/local /media/mount/dlserver \
# This is for allowing users other than the user running rclone access to the mount
--allow-other \
# Dropbox is a polling remote so this value can be set very high and any changes are detected via polling.
--dir-cache-time 9999h \
# Log file location
--log-file /media/mount/rclonedlserver.log \
# Set the log level
--log-level INFO \
# This is setting the file permission on the mount to user and group have the same access and other can read
--umask 002 \
# This sets up the remote control daemon so you can issue rc commands locally
--rc \
# This is the default port it runs on
--rc-addr 127.0.0.1:5571 \
# no-auth is used as no one else uses my server and it is not a shared seedbox
--rc-no-auth \
# The local disk used for caching
--cache-dir=/media/mount/cache \
--vfs-cache-mode full \
# This limits the cache size to the value below
--vfs-cache-max-size 50G \
# Speed up the reading: Use fast (less accurate) fingerprints for change detection
--vfs-fast-fingerprint \
# Set chunk size
#--vfs-read-chunk-size 32M \
# Wait before uploading
--vfs-write-back 1h \
# This limits the age in the cache if the size is reached and it removes the oldest files first
--vfs-cache-max-age 3h \
# Disable HTTP2
--disable-http2 \
# Set the tpslimit
--tpslimit 12 \
# Set the tpslimit-burst
--tpslimit-burst 0



ExecStop=/usr/bin/fusermount -uz /media/mount/dlserver/
ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --url 127.0.0.1:5571 _async=true
Restart=on-failure


[Install]
WantedBy=multi-user.target
________

[dlserver]
type = sftp
host = xxx.xxx.xxx.xxx
pass = xxxxxx
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

Let me if you know something about that.

Can you share the debug log as you neglected that as that has your answer :slight_smile:

2023/06/26 18:33:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:34:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:35:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:36:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:37:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:38:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:39:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:40:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:41:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:42:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:43:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:44:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:45:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:46:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)
2023/06/26 18:47:50 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 16.062Mi (was 16.062Mi)

this or should i do debug log with --vv?

-vv and share the whole thing.

i changed my crontab command to rclone rc vfs/refresh ecursive=true --rc-addr=127.0.0.1:5571
now works as it should.

thank you. Because of it's a live system i can not restart mount, or merge. otherwise my server notice a downtime during peak hours. :slight_smile:

That's not quite it.

The SFTP remote isn't polling as the debug log shows that.

[felix@gemini rclone]$ rclone mount SFTP: /home/felix/test -vv
2023/06/26 14:17:09 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/06/26 14:17:09 DEBUG : rclone: Version "v1.63.0-beta.7013.d05393b86.fix-6986-times" starting with parameters ["rclone" "mount" "SFTP:" "/home/felix/test" "-vv"]
2023/06/26 14:17:09 DEBUG : Creating backend with remote "SFTP:"
2023/06/26 14:17:09 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/06/26 14:17:09 DEBUG : sftp://felix@localhost:22/: New connection [::1]:42692->[::1]:22 to "SSH-2.0-OpenSSH_9.0"
2023/06/26 14:17:09 DEBUG : sftp://felix@localhost:22/: Shell type "unix" from config
2023/06/26 14:17:09 DEBUG : sftp://felix@localhost:22/: Relative path resolved to "/home/felix"
2023/06/26 14:17:09 DEBUG : sftp://felix@localhost:22/: Using root directory "/home/felix"
2023/06/26 14:17:09 INFO  : sftp://felix@localhost:22/: poll-interval is not supported by this remote
2023/06/26 14:17:09 DEBUG : sftp://felix@localhost:22/: Mounting on "/home/felix/test"
2023/06/26 14:17:09 DEBUG : : Root:
2023/06/26 14:17:09 DEBUG : : >Root: node=/, err=<nil>

Without a poll interval, you have a directory cache time of:

So if something modifies the files outside of the remote, it'll take 9999 hours for that to expire and be seen in the remote.

The command you ran was a one time thing to refresh the cache as that hides the issue.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.