Detect changes to rclone mounted s3 bucket

Help and Support
1

What is the problem you are having with rclone?

I am using rclone to mount an s3 bucket where we "drop off" nginx configs which are then picked up by a script that watches the directory. I have been looking at using the inotifywait tool to watch the folder but it doesn't seem to be able to detect the changes into that folder when a file is uploaded to the bucket. However, if I directly go to the mounted directory and modify a file then the change is detected by inotifywait. Any ideas why it might behave this way?

What is your rclone version (output from rclone version)

1.55.1-DEV

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sudo rclone mount s3:bucket-name /etc/nginx/customer-config --config=/etc/rclone/rclone.conf --daemon --vfs-cache-mode writes --dir-cache-time 10s

monitoring using the inotifywait tool using:

sudo inotifywait -m /etc/nginx/customer-config

The rclone config contents with secrets removed.

[s3]
type = s3
provider = AWS
env_auth = true
region = eu-west-2
acl = private

A log from the command with the -vv flag

not really an issue that would be identified from the log I don't think...

2

Fuse doesn't support that as I guess you are using a mount but since you neglected the help template, I can just guess.

3

thanks. updated with the template.

4

I retested with your config and seems to work fine for me.

felix@gemini:~$ sudo inotifywait -m /home/felix/test
Setting up watches.
Watches established.
/home/felix/test/ OPEN,ISDIR
/home/felix/test/ ACCESS,ISDIR
/home/felix/test/ ACCESS,ISDIR
/home/felix/test/ CLOSE_NOWRITE,CLOSE,ISDIR
/home/felix/test/ MODIFY hosts
/home/felix/test/ OPEN hosts
/home/felix/test/ MODIFY hosts
/home/felix/test/ CLOSE_WRITE,CLOSE hosts

I just did

felix@gemini:~/test$ cp /etc/hosts .
felix@gemini:~/test$

Simple mount command:

felix@gemini:~$ rclone mount DB: /home/felix/test -vvv --allow-other

Debug log from mount:

2021/12/10 12:15:00 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2021/12/10 12:15:00 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "DB:" "/home/felix/test" "-vvv" "--allow-other"]
2021/12/10 12:15:00 DEBUG : Creating backend with remote "DB:"
2021/12/10 12:15:00 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/12/10 12:15:00 DEBUG : Dropbox root '': Mounting on "/home/felix/test"
2021/12/10 12:15:00 DEBUG : : Root:
2021/12/10 12:15:00 DEBUG : : >Root: node=/, err=<nil>
2021/12/10 12:15:02 DEBUG : /: Attr:
2021/12/10 12:15:02 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/10 12:15:09 DEBUG : : Statfs:
2021/12/10 12:15:09 DEBUG : : >Statfs: stat={Blocks:53207891968 Bfree:4178150339 Bavail:4178150339 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/12/10 12:15:14 DEBUG : /: Attr:
2021/12/10 12:15:14 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/10 12:15:15 DEBUG : /: ReadDirAll:
2021/12/10 12:15:15 DEBUG : /: >ReadDirAll: item=6, err=<nil>
2021/12/10 12:15:15 DEBUG : /: Lookup: name="SecureM"
2021/12/10 12:15:15 DEBUG : /: >Lookup: node=SecureM/, err=<nil>
2021/12/10 12:15:15 DEBUG : SecureM/: Attr:
2021/12/10 12:15:15 DEBUG : SecureM/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/10 12:15:15 DEBUG : /: Lookup: name="SecureT"
2021/12/10 12:15:15 DEBUG : /: >Lookup: node=SecureT/, err=<nil>
2021/12/10 12:15:15 DEBUG : SecureT/: Attr:
2021/12/10 12:15:15 DEBUG : SecureT/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/10 12:15:15 DEBUG : /: Lookup: name="SecureU"
2021/12/10 12:15:15 DEBUG : /: >Lookup: node=SecureU/, err=<nil>
2021/12/10 12:15:15 DEBUG : SecureU/: Attr:
2021/12/10 12:15:15 DEBUG : SecureU/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/10 12:15:15 DEBUG : /: Lookup: name="hosts"
2021/12/10 12:15:15 DEBUG : /: >Lookup: node=hosts, err=<nil>
2021/12/10 12:15:15 DEBUG : hosts: Attr:
2021/12/10 12:15:15 DEBUG : hosts: >Attr: a=valid=1s ino=0 size=130 mode=-rw-rw-r--, err=<nil>
2021/12/10 12:15:21 DEBUG : /: Attr:
2021/12/10 12:15:21 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/10 12:15:21 DEBUG : /: Lookup: name="hosts"
2021/12/10 12:15:21 DEBUG : /: >Lookup: node=hosts, err=<nil>
2021/12/10 12:15:21 DEBUG : hosts: Attr:
2021/12/10 12:15:21 DEBUG : hosts: >Attr: a=valid=1s ino=0 size=130 mode=-rw-rw-r--, err=<nil>
2021/12/10 12:15:21 DEBUG : hosts: Open: flags=OpenWriteOnly
2021/12/10 12:15:21 DEBUG : hosts: Open: flags=O_WRONLY
2021/12/10 12:15:21 DEBUG : hosts: >Open: fd=hosts (w), err=<nil>
2021/12/10 12:15:21 DEBUG : hosts: >Open: fh=&{hosts (w)}, err=<nil>
2021/12/10 12:15:21 DEBUG : hosts: Setattr: a=Setattr [ID=0x24 Node=0x5 Uid=1000 Gid=1000 Pid=439957] size=0 handle=INVALID-0x0 lockowner
2021/12/10 12:15:21 DEBUG : hosts: Truncating 1 file handles
2021/12/10 12:15:21 DEBUG : hosts: >Setattr: err=<nil>
2021/12/10 12:15:21 DEBUG : hosts: Attr:
2021/12/10 12:15:21 DEBUG : hosts: >Attr: a=valid=1s ino=0 size=130 mode=-rw-rw-r--, err=<nil>
2021/12/10 12:15:21 DEBUG : &{hosts (w)}: Write: len=130, offset=0
2021/12/10 12:15:21 DEBUG : : Added virtual directory entry vAddFile: "hosts"
2021/12/10 12:15:21 DEBUG : &{hosts (w)}: >Write: written=130, err=<nil>
2021/12/10 12:15:21 DEBUG : &{hosts (w)}: Flush:
2021/12/10 12:15:21 DEBUG : Dropbox root '': File to upload is small (130 bytes), uploading instead of streaming
2021/12/10 12:15:21 DEBUG : hosts: Uploading chunk 1/1
2021/12/10 12:15:22 DEBUG : hosts: Uploading chunk 2/1
2021/12/10 12:15:23 DEBUG : Dropbox root '': Adding "/hosts" to batch
2021/12/10 12:15:23 DEBUG : Dropbox root '': Batch idle for 500ms so committing
2021/12/10 12:15:23 DEBUG : Dropbox root '': Committing sync batch length 1 starting with: /hosts
2021/12/10 12:15:23 DEBUG : Dropbox root '': Upload batch completed in 72.535494ms
2021/12/10 12:15:23 DEBUG : Dropbox root '': Committed sync batch length 1 starting with: /hosts
2021/12/10 12:15:23 DEBUG : hosts: dropbox = 9f3bf463abb0cc75de58ba43cb37a5246862722fa748219b2de6b910e8995a66 OK
2021/12/10 12:15:23 INFO  : hosts: Copied (new)
2021/12/10 12:15:23 DEBUG : : Added virtual directory entry vAddFile: "hosts"
2021/12/10 12:15:23 DEBUG : &{hosts (w)}: >Flush: err=<nil>
2021/12/10 12:15:23 DEBUG : &{hosts (w)}: Release:
2021/12/10 12:15:23 DEBUG : hosts: WriteFileHandle.Release nothing to do
2021/12/10 12:15:23 DEBUG : &{hosts (w)}: >Release: err=<nil>
2021/12/10 12:15:24 DEBUG : : Statfs:
2021/12/10 12:15:24 DEBUG : : >Statfs: stat={Blocks:53207891968 Bfree:4178150339 Bavail:4178150339 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/12/10 12:15:39 DEBUG : : Statfs:
2021/12/10 12:15:39 DEBUG : : >Statfs: stat={Blocks:53207891968 Bfree:4178150339 Bavail:4178150339 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/12/10 12:15:54 DEBUG : : Statfs:
2021/12/10 12:15:54 DEBUG : : >Statfs: stat={Blocks:53207891968 Bfree:4178150339 Bavail:4178150339 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/12/10 12:16:00 DEBUG : Dropbox root '': Checking for changes on remote
2021/12/10 12:16:09 DEBUG : : Statfs:
2021/12/10 12:16:09 DEBUG : : >Statfs: stat={Blocks:53207891968 Bfree:4178150339 Bavail:4178150339 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/12/10 12:16:24 DEBUG : : Statfs:
2021/12/10 12:16:24 DEBUG : : >Statfs: stat={Blocks:53207891968 Bfree:4178150339 Bavail:4178150339 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
5

ok, thanks I will check my config out again and test. I have reverted to my previous "watcher" which uses screen in the background using an ls piped to md5 has looking for changes.. which works well

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.