File added to S3 on one machine not visible on 2nd machine unless mount is restarted

Help and Support
1

What is the problem you are having with rclone?

We have a web service that uploads a file to a rclone mounted path. The file makes it up just fine but other machines that have that same mount cannot see the files unless I restart the service (unmount/remount). I have tried --disable ListR as I spotted a similar issue for Google Drive. It is not a matter of waiting and the file will eventually show up either, my last test I came back after 35 minutes and it still was not there.

You may notice the timings for the writing/reading server in the logs below. I have tested with a delay before checking on the reading server and the file still never shows up.

From the writing server:

[root@ip-172-31-0-250 ~]# ls -la /var/www/customer-staging/ams/v2/files/clientname/assets/2020/11/
total 1479
-rw-r--r--. 1 apache apache 125295 Nov 23 14:00 cycle-15fbbbad021d9e6.36131588.png
-rw-r--r--. 1 apache apache 125295 Nov 23 14:00 cycle.png
-rw-r--r--. 1 apache apache 628813 Nov 23 14:00 R2_Grid-1-15fbb8202476935.85988717.jpg
-rw-r--r--. 1 apache apache 628813 Nov 23 14:00 R2_Grid-1.jpg
-rw-r--r--. 1 apache apache   4907 Nov 23 14:00 thumb.jpeg

and the reading server

[root@ip-172-31-0-75 log]# ls -la /var/www/customer-staging/ams/v2/files/clientname/assets/2020/11/
total 865
-rw-r--r--. 1 apache apache 125295 Nov 23 14:02 cycle-15fbbbad021d9e6.36131588.png
-rw-r--r--. 1 apache apache 125295 Nov 23 14:02 cycle.png
-rw-r--r--. 1 apache apache 628813 Nov 23 14:02 R2_Grid-1-15fbb8202476935.85988717.jpg
-rw-r--r--. 1 apache apache   4907 Nov 23 14:02 thumb.jpeg

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: linux/amd64
  • go version: go1.15.5

Also tested with 1.50

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 8

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The ExecStart from our systemd config:

/usr/bin/rclone mount \
        --config=/root/.rclone/staging-files-all/rclone-s3.conf \
        --allow-other \
        --cache-tmp-upload-path=/tmp/rclone/upload \
        --cache-chunk-path=/tmp/rclone/chunks \
        --cache-workers=8 \
        --cache-writes \
        --cache-dir=/tmp/rclone/vfs \
        --cache-db-path=/tmp/rclone/db \
        --no-modtime \
        --drive-use-trash \
        --stats=0 \
        --checkers=16 \
        --bwlimit=40M \
        --dir-cache-time=60m \
        --vfs-cache-mode writes \
        --uid=48 \
        --gid=48 \
        --disable ListR \
        --log-file=/var/log/rclone.log \
        -vv \
        --cache-info-age=60m staging-files-all:/staging-files-all/ /var/www/customer-staging/ams/v2/files

The rclone config contents with secrets removed.

[staging-files-all]
type = s3
provider = AWS
env_auth = false
access_key_id = ********************
secret_access_key = **********************************
region = eu-west-1
location_constraint = eu-west-1
acl = private
endpoint =

A log from the command with the -vv flag

Upload server

2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: Open: flags=OpenReadOnly
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: Open: flags=O_RDONLY
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: newRWFileHandle: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >newRWFileHandle: err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >Open: fd=clientname/assets/2020/11/R2_Grid-1.jpg (rw), err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >Open: fh=&{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: Attr: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >Attr: a=valid=1s ino=0 size=628813 mode=-rw-r--r--, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=16384, offset=0
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): _readAt: size=16384, off=0
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): openPending: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: truncate to size=628813
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11: Added virtual directory entry vAdd: "R2_Grid-1.jpg"
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >openPending: err=<nil>
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:0 Size:16384} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >_readAt: n=16384, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=16384, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=32768, offset=16384
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=65536, offset=49152
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): _readAt: size=32768, off=16384
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:16384 Size:32768} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): _readAt: size=65536, off=49152
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:49152 Size:65536} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >_readAt: n=32768, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=32768, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >_readAt: n=65536, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=65536, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=131072, offset=114688
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): _readAt: size=131072, off=114688
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=131072, offset=245760
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:114688 Size:131072} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): _readAt: size=131072, off=245760
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >_readAt: n=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:245760 Size:131072} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >_readAt: n=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=131072, offset=376832
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): _readAt: size=131072, off=376832
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=122880, offset=507904
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): _readAt: size=122880, off=507904
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:376832 Size:131072} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:507904 Size:120909} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >_readAt: n=120909, err=EOF
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=120909, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >_readAt: n=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Flush: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): RWFileHandle.Flush
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Flush: err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Release: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): RWFileHandle.Release
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): close: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: Open: flags=OpenReadOnly
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: Open: flags=O_RDONLY
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: setting modification time to 2020-11-23 14:02:18.828107175 +0000 GMT m=+150.616662072
2020/11/23 14:02:20 INFO  : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: queuing for upload in 5s
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6a40): >close: err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Release: err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: newRWFileHandle: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >newRWFileHandle: err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >Open: fd=clientname/assets/2020/11/R2_Grid-1.jpg (rw), err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >Open: fh=&{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: Attr: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: >Attr: a=valid=1s ino=0 size=628813 mode=-rw-r--r--, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=16384, offset=0
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): _readAt: size=16384, off=0
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): openPending: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: truncate to size=628813
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11: Added virtual directory entry vAdd: "R2_Grid-1.jpg"
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >openPending: err=<nil>
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:0 Size:16384} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >_readAt: n=16384, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=16384, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=32768, offset=16384
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): _readAt: size=32768, off=16384
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=65536, offset=49152
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:16384 Size:32768} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): _readAt: size=65536, off=49152
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >_readAt: n=32768, err=<nil>
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:49152 Size:65536} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=32768, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >_readAt: n=65536, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=65536, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=131072, offset=114688
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): _readAt: size=131072, off=114688
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=131072, offset=245760
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:114688 Size:131072} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): _readAt: size=131072, off=245760
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:245760 Size:131072} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >_readAt: n=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >_readAt: n=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=131072, offset=376832
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Read: len=122880, offset=507904
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): _readAt: size=131072, off=376832
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:376832 Size:131072} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >_readAt: n=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=131072, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): _readAt: size=122880, off=507904
2020/11/23 14:02:20 DEBUG : vfs cache: looking for range={Pos:507904 Size:120909} in [{Pos:0 Size:628813}] - present true
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >_readAt: n=120909, err=EOF
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Read: read=120909, err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Flush: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): RWFileHandle.Flush
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Flush: err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: Release: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): RWFileHandle.Release
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): close: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: setting modification time to 2020-11-23 14:02:18.828107175 +0000 GMT m=+150.616662072
2020/11/23 14:02:20 INFO  : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: queuing for upload in 5s
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg(0xc000cb6c40): >close: err=<nil>
2020/11/23 14:02:20 DEBUG : &{clientname/assets/2020/11/R2_Grid-1.jpg (rw)}: >Release: err=<nil>
2020/11/23 14:02:25 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: starting upload
2020/11/23 14:02:25 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: MD5 = 232482546a285793b4b8b8f7aaaf7bbd OK
2020/11/23 14:02:25 INFO  : clientname/assets/2020/11/R2_Grid-1.jpg: Copied (new)
2020/11/23 14:02:25 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: fingerprint now "628813,2020-11-23 14:02:18.828107175 +0000 UTC,232482546a285793b4b8b8f7aaaf7bbd"
2020/11/23 14:02:25 DEBUG : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: writeback object to VFS layer
2020/11/23 14:02:25 DEBUG : clientname/assets/2020/11: Added virtual directory entry vAdd: "R2_Grid-1.jpg"
2020/11/23 14:02:25 INFO  : clientname/assets/2020/11/R2_Grid-1.jpg: vfs cache: upload succeeded try #1

Reading server

2020/11/23 14:02:20 DEBUG : /: Lookup: name="clientname"
2020/11/23 14:02:20 DEBUG : /: >Lookup: node=clientname/, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/: Attr: 
2020/11/23 14:02:20 DEBUG : clientname/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/: Lookup: name="assets"
2020/11/23 14:02:20 DEBUG : clientname/: >Lookup: node=clientname/assets/, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/: Attr: 
2020/11/23 14:02:20 DEBUG : clientname/assets/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/: Lookup: name="2020"
2020/11/23 14:02:20 DEBUG : clientname/assets/: >Lookup: node=clientname/assets/2020/, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/: Attr: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/: Lookup: name="11"
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/: >Lookup: node=clientname/assets/2020/11/, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/: Attr: 
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/: Lookup: name="R2_Grid-1.jpg"
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/: >Lookup: node=<nil>, err=no such file or directory
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/: Lookup: name="R2_Grid-1.jpg"
2020/11/23 14:02:20 DEBUG : clientname/assets/2020/11/: >Lookup: node=<nil>, err=no such file or directory
2

Currently, RClone does not support polling for S3 remotes which means that any changes will appear only after the directory cache expires, which is 60m as per your command.

3

Thanks for the quick reply @darthShadow. Curious what the implications of a very short directory cache expiration would be?

Testing --dir-cache-time=1s worked a treat. After the 5s delay from vfs cache: queuing for upload in 5s the file shows up.

From Dir-cache-time vs poll-interval I see the concern about a very low poll interval but from a bit of search I can't see if there are the same for a very low directory cache expiration.

4

The only concern is the time taken to list directories or files. If you have a large enough listing, then it can take few minutes or even an hour to list so you will have to check and adjust this accordingly for your use-case.

I would recommend setting it to 15m or 30m instead and see if that works for your use-case since that seems like a reasonable compromise between availability of changes and caching.

5

Btw, which S3 remote is this? AWS, Minio or something else? Just asking since polling for S3 is in the pipeline so will add this specific variant to check for that too.

6

hi, about the polling in the pipeline
does that mean polling functionality already exists in s3 and rclone does not support that.
or
aws is adding polling as a new feature?
i use wasabi and aws.

7

Depending on the remote, some have API support for polling which needs to be added to rclone and some don't have API support which cannot be added.

AWS, Minio & Ceph do, according to their docs, but Wasabi doesn't: https://wasabi.com/wp-content/themes/wasabi/docs/API_Guide/index.html#t=topics%2FOperations_on_Buckets_Not_Supported_in_Wasabi.htm

1 Like
8

This remote in particular is S3, but we also use Minio for a different use case.

I will keep an eye out for the S3 polling as the current version of our application needs rather quick directory updates.

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.