Rclone missing directories

What is the problem you are having with rclone?

Rclone is being used to present an Openstack Swift object store as a file system for CIFS use. Right now CIFS isn't the problem since Rclone (after working for years) has stopped showing entire directories' contents. Some of these directories have 100,000+ entries in them which wasn't a problem before.

What is your rclone version (output from rclone version)

rclone v1.55.1

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: static
  • go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18.04LTS / 64 bit

Which cloud storage system are you using? (eg Google Drive)

Openstack Swift

The command you were trying to run (eg rclone copy /tmp remote:tmp)

ls or ls -al

The rclone config contents with secrets removed.

[remote]
type = swift
user = storage_XXXX
key = XXXX
auth = http://XXX:5000/v2.0
tenant = XXX
auth_version = 2

I simplified the launch command:

       /usr/bin/rclone --config=/etc/ABCconf/ABCstore.config mount remote:ABCVIDEOSTORE /root/ABCVIDEOSTORE/ABCVIDEOSTORE \
                        --transfers=8 \
                        --log-level DEBUG --log-file /var/log/rclone/ABCVIDEOSTORE.log  &


A log from the command with the -vv flag

The relevant log excerpt is short:

2021/05/30 14:43:46 DEBUG : : >Statfs: stat={Blocks:312985161511 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2021/05/30 14:43:47 DEBUG : ABCDvideos/ABCDLibrary/: Attr:
2021/05/30 14:43:47 DEBUG : ABCDvideos/ABCDLibrary/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/05/30 14:43:47 DEBUG : ABCDvideos/ABCDLibrary/: ReadDirAll:
2021/05/30 14:43:47 DEBUG : ABCDvideos/ABCDLibrary/: >ReadDirAll: item=2, err=<nil>
2021/05/30 14:43:47 DEBUG : /: Attr:
2021/05/30 14:43:47 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2021/05/30 14:43:47 DEBUG : : Statfs:
2021/05/30 14:43:47 DEBUG : : >Statfs: stat={Blocks:312985161511 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>

I ran the ls command and got nothing returned. 

# ls -al
total 0
drwxr-xr-x 1 root root 0 May 30 14:43 .
drwxr-xr-x 1 root root 0 May 30 14:42 ..
#

https://pastebin.com/B2bLNMpi

shows a bunch of directory look ups that do exist on the object store, but are failing. 
The total volume size reported by df -kh seems correct, but the total file size reported by du -kh is vastly understated. 

The following excerpt shows the relevant directory being rescanned at the end of 5 minutes, but no errors, and no contents result.

2021/05/30 14:56:02 DEBUG : /: Lookup: name="ABCDvideos"
2021/05/30 14:56:02 DEBUG : : Re-reading directory (5m0.483601361s old)
2021/05/30 14:56:02 DEBUG : /: >Lookup: node=ABCDvideos/, err=
2021/05/30 14:56:02 DEBUG : ABCDvideos/: Attr:
2021/05/30 14:56:02 DEBUG : ABCDvideos/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2021/05/30 14:56:02 DEBUG : ABCDvideos/: Lookup: name="ABCDLibrary"
2021/05/30 14:56:02 DEBUG : ABCDvideos: Re-reading directory (5m0.457610857s old)
2021/05/30 14:56:02 DEBUG : ABCDvideos/: >Lookup: node=ABCDvideos/ABCDLibrary/, err=
2021/05/30 14:56:02 DEBUG : ABCDvideos/ABCDLibrary/: Attr:
2021/05/30 14:56:02 DEBUG : ABCDvideos/ABCDLibrary/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2021/05/30 14:56:02 DEBUG : ABCDvideos/ABCDLibrary/: Lookup: name="8a164c4c4ba14570862759f627d61ca7"
2021/05/30 14:56:02 DEBUG : ABCDvideos/ABCDLibrary: Re-reading directory (5m0.524703817s old)
2021/05/30 14:56:02 DEBUG : /: Attr:


I have tried a wide variety of caching/no caching and these settings and the rclone version were all working stably until a few days ago.

If rclone were having problems with the object store, I'm assuming it would log something. I did see some kernel timeouts, but rebooted and they have not reoccurred.

Thanks in advance!

Can you post the full debug log?

Its about 5mbs and covers two restarts including one where I turned vfs-writes back on.

https://zerobin.net/?0e5667da6a71f28c#fesN2c5fkpNgsRbO+VoEF/y9+G3k2DLUxx0ZoNc+h5E=

Please let me know if you think anything else would be helpful.

Did you run dedupe? rclone dedupe remote: --dedupe-mode=newest (or largest)

Hi. Thanks for your response.

It didn't occur to try, but at the same point, I don't think we can. Particularly, the way I read it, the renaming of duplicates.

The object store can't really have duplicates, and the data we are storing there is managed (video files) by a separate video management system, so if a file name disappears, it assumes it wasn't written and just writes it again.

This feels like an authentication issue of some kind, but I can't see where it could be occurring or logging

thanks again!