Vfs/refresh does not cache all files

Suspected Bug
1

What is the problem you are having with rclone?

When using rclone rc vfs/refresh recursive=true _async=true as the ExecStartPost of a rclone mount command, there are a lot of files that are not cached.

E.g. find /path/to/mount | wc -l with the above command enabled, I get 16173 as the no. of files.

However without the vfs/refresh command, I get 19633, although it takes nearly 5 mins. to run the command.

rclone version 

rclone v1.58.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-1059-raspi (aarch64)
- os/type: linux
- os/arch: arm64
- go/version: go1.17.9
- go/linking: static
- go/tags: none

Which cloud storage system are you using?

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Environment=RCLONE_UMASK=002
Environment=RCLONE_ALLOW_OTHER=true
Environment=RCLONE_DIR_CACHE_TIME=1000h
Environment=RCLONE_POLL_INTERVAL=15s
Environment=RCLONE_RC_ADDR=127.0.0.1:5805
ExecStart=/usr/bin/rclone mount -q \
        mydrive: /mnt/rc-mydrive \
        --buffer-size=64M \
        --exclude=crypt/** \
        --rc
ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true _async=true --rc-addr=${RCLONE_RC_ADDR}

The rclone config contents with secrets removed.

N/A

A log from the command with the -vv flag

Paste  log here
2

Removing _async=true from the command, brings the count up to 19308, still short of 19633.

3

The find command counts directories and files.

I'd imagine the cache is only 'caching' the files and not the directories.

How are you finding the numbers per job since there's no log, I can't tell what you are doing.

4

I'd imagine the cache is only 'caching' the files and not the directories.

No, both are cached.

It's hard to share detailed logs since the files are rather personal, but I'm able to share the process.

agneev@falcon ~> find /mnt/rc-mydrive|wc -l
18414

# After commenting out refresh command

agneev@falcon /e/s/system> sudo systemctl restart rc-mydrive
agneev@falcon ~> time find /mnt/rc-mydrive | wc -l
19943

________________________________________________________
Executed in  543.08 secs      fish           external
   usr time   77.83 millis    0.56 millis   77.27 millis
   sys time  457.37 millis    1.80 millis  455.57 millis
5

It's also much faster if I use rclone size instead:

agneev@falcon ~> time rclone size /mnt/rc-mydrive/
Total objects: 19.332k (19332)
Total size: 47.119 GiB (50593833203 Byte)

________________________________________________________
Executed in   71.58 secs    fish           external
   usr time    1.10 secs  966.00 micros    1.10 secs
   sys time    2.64 secs  610.00 micros    2.64 secs
6

Sorry as my wording is a poor choice. There isn't actual size / stat information on a directory so it's not really relevant in a caching perspective. You are caching the file metadata along with the directory structure.

If you are mounting and only seeing 18414 items one time and remounting and seeing 19943 items, something is dreadfully wrong.

Have to look at some full debug logs with rclone.conf as well so you have to decide what you want to share or not if you want to get to the bottom of it. You can share with @ncw instead but without any data to look at, addressing it will be impossible.

7

If you are mounting and only seeing 18414 items one time and remounting and seeing 19943 items, something is dreadfully wrong.

This happens only with the rclone vfs/refresh ExecStartPost part of the systemd unit.

If I do not run this process, but instead use rclone size after a successful mount, I do not observe this behavior.

A workaround I've discovered is to use rclone size in a screen session.

ExecStartPost=/usr/bin/screen -dm -S rc-mydrive rclone size /mnt/rc-mydrive
8

Right.

Up to you as we need logs to figure out why.

9

My guess is that this is caused by duplicate file or directory names. Have you runrclone dedupe on the source?

Can you run rclone size mydrive: too please?

10

My guess is that this is caused by duplicate file or directory names. Have you runrclone dedupe on the source?

There are several "shortcuts" in the drive, that contain the majority of files.

While I'm not sure about duplicates, I can confirm that I initially investigated this issue because there were missing files that I noticed in folders where there are no duplicates.

Yes:

agneev@falcon ~> rclone size mydrive:
Total objects: 17.758k (17758)
Total size: 38.582 GiB (41427170208 Byte)

agneev@falcon ~> rclone size /mnt/rc-mydrive
Total objects: 19.332k (19332)
Total size: 47.117 GiB (50591786300 Byte)
11

Have you got a single folder with missing files? We can run more tests on that if you have. I'd like to see the results of rclone lsf in that directory on the mount and on drive.

Running rclone dedupe --dry-run will at least tell you if you have duplicate file names.

Less files! That's odd.

Can you retry that with --disable ListR

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.