--fast-list sometimes skips google drive shortcuts

Suspected Bug
1

What is the problem you are having with rclone?

when running rclone sync using --fast-list on a large google shared drive it misses syncing some files in google shortcut directories. the files will be deleted from the local drive.
running rclone sync again without --fast-list will restore the local files.

on a drive with a small number of files the behavior does not seem to occur. i suspect it requires a large or deeply nested drive.

has anyone else experienced this?

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.2

  • os/version: arch (64 bit)
  • os/kernel: 5.15.68-1-lts (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.3
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using?

Google Drive

The command you were trying to run:

rclone sync --fast-list DRIVE: /mnt/drive
1 Like
2

Could you share with me the results of

rclone lsf -R DRIVE:

And

rclone lsf --disable ListR -R DRIVE:

Which should show the difference between the recursive listing used by --fast-list and the non recursive listing.

The file names may be confidential so you can email the listings to me nick@craig-wood.com if you prefer. Put a link to the forum post in the email if you do please.

3

You are correct, unfortunately the file names may be confidential. i am not sure what i can do about that at the moment.

I counted the number of results from your above commands though, and they differ:
rclone lsf --disable ListR -R DRIVE: | wc -l
6419

rclone lsf -R DRIVE: | wc -l
6257
^ this one actually gives different results on separate invocations: 6241, 6378.

I also created a diff of the listings from those two commands and examined the remote locations of each file missing from rclone lsf -R DRIVE: - they all resided within some google drive shortcut folder.

(As context, my google drive repository contains many (300+) subfolders directly on the root DRIVE)
I additionally tried these commands constrained to just one of those subfolders, like:
rclone lsf --disable ListR -R DRIVE:subfolder
rclone lsf -R DRIVE:subfolder
and the results were always identical as expected.

4

That is good info

Is it that the shortcut is not followed at all? Or that things only go missing in the directories under shortcuts?

If you could post some (5 maybe) differing runs and what it should be, but pipe through

tr -c '/\n' 'X'

To clean the file names that would be useful. That command should replace all the file names with XXX

You could zip these and share them on Google drive.

Could you also check your drive for duplicates using rclone dedupe? You will need --disable ListR on this command too.

1 Like
5

There are some duplicates on the drive but i don't think they are relevant as none of the duplicate files match with any problem files/directories i encountered.

I tried:
rclone dedupe --dry-run --dedupe-mode list DRIVE:
with and without --disable ListR and got the same results.

Is it that the shortcut is not followed at all? Or that things only go missing in the directories under shortcuts?

I am unsure but the shortcut directories themselves are always listed when doing rclone lsf -R DRIVE: - just not the files within them.

It also looks like, on separate executions of rclone lsf -R DRIVE: either all the files under a particular shortcut are missing or all of them exist.

Shortcut files (not directories) also appear to be listed.

Finally, here are some outputs of the commands:
rclone lsf --disable ListR -R DRIVE: (1 output)
rclone lsf -R DRIVE: (5 outputs)
(both sorted and unsorted)

I tried to do better than just replacing the file names with XXX's. instead i replaced every unique file and directory name with a random word. always the same random word for the same file/directory. the same word replacement was also used in each file so they can all be compared.

You can run diff on the sorted .txt files to see which files were missed.

1 Like