Accounting for difference between rclone summary and find file count

Help and Support
1

What is the problem you are having with rclone?

At the completion of an rclone session, the log shows the following:
Transferred: 141.303G / 141.303 GBytes, 100%, 524.104 kBytes/s, ETA 0s
Checks: 61 / 61, 100%
Transferred: 52449 / 52449, 100%
Elapsed time: 78h31m44.6s

The rclone command is:
rclone sync ~/Documents somewhere:there --copy-links

It is run by the owner of the ~/Documents, who is also the owner of all files thereunder.

The target has a quota of 400 GB

If I look at the count according to ... it says the contents of the directory are:
59’298 items, totalling 147.7 GB.

Running the following command
find ~/Documents/ -type f | wc -l
yields the count 52518

The rclone log does not show any errors, failures or warnings.
There is only one symbolic link, to a file, both under ~/Documents

If I run rclone check, I get the following summary:

2023/01/07 10:02:25 NOTICE: Local file system at /home/robert/Documents: 1 files missing
2023/01/07 10:02:25 NOTICE: Swift container default: 13 differences found
2023/01/07 10:02:25 NOTICE: Swift container default: 13 errors while checking
2023/01/07 10:02:25 NOTICE: Swift container default: 52506 matching files
2023/01/07 10:02:25 Failed to check with 14 errors: last error was: 13 differences found

The 9 missing files are files created after the rclone session started. There were also 4 files whose size changed after being synced, accounting for the 13 errors. In any case, the file counts still do not add up.

So, what accounts for the difference in file count between the number of files listed by find (=52518-9) and the number of files copies by rclone (52449)? Why does rclone report 52449 files copied in the log, but rclone check reports 52506 matching files when run immediately after the first rclone session?

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV

  • os/arch: linux/amd64
  • go version: go1.18

Which cloud storage system are you using? (eg Google Drive)

A swift system

The command you were trying to run (eg rclone copy /tmp remote:tmp)

```rclone sync ~/Documents somewhere:there --copy-links``

The rclone config contents with secrets removed.

``[my_project]
type = swift
user = me
key = password
auth = https://somewhere
domain = default
tenant = tenant_name
tenant_domain = default
region = RegionOne
storage_url =
auth_version =```

A log from the command with the -vv flag

see above
2

If you want to compare the source and directory counts then run rclone size --copy-links remote: on them both. This will give you a like for like comparison.

This may be treating symlinks differently.

You can also try listing the find to a file and comparing it with the rclone equivalent generated with rclone lsf -R --copy-links to find the differences.

3

Thank you for that advice.
When I run rclone size for the remote platform, it yields a file count that corresponds to the count of the local platform file count using find. I am not sure how one would run rclone size for the local platform

So I am comforted by this result, which could hardly be a coincidence.

That being said it leaves as a mystery which the initial session running rclone sync said that the count of files copied was 69 files fewer. I leave this to others to determine if there is a bug somewhere.

In any case, I consider this issue resolved.

4

hi,

with rclone, local is just another storage system.
https://rclone.org/local/

rclone size ~/Documents
Total objects: 13 (13)
Total size: 14.968 KiB (15327 Byte)
5

Difficult to say - maybe you copied some files in an initial test, or maybe there was a retry which messed up the numbers. In general if rclone check says you are good, then you are good :slight_smile:

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.