Rclone sync problem unequal file size

rclone script

rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_1.log" lottodress28_TD1:"My Documents" lottodress28_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_2.log" lottodresswow28_TD1:"My Documents" lottodresswow28_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_3.log" n28embymanager_TD1:"My Documents" n28embymanager_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_4.log" neungcoc001_TD1:"My Documents" neungcoc001_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_5.log" neungcoc002_TD1:"My Documents" neungcoc002_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_6.log" neungcoc003_TD1:"My Documents" neungcoc003_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_7.log" neungcoc004_TD1:"My Documents" neungcoc004_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_8.log" neungcoc005_TD1:"My Documents" neungcoc005_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_9.log" neungcoc006_TD1:"My Documents" neungcoc006_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_10.log" neungcoc007_TD1:"My Documents" neungcoc007_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_11.log" neungcoc008_TD1:"My Documents" neungcoc008_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_12.log" neungcoc009_TD1:"My Documents" neungcoc009_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_13.log" neungcoc010_TD1:"My Documents" neungcoc010_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_14.log" neungcoc011_TD1:"My Documents" neungcoc011_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_15.log" neungcoc012_TD1:"My Documents" neungcoc012_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_16.log" neungcoc013_TD1:"My Documents" neungcoc013_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_17.log" neungcoc014_TD1:"My Documents" neungcoc014_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_18.log" neungcoc015_TD1:"My Documents" neungcoc015_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_19.log" neungcoc016_TD1:"My Documents" neungcoc016_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_20.log" neungcoc017_TD1:"My Documents" neungcoc017_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_21.log" neungcoc018_TD1:"My Documents" neungcoc018_TD2:"My Documents"

sync complete.

i check size

how to fix help me.

run on rclone v.1.49.5

Is neungcoc012_TD1 a team drive (shared drive)?

We have seen this behaviour from team drives before where it takes a while for the directory listing to settle down.

Is it OK now?

1 Like

neungcoc012_TD1 : team drive

rclone script team drive to team drive

log file

2019/10/24 14:18:52 INFO : Google drive root 'My Documents': Waiting for transfers to finish
2019/10/24 14:18:52 INFO : Waiting for deletions to finish
2019/10/24 14:18:52 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 11099 / 11099, 100%
Transferred: 0 / 0, -
Elapsed time: 0s

2019/10/24 14:18:52 DEBUG : 35 go routines active
2019/10/24 14:18:52 DEBUG : rclone: Version "v1.49.5" finishing with parameters ["rclone" "sync" "--fast-list" "--size-only" "--transfers=1" "--max-transfer=650G" "--checkers" "8" "--drive-server-side-across-configs=true" "-vv" "--filter-from" "D:\Programs\rclone\filter.txt" "--log-file" "D:\Programs\rclone\TD2_21.log" "neungcoc018_TD1:My Documents" "neungcoc018_TD2:My Documents"]

check size

Are you sure the filters are not the cause for the difference in sizes and number of files ?

I'm sure.

filter.txt

- *.jpg
- *.png
- *.txt
- *.xml
- *.dat
- *.bat

source no file extension .txt .xml .dat .bat

I think this is caused by eventual consistency in the team drive listings.

Try running the rclone size again after waiting some time and see if it stabilizes.

There are many reports of this in the rclone issues.

True, but that doesn't explain the increase in the number of files in the destination compared to the source. Generally, the count is lower instead of higher. One possibility is that they are dupes.

@showkubii Would you mind running a rclone dedupe on both the source & destination and check the sizes after that ?

1 Like

If there is being eventual consistency and a retry happens then rclone won't see it has uploaded the files that are missing and upload them again, thus causing duplicates.

Dedupe is a good idea!

Yea I've noted the same on my teamdrives.

If you run operations quickly in series then --fast-list may not be fully up to date immediately after finishing.
It seems to be correct within a minute or two usually - but as a result of this I generally try to avoid using --fast-list for any sequential series of operations that rely on the previous command. I suppose an alternative would just be to force a short pause in the script, but that's less reliable since it's difficult to know how long it takes to be consistent - and this probably varies too.

1 Like

I checking rclone dedupe

rclone dedupe --dedupe-mode newest

Thank you everyone for helping me. :heart_eyes::heart_eyes::heart_eyes:

1 Like

That is something else that we should stick in the docs!

Hehe, alright I get the hint :wink:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.