Rclone sync problem unequal file size

showkubii · October 24, 2019, 3:17am

rclone script

rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_1.log" lottodress28_TD1:"My Documents" lottodress28_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_2.log" lottodresswow28_TD1:"My Documents" lottodresswow28_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_3.log" n28embymanager_TD1:"My Documents" n28embymanager_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_4.log" neungcoc001_TD1:"My Documents" neungcoc001_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_5.log" neungcoc002_TD1:"My Documents" neungcoc002_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_6.log" neungcoc003_TD1:"My Documents" neungcoc003_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_7.log" neungcoc004_TD1:"My Documents" neungcoc004_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_8.log" neungcoc005_TD1:"My Documents" neungcoc005_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_9.log" neungcoc006_TD1:"My Documents" neungcoc006_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_10.log" neungcoc007_TD1:"My Documents" neungcoc007_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_11.log" neungcoc008_TD1:"My Documents" neungcoc008_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_12.log" neungcoc009_TD1:"My Documents" neungcoc009_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_13.log" neungcoc010_TD1:"My Documents" neungcoc010_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_14.log" neungcoc011_TD1:"My Documents" neungcoc011_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_15.log" neungcoc012_TD1:"My Documents" neungcoc012_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_16.log" neungcoc013_TD1:"My Documents" neungcoc013_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_17.log" neungcoc014_TD1:"My Documents" neungcoc014_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_18.log" neungcoc015_TD1:"My Documents" neungcoc015_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_19.log" neungcoc016_TD1:"My Documents" neungcoc016_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_20.log" neungcoc017_TD1:"My Documents" neungcoc017_TD2:"My Documents"
rclone sync --fast-list --size-only --transfers=1 --max-transfer=650G --checkers 8 --drive-server-side-across-configs=true -vv --filter-from "D:\Programs\rclone\filter.txt" --log-file "D:\Programs\rclone\TD2_21.log" neungcoc018_TD1:"My Documents" neungcoc018_TD2:"My Documents"

sync complete.

i check size

how to fix help me.

run on rclone v.1.49.5

ncw · October 24, 2019, 6:48am

Is neungcoc012_TD1 a team drive (shared drive)?

We have seen this behaviour from team drives before where it takes a while for the directory listing to settle down.

Is it OK now?

showkubii · October 24, 2019, 7:15am

neungcoc012_TD1 : team drive

rclone script team drive to team drive

log file

2019/10/24 14:18:52 INFO : Google drive root 'My Documents': Waiting for transfers to finish
2019/10/24 14:18:52 INFO : Waiting for deletions to finish
2019/10/24 14:18:52 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 11099 / 11099, 100%
Transferred: 0 / 0, -
Elapsed time: 0s

2019/10/24 14:18:52 DEBUG : 35 go routines active
2019/10/24 14:18:52 DEBUG : rclone: Version "v1.49.5" finishing with parameters ["rclone" "sync" "--fast-list" "--size-only" "--transfers=1" "--max-transfer=650G" "--checkers" "8" "--drive-server-side-across-configs=true" "-vv" "--filter-from" "D:\Programs\rclone\filter.txt" "--log-file" "D:\Programs\rclone\TD2_21.log" "neungcoc018_TD1:My Documents" "neungcoc018_TD2:My Documents"]

check size

darthShadow · October 24, 2019, 7:23am

Are you sure the filters are not the cause for the difference in sizes and number of files ?

showkubii · October 24, 2019, 7:28am

I'm sure.

filter.txt

- *.jpg
- *.png
- *.txt
- *.xml
- *.dat
- *.bat

source no file extension .txt .xml .dat .bat

ncw · October 24, 2019, 7:30am

I think this is caused by eventual consistency in the team drive listings.

Try running the rclone size again after waiting some time and see if it stabilizes.

There are many reports of this in the rclone issues.

darthShadow · October 24, 2019, 9:37am

True, but that doesn't explain the increase in the number of files in the destination compared to the source. Generally, the count is lower instead of higher. One possibility is that they are dupes.

@showkubii Would you mind running a rclone dedupe on both the source & destination and check the sizes after that ?

ncw · October 24, 2019, 11:41am

If there is being eventual consistency and a retry happens then rclone won't see it has uploaded the files that are missing and upload them again, thus causing duplicates.

Dedupe is a good idea!

thestigma · October 24, 2019, 9:56pm

Yea I've noted the same on my teamdrives.

If you run operations quickly in series then --fast-list may not be fully up to date immediately after finishing.
It seems to be correct within a minute or two usually - but as a result of this I generally try to avoid using --fast-list for any sequential series of operations that rely on the previous command. I suppose an alternative would just be to force a short pause in the script, but that's less reliable since it's difficult to know how long it takes to be consistent - and this probably varies too.

showkubii · October 25, 2019, 12:08am

I checking rclone dedupe

rclone dedupe --dedupe-mode newest

showkubii · October 25, 2019, 12:12am

Thank you everyone for helping me.

ncw · October 25, 2019, 12:04pm

That is something else that we should stick in the docs!

thestigma · October 25, 2019, 12:05pm

Hehe, alright I get the hint

system · October 28, 2019, 12:05pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.