Incorrect debug output

[Harry]

rclone v1.51.0-135-g1af9fcbb-beta
- os/arch: linux/amd64
- go version: go1.14.1


./rclone-beta sync gdrive: onedrive:mydir --fast-list --stats=0 --delete-before -vv


2020/03/30 14:57:23 NOTICE: G925FXXU3DPCM_G925FOJK3DPC4_CEL.zip: Duplicate object found in source - ignoring
2020/03/30 14:57:23 NOTICE: G960FXXU6CSGD_G960FOXM6CSH1_EUR.zip: Duplicate object found in source - ignoring
2020/03/30 14:57:23 NOTICE: J111MUBU0AQC2_J111MUUB0AQC1_CHO.zip: Duplicate object found in source - ignoring
2020/03/30 14:57:23 NOTICE: J200HXXU0APD1_J200HODD0APC1_BNG.zip: Duplicate object found in source - ignoring
2020/03/30 14:57:23 NOTICE: J730GDXU7CSG3_J730GXNZ7CSG3_XNZ.zip: Duplicate object found in source - ignoring
2020/03/30 14:57:23 NOTICE: N920GDDU3BQA4_N920GODD3BQA1_INS.zip: Duplicate object found in source - ignoring
2020/03/30 14:57:23 INFO  : There was nothing to transfer
2020/03/30 15:02:06 NOTICE: G925FXXU3DPCM_G925FOJK3DPC4_CEL.zip: Duplicate object found in source - ignoring
2020/03/30 15:02:06 NOTICE: G960FXXU6CSGD_G960FOXM6CSH1_EUR.zip: Duplicate object found in source - ignoring
2020/03/30 15:02:06 NOTICE: J111MUBU0AQC2_J111MUUB0AQC1_CHO.zip: Duplicate object found in source - ignoring
2020/03/30 15:02:06 NOTICE: J200HXXU0APD1_J200HODD0APC1_BNG.zip: Duplicate object found in source - ignoring
2020/03/30 15:02:06 NOTICE: J730GDXU7CSG3_J730GXNZ7CSG3_XNZ.zip: Duplicate object found in source - ignoring
2020/03/30 15:02:06 NOTICE: N920GDDU3BQA4_N920GODD3BQA1_INS.zip: Duplicate object found in source - ignoring

There is 2 problem with the logs.

1. Why rclone ouputs INFO : There was nothing to transfer when clearly there is files to sync/copy.. ?

2. Why the duplicate file names echoed twice? There are total 6 duplicates but echoed twice each one of them.

[ncw (Nick Craig-Wood)]

@Harry

Why rclone ouputs INFO : There was nothing to transfer when clearly there is files to sync/copy.. ?

Rclone didn't find any files to transfer because of the duplicates.

@Harry

Why the duplicate file names echoed twice? There are total 6 duplicates but echoed twice each one of them.

I think this is because you are using --delete-before which has to do two traversals over the source directory listing.

[Harry]

@ncw

Rclone didn't find any files to transfer because of the duplicates.

That's exactly the problem. The execution did not stop there. It actually copied 500+ files after that.

@ncw

I think this is because you are using --delete-before which has to do two traversals over the source directory listing.

It means rclone first did a full list on source and dest, then again did a full list on source and dest ? If yes then it mean twice api calls on both remotes?

[ncw (Nick Craig-Wood)]

@Harry

That's exactly the problem. The execution did not stop there. It actually copied 500+ files after that.

Ah, that is because of --delete-before too. I just pushed a fix to the latest beta for that.

@Harry

It means rclone first did a full list on source and dest, then again did a full list on source and dest ? If yes then it mean twice api calls on both remotes?

Yes It does say this in the docs

Specifying the value --delete-before will delete all files present on the destination, but not on the source before starting the transfer of any new or updated files. This uses two passes through the file systems, one for the deletions and one for the copies.

Though if you are using --fast-list then in theory rclone could re-use the cached --fast-list contents.