Rclone sync with --files-from tag does not remove deleted files/folders from destination

Hi,

I am using --files-from tag with the copy and sync command to migrate data from local to Google cloud storage. I am using flat file generated from an external script to track/migrate files on my local storage.

Steps to reproduce:

  1. I copied a source directory with the copy command and --files-from tag (File calculated through os walk). It worked fine.
  2. Later I deleted some files and folders from inside the source directory. I calculated the flat-file again and the new file does not contain the deleted files. I used the new flat file with sync command.

Theoretically, Rclone should delete the files present on the destination which are not in the flat file. But the observed behavior is that the Rclone only tracks files present in the flat file. For a file which is not present in the flat file, but some kind of operation has been performed on that file (Update, Delete), Rclone will not migrate the changes to destination for that file.

Is this the intended behavior? I mean, I can calculate the difference in files using a script but this is just rewriting the behavior intended for the sync command.

Any thoughts on this issue?

Thanks

EDIT:

I am using Rclone: Version "v1.48.0-DEV" on a Windows 10 Machine.

Added Sample Flat file used:

With copy command (I uploaded a directory say C:\Users\Manish\Test to destination) with the following tag --files-from C:\Users\Manish\Temp\rclone_files.txt where content of rclone_files.txt is:

/_5/_
/_6/_
/_7/_
/cvgr4wedfgbht4rwefgr3ef.rtf
/sdefr4wefgre.txt
/xcfedqwe.txt

Logs when copying:

PS C:\Users\manis> go run .\rclone.go copy C:\Users\manis\Desktop\0\ --files-from C:\Users\manish\rclone_report.txt --config .\rclone.config -vvP
2019/07/08 22:41:04 DEBUG : rclone: Version "v1.48.0-DEV" starting with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]
2019/07/08 22:41:04 DEBUG : Using config file from "C:\Users\manis\rclone.config"
2019-07-08 22:41:06 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for checks to finish
2019-07-08 22:41:06 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for transfers to finish
2019-07-08 22:41:06 INFO : sdefr4wefgre.txt: Copied (new) ETA 0s
2019-07-08 22:41:06 INFO : xcfedqwe.txt: Copied (new)/s, ETA 0ss
2019-07-08 22:41:06 INFO : 7/: Copied (new)98 Bytes/s, ETA 0s
2019-07-08 22:41:07 INFO : cvgr4wedfgbht4rwefgr3ef.rtf: Copied (new)
2019-07-08 22:41:07 INFO : 6/: Copied (new)84 Bytes/s, ETA 0s
2019-07-08 22:41:08 INFO : 5/: Copied (new)65 Bytes/s, ETA 0s
Transferred: 282 / 282 Bytes, 100%, 58 Bytes/s, ETA 0s-
Errors: 0 / 0, -7% 7/: 0% /0, 0/s, --
Checks: 0 / 0, -3% 6/: 0% /0, 0/s, ---
Transferred: 6 / 6, 100% 6/: 0% /0, 0/s, - -
Elapsed time: 4.8s 6/: 0% /0, 0/s, - -
2019/07/08 22:41:08 INFO : 5/: 0% /0, 0/s, - -
Transferred: 282 / 282 Bytes, 100%, 58 Bytes/s, ETA 0s -
Errors: 0 xcfedqwe.txt: transferring
Checks: 0 / 0, -ht4rwefgr3ef.rtf: transferring
Transferred: 6 / 6, 100%
Elapsed time: 4.8s
2019/07/08 22:41:08 DEBUG : 16 go routines active
2019/07/08 22:41:09 INFO : The success report is available at C:\Users\manis\AppData\Local\rclone\success_07082019224105.csv
2019/07/08 22:41:10 DEBUG : success_07082019224105.csv: Couldn't find file - need to transfer
2019/07/08 22:41:10 INFO : success_07082019224105.csv: Copied (new)
2019/07/08 22:41:11 INFO : The failure report is available at C:\Users\manis\AppData\Local\rclone\failure_07082019224105.csv
2019/07/08 22:41:12 DEBUG : failure_07082019224105.csv: Couldn't find file - need to transfer
2019/07/08 22:41:12 INFO : failure_07082019224105.csv: Copied (new)
2019/07/08 22:41:12 DEBUG : rclone: Version "v1.48.0-DEV" finishing with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]

Then I deleted folder /_5/_ and added a new file dwewwd.txt on local machine but the delete changes were not reflected on destination. I used the updated rclone_files.txt with the sync command this time and the content of the rclone_files.txt is:

/_6/_
/_7/_
/cvgr4wedfgbht4rwefgr3ef.rtf
/sdefr4wefgre.txt
/xcfedqwe.txt
dwewwd.txt

Logs when syncing

PS C:\Users\manis> go run .\rclone.go copy C:\Users\manis\Desktop\0\ --files-from C:\Users\manis\rclone_report.txt --config .\rclone.config -vvP
2019/07/08 22:44:27 DEBUG : rclone: Version "v1.48.0-DEV" starting with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]
2019/07/08 22:44:27 DEBUG : Using config file from "C:\Users\manis\rclone.config"
2019-07-08 22:44:29 DEBUG : 5/: Excluded from sync (and deletion)
2019-07-08 22:44:29 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for checks to finish
2019-07-08 22:44:29 DEBUG : cvgr4wedfgbht4rwefgr3ef.rtf: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : cvgr4wedfgbht4rwefgr3ef.rtf: Unchanged skipping
2019-07-08 22:44:29 DEBUG : sdefr4wefgre.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : xcfedqwe.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : 6/: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : 7/: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : sdefr4wefgre.txt: Unchanged skipping
2019-07-08 22:44:29 DEBUG : xcfedqwe.txt: Unchanged skipping0s
2019-07-08 22:44:29 DEBUG : 6/: Unchanged skipping/s, ETA 0s
2019-07-08 22:44:29 DEBUG : 7/: Unchanged skipping/s, ETA 0s
2019-07-08 22:44:29 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for transfers to finish
2019-07-08 22:44:29 INFO : dwewwd.txt: Copied (new)/s, ETA 0s
Transferred: 18 / 18 Bytes, 100%, 6 Bytes/s, ETA 0s
Errors: 0 / 5, 100% 6/: checking 0/s, -
Checks: 5 / 5, 100% 6/: checking 0/s, -
Transferred: 1 / 1, 100% 6/: checking 0/s, -
Elapsed time: 2.7s dwewwd.txt:100% /18, 0/s, -
2019/07/08 22:44:29 INFO : dwewwd.txt:100% /18, 0/s, -
Transferred: 18 / 18 Bytes, 100%, 6 Bytes/s, ETA 0s, -
Errors: 0
Checks: 5 / 5, 100%
Transferred: 1 / 1, 100%
Elapsed time: 2.8s
2019/07/08 22:44:29 DEBUG : 10 go routines active
2019/07/08 22:44:30 INFO : The success report is available at C:\Users\manis\AppData\Local\rclone\success_07082019224428.csv
2019/07/08 22:44:31 DEBUG : success_07082019224428.csv: Couldn't find file - need to transfer
2019/07/08 22:44:32 INFO : success_07082019224428.csv: Copied (new)
2019/07/08 22:44:32 INFO : The failure report is available at C:\Users\manis\AppData\Local\rclone\failure_07082019224428.csv
2019/07/08 22:44:33 DEBUG : failure_07082019224428.csv: Couldn't find file - need to transfer
2019/07/08 22:44:33 INFO : failure_07082019224428.csv: Copied (new)
2019/07/08 22:44:33 DEBUG : rclone: Version "v1.48.0-DEV" finishing with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]

Can you share rclone version, log file of the issue with -vv on it ?

Hi, I updated the code with a sample file used and the Rclone version.

No log file still. Is there a reason you are using a DEV version and not stable?

Because of this issue: https://github.com/ncw/rclone/issues/3248

Awesome. Can you share a log file.

Hi Added Logs from -vv Flag.

That's the whole log? The command is cut off so I can't tell what you ran unfortunately.

Hi, Sorry about that. Added complete logs.

@Animosity022 Please check the complete logs...

Seems to be a duplicate of:

Which seems to be a bug when using filter-from, if the other OP didn't file a bug yet, I'd file one on the github and reference these 2 threads for more information.

@Animosity022 I don't think a bug has been filed for this issue yet. I checked on github. Can you please confirm?
Thanks

I didn't see one when I looked earlier. I think you are fine to file it.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.