Hi,
I am using --files-from tag with the copy and sync command to migrate data from local to Google cloud storage. I am using flat file generated from an external script to track/migrate files on my local storage.
Steps to reproduce:
- I copied a source directory with the copy command and --files-from tag (File calculated through os walk). It worked fine.
- Later I deleted some files and folders from inside the source directory. I calculated the flat-file again and the new file does not contain the deleted files. I used the new flat file with sync command.
Theoretically, Rclone should delete the files present on the destination which are not in the flat file. But the observed behavior is that the Rclone only tracks files present in the flat file. For a file which is not present in the flat file, but some kind of operation has been performed on that file (Update, Delete), Rclone will not migrate the changes to destination for that file.
Is this the intended behavior? I mean, I can calculate the difference in files using a script but this is just rewriting the behavior intended for the sync command.
Any thoughts on this issue?
Thanks
EDIT:
I am using Rclone: Version "v1.48.0-DEV" on a Windows 10 Machine.
Added Sample Flat file used:
With copy command (I uploaded a directory say C:\Users\Manish\Test to destination) with the following tag --files-from C:\Users\Manish\Temp\rclone_files.txt where content of rclone_files.txt is:
/_5/_
/_6/_
/_7/_
/cvgr4wedfgbht4rwefgr3ef.rtf
/sdefr4wefgre.txt
/xcfedqwe.txt
Logs when copying:
PS C:\Users\manis> go run .\rclone.go copy C:\Users\manis\Desktop\0\ --files-from C:\Users\manish\rclone_report.txt --config .\rclone.config -vvP
2019/07/08 22:41:04 DEBUG : rclone: Version "v1.48.0-DEV" starting with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]
2019/07/08 22:41:04 DEBUG : Using config file from "C:\Users\manis\rclone.config"
2019-07-08 22:41:06 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for checks to finish
2019-07-08 22:41:06 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for transfers to finish
2019-07-08 22:41:06 INFO : sdefr4wefgre.txt: Copied (new) ETA 0s
2019-07-08 22:41:06 INFO : xcfedqwe.txt: Copied (new)/s, ETA 0ss
2019-07-08 22:41:06 INFO : 7/: Copied (new)98 Bytes/s, ETA 0s
2019-07-08 22:41:07 INFO : cvgr4wedfgbht4rwefgr3ef.rtf: Copied (new)
2019-07-08 22:41:07 INFO : 6/: Copied (new)84 Bytes/s, ETA 0s
2019-07-08 22:41:08 INFO : 5/: Copied (new)65 Bytes/s, ETA 0s
Transferred: 282 / 282 Bytes, 100%, 58 Bytes/s, ETA 0s-
Errors: 0 / 0, -7% 7/: 0% /0, 0/s, --
Checks: 0 / 0, -3% 6/: 0% /0, 0/s, ---
Transferred: 6 / 6, 100% 6/: 0% /0, 0/s, - -
Elapsed time: 4.8s 6/: 0% /0, 0/s, - -
2019/07/08 22:41:08 INFO : 5/: 0% /0, 0/s, - -
Transferred: 282 / 282 Bytes, 100%, 58 Bytes/s, ETA 0s -
Errors: 0 xcfedqwe.txt: transferring
Checks: 0 / 0, -ht4rwefgr3ef.rtf: transferring
Transferred: 6 / 6, 100%
Elapsed time: 4.8s
2019/07/08 22:41:08 DEBUG : 16 go routines active
2019/07/08 22:41:09 INFO : The success report is available at C:\Users\manis\AppData\Local\rclone\success_07082019224105.csv
2019/07/08 22:41:10 DEBUG : success_07082019224105.csv: Couldn't find file - need to transfer
2019/07/08 22:41:10 INFO : success_07082019224105.csv: Copied (new)
2019/07/08 22:41:11 INFO : The failure report is available at C:\Users\manis\AppData\Local\rclone\failure_07082019224105.csv
2019/07/08 22:41:12 DEBUG : failure_07082019224105.csv: Couldn't find file - need to transfer
2019/07/08 22:41:12 INFO : failure_07082019224105.csv: Copied (new)
2019/07/08 22:41:12 DEBUG : rclone: Version "v1.48.0-DEV" finishing with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]
Then I deleted folder /_5/_ and added a new file dwewwd.txt on local machine but the delete changes were not reflected on destination. I used the updated rclone_files.txt with the sync command this time and the content of the rclone_files.txt is:
/_6/_
/_7/_
/cvgr4wedfgbht4rwefgr3ef.rtf
/sdefr4wefgre.txt
/xcfedqwe.txt
dwewwd.txt
Logs when syncing
PS C:\Users\manis> go run .\rclone.go copy C:\Users\manis\Desktop\0\ --files-from C:\Users\manis\rclone_report.txt --config .\rclone.config -vvP
2019/07/08 22:44:27 DEBUG : rclone: Version "v1.48.0-DEV" starting with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]
2019/07/08 22:44:27 DEBUG : Using config file from "C:\Users\manis\rclone.config"
2019-07-08 22:44:29 DEBUG : 5/: Excluded from sync (and deletion)
2019-07-08 22:44:29 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for checks to finish
2019-07-08 22:44:29 DEBUG : cvgr4wedfgbht4rwefgr3ef.rtf: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : cvgr4wedfgbht4rwefgr3ef.rtf: Unchanged skipping
2019-07-08 22:44:29 DEBUG : sdefr4wefgre.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : xcfedqwe.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : 6/: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : 7/: Size and modification time the same (differ by 0s, within tolerance 100ns)
2019-07-08 22:44:29 DEBUG : sdefr4wefgre.txt: Unchanged skipping
2019-07-08 22:44:29 DEBUG : xcfedqwe.txt: Unchanged skipping0s
2019-07-08 22:44:29 DEBUG : 6/: Unchanged skipping/s, ETA 0s
2019-07-08 22:44:29 DEBUG : 7/: Unchanged skipping/s, ETA 0s
2019-07-08 22:44:29 INFO : Storage bucket cloud-dev path data/5731798554247168/: Waiting for transfers to finish
2019-07-08 22:44:29 INFO : dwewwd.txt: Copied (new)/s, ETA 0s
Transferred: 18 / 18 Bytes, 100%, 6 Bytes/s, ETA 0s
Errors: 0 / 5, 100% 6/: checking 0/s, -
Checks: 5 / 5, 100% 6/: checking 0/s, -
Transferred: 1 / 1, 100% 6/: checking 0/s, -
Elapsed time: 2.7s dwewwd.txt:100% /18, 0/s, -
2019/07/08 22:44:29 INFO : dwewwd.txt:100% /18, 0/s, -
Transferred: 18 / 18 Bytes, 100%, 6 Bytes/s, ETA 0s, -
Errors: 0
Checks: 5 / 5, 100%
Transferred: 1 / 1, 100%
Elapsed time: 2.8s
2019/07/08 22:44:29 DEBUG : 10 go routines active
2019/07/08 22:44:30 INFO : The success report is available at C:\Users\manis\AppData\Local\rclone\success_07082019224428.csv
2019/07/08 22:44:31 DEBUG : success_07082019224428.csv: Couldn't find file - need to transfer
2019/07/08 22:44:32 INFO : success_07082019224428.csv: Copied (new)
2019/07/08 22:44:32 INFO : The failure report is available at C:\Users\manis\AppData\Local\rclone\failure_07082019224428.csv
2019/07/08 22:44:33 DEBUG : failure_07082019224428.csv: Couldn't find file - need to transfer
2019/07/08 22:44:33 INFO : failure_07082019224428.csv: Copied (new)
2019/07/08 22:44:33 DEBUG : rclone: Version "v1.48.0-DEV" finishing with parameters ["C:\Users\manis\rclone.exe" "copy" "C:\Users\manis\Desktop\0\" "--files-from" "C:\Users\manis\rclone_report.txt" "--config" ".\rclone.config" "-vvP"]