Okay, after transferring some more files, it seems to not create any additional unwanted versions anymore 
1 Like
Well, looks like the thing is not yet done:
I found out that even today, with the updated exe and settings, sometimes files simply disappear.
Example:
2021/04/14 01:58:30 DEBUG : 290/iDK Namibia/2693-0-NGC3576/RAW/RAW-2693-0-NGC3576-O3-4-4-iDKNamibia-20210412-232026-O3-1200-1.fts: SHA-1 = 0323cd833bfddb2912c2fcf56c516806ba55a9b9 OK
2021/04/14 01:58:30 INFO : 290/iDK Namibia/2693-0-NGC3576/RAW/RAW-2693-0-NGC3576-O3-4-4-iDKNamibia-20210412-232026-O3-1200-1.fts: Copied (new)
2021/04/14 01:59:18 DEBUG : 290/iDK Namibia/2693-0-NGC3576/RAW/RAW-2693-0-NGC3576-O3-3-3-iDKNamibia-20210412-192918-O3-1200-1.fts: SHA-1 = 639b53905e71bc2c444a13a32c65021a55df359a OK
2021/04/14 01:59:18 INFO : 290/iDK Namibia/2693-0-NGC3576/RAW/RAW-2693-0-NGC3576-O3-3-3-iDKNamibia-20210412-192918-O3-1200-1.fts: Copied (new)
2021/04/14 01:59:18 ERROR : 27/iDK Namibia/2552-0-M104/RAW/RAW-2552-0-M104-Blue-1-3-iDKNamibia-20210412-222951-Blue-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/14 01:59:18 ERROR : 27/iDK Namibia/2552-0-M104/RAW/RAW-2552-0-M104-Blue-1-3-iDKNamibia-20210412-223611-Blue-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
Note the failed to copy errors. Logic would tell me that something deleted the file before rclone was able to access and upload it. Okay, could be that those RAW files are temporary ones, osmetimes. But the following was definitively a file not being fiddled with by anybody:
2021/04/13 21:57:49 ERROR : 27/iDK Namibia/2552-0-M104/2552-0-M104-Luminance-4-6-iDKNamibia-20210412-203707-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/13 21:57:49 ERROR : 27/iDK Namibia/2552-0-M104/2552-0-M104-Luminance-4-6-iDKNamibia-20210412-203707-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "24cca9f5ae9e50af1c833c708f5d5b2ecef07dfc"
2021/04/13 21:57:49 INFO : 27/iDK Namibia/2552-0-M104/2552-0-M104-Luminance-4-6-iDKNamibia-20210412-203707-Luminance-300-1.fts: Removing failed copy
The file was definitively there and has disappeared between rclone starting and the mentioned error. Could it be that rclone, erroneously of course, removes the local file?
Okay, the problem seems to persist even with a rclone copy, basically deleting the local file when a transfer is not succeeding:
2021/04/21 08:02:05 DEBUG : 143/iDK Namibia/2740-0-098P/2740-0-098P-iDKNamibia-20210421-041652-Red-060-1.fts: SHA-1 = 8bb2e1ad572e29c5401f373ec7fa452fe5d63b7b OK
2021/04/21 08:02:05 INFO : 143/iDK Namibia/2740-0-098P/2740-0-098P-iDKNamibia-20210421-041652-Red-060-1.fts: Copied (new)
2021/04/21 08:40:54 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-013359-S2-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/21 08:40:54 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-013359-S2-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "151a190006a4b01e13374f67d0d8e05192c94bc0"
2021/04/21 08:40:54 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-013359-S2-300-1.fts: Removing failed copy
2021/04/21 08:42:00 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-014728-O3-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/21 08:42:00 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-014728-O3-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "fbfb5a7f47b6278b999a2c4d069cd4b7842779f9"
2021/04/21 08:42:00 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-014728-O3-300-1.fts: Removing failed copy
2021/04/21 08:42:02 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-015329-O3-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/21 08:42:02 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-015329-O3-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "ecd0c18a7a1c4d3f5d7e53ec41f4eb4fbe7c4966"
2021/04/21 08:42:02 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-015329-O3-300-1.fts: Removing failed copy
2021/04/21 08:42:35 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-014124-O3-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/21 08:42:35 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-014124-O3-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "243a2ecde1514a260ead638286ebcb38932f2f7a"
2021/04/21 08:42:35 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-014124-O3-300-1.fts: Removing failed copy
2021/04/21 09:19:09 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-023850-Ha7-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/21 09:19:09 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-023850-Ha7-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "cd5fe839ec2d06c3259808c99017cdfb34bae21d"
2021/04/21 09:19:09 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-023850-Ha7-300-1.fts: Removing failed copy
2021/04/21 09:23:42 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-024537-Ha7-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/21 09:23:42 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-024537-Ha7-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "2d8a04d523c613cc62b96946badb1bb15ada9eef"
2021/04/21 09:23:42 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-024537-Ha7-300-1.fts: Removing failed copy
2021/04/21 10:01:46 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-025146-Ha7-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/21 10:01:46 ERROR : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-025146-Ha7-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "2bcdf6b56efef46e4d3ec29447605cf4d361e261"
2021/04/21 10:01:46 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-025146-Ha7-300-1.fts: Removing failed copy
2021/04/21 10:02:35 DEBUG : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-030552-S2-300-1.fts: SHA-1 = edd43d6e7e43ca99554956a89503444be84e65d6 OK
2021/04/21 10:02:35 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-030552-S2-300-1.fts: Copied (new)
2021/04/21 10:03:29 DEBUG : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-031154-S2-300-1.fts: SHA-1 = 30b9d837107f8619103fca49dc75df6ecb71179d OK
2021/04/21 10:03:29 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-031154-S2-300-1.fts: Copied (new)
2021/04/21 10:04:14 DEBUG : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-025903-S2-300-1.fts: SHA-1 = 6767da2a186e4a82e0719de84cea0c40ccda7c5d OK
2021/04/21 10:04:14 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-025903-S2-300-1.fts: Copied (new)
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-002040-S2-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-002709-S2-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-003307-S2-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-004030-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-004700-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-005327-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-010041-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-010700-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-011415-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-012117-S2-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-012731-S2-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:04:14 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-013359-S2-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:39:39 DEBUG : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-031851-Ha7-300-1.fts: SHA-1 = e0a7058d8d46cea66c36cf2dbd7cca1d89254300 OK
2021/04/21 10:39:39 INFO : 193/iDK Namibia/2720-0-NGC6357/2720-0-NGC6357-iDKNamibia-20210421-031851-Ha7-300-1.fts: Copied (new)
2021/04/21 10:39:39 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-014728-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:39:39 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-015329-O3-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/21 10:39:39 ERROR : 193/iDK Namibia/2720-0-NGC6357/RAW/RAW-2720-0-NGC6357-iDKNamibia-20210421-023850-Ha7-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
Let's take for example the 2720-0-NGC6357-iDKNamibia-20210421-014728-O3-300-1.fts file.
rclone starts transferring it correctly, and at some point is trying to compare the CRC of the local vs remote file. For whatever reason the local file disappeared, and it also deletes the remote one. However, I have no idea as to why it would disappear locally at all. There are no other programs or scripts running locally.
I am getting these kind of errors on three different systems, and they are reproducible particularily with big files and slow connection (10-20 kb/sec).
Is there a way of logging even more details to grind this down further?
Thanks
Lukas
Yes it looks like the file disappeared after the upload started but before it finished.
Rclone does that when it detects a data integrity problem. It isn't ideal in this case, I agree.
I doubt it is rclone deleting the local object - could it be the program which creates those images has renamed it or something like that?
Okay I think I found out what was happening. There was another instance still transferring files to another Bucket, and the two rlone instances basically deleted the completed files before the other instance could transfe rit. Very stupid of me 
1 Like
I'm glad we got to the bottom of it
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.