Move to B2 creates empty file versions

Help and Support
1

What is the problem you are having with rclone?

I am regularily moving image data to a B2 bucket, and it usually works flawlessy. However, it happens that I end up with some files having two or more versions, most of them with size zero. This only if in B2 versioning is enabled. If I disable versioning, I often end up with no file/empty file on B2, and also a deleted file locally.

In this run, for example, some files ended up being double or more (albeit with the doubles being zero size), and some others are not even present:

2

What is your rclone version (output from rclone version)

rclone v1.55.0
os/type: windows

3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

4

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "RAW Images" "B2:/skygems-userframes" -P --no-traverse --transfers=4 --min-size 5M --bwlimit 2M --suffix .bak --exclude _gsdata_/* --log-file=c:\AstroData\rclone.log

The rclone config contents with secrets removed.

[B2]
type = b2
account = 00xxxxxxxxxxxxxxxxxxxxx
key = K00xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
5

A log from the command with the -vv flag

2021/04/09 21:52:14 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-182110-Blue-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 21:52:14 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-182110-Blue-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "2f403fad657a54b0f40af14d733ea1bdbf6a5123"
2021/04/09 21:53:21 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-182253-Blue-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 21:53:21 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-182253-Blue-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "c383827871aae8d4702c98a5670d80e6a7cf27f3"
2021/04/09 21:54:35 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-181924-Blue-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 21:54:35 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-181924-Blue-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "19b86de7dfd0fb1853b6b2be477aa8f1839d9077"
2021/04/09 22:30:04 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-182438-Blue-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 22:30:04 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Blue-1-5-iDKNamibia-20210409-182438-Blue-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "2037f0ae16c4d9f1cdd703a1398f8c144a49e42b"
2021/04/09 23:09:00 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Ha7-1-5-iDKNamibia-20210409-183413-Ha7-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 23:09:00 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Ha7-1-5-iDKNamibia-20210409-183413-Ha7-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "09b1294e9ea5e3144460e93a8b182243b6a8f9cf"
2021/04/09 23:11:48 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Ha7-1-5-iDKNamibia-20210409-183557-Ha7-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 23:11:48 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Ha7-1-5-iDKNamibia-20210409-183557-Ha7-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "56dc3a9f6488e73345f8e2286c6d0216172929cd"
2021/04/09 23:48:11 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Luminance-1-5-iDKNamibia-20210409-180739-Luminance-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 23:48:11 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Luminance-1-5-iDKNamibia-20210409-180739-Luminance-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "3e5ee75587847973c7a1556bcaf95ade9ba8cf5e"
2021/04/09 23:51:22 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Luminance-1-5-iDKNamibia-20210409-180924-Luminance-060-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/09 23:51:22 ERROR : 115/iDK Namibia/2548-0-NGC 2610/2548-0-NGC 2610-Luminance-1-5-iDKNamibia-20210409-180924-Luminance-060-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "5858cec92791249feda89ba6e4f76139abd3eae3"
2021/04/10 00:36:55 ERROR : Attempt 1/3 failed with 16 errors and: corrupted on transfer: SHA-1 hash differ "" vs "5858cec92791249feda89ba6e4f76139abd3eae3"
2021/04/10 03:10:41 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Blue-1-3-iDKNamibia-20210409-202036-Blue-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/10 03:10:41 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Blue-1-3-iDKNamibia-20210409-202820-Blue-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/10 04:31:15 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-10-10-iDKNamibia-20210409-203850-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 04:31:15 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-10-10-iDKNamibia-20210409-203850-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "340a47fc2e9b11da09a18fc04e4f6efd05862d58"
2021/04/10 04:32:37 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-200447-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 04:32:37 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-200447-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "3cadb9e2e32766c6b8983394c016490af0ba21f0"
2021/04/10 04:32:57 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-195602-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 04:32:57 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-195602-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "357f3ec9561d6b22c877230cde307ee14b274890"
2021/04/10 04:51:49 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-201337-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 04:51:49 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-201337-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "ce2e182fd214644f04a2152b9091d6077a0c24eb"
2021/04/10 05:09:01 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-1-3-iDKNamibia-20210409-193027-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 05:09:01 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-1-3-iDKNamibia-20210409-193027-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "b72825f28a3401f7a0a75d3c9612ff8737945c5f"
2021/04/10 05:12:03 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-1-3-iDKNamibia-20210409-193648-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 05:12:03 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-1-3-iDKNamibia-20210409-193648-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "30512e267294f55fc23bb7d8de2818a8c2753039"
2021/04/10 05:52:18 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-1-3-iDKNamibia-20210409-194334-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 05:52:18 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-1-3-iDKNamibia-20210409-194334-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "72a5d84212d0a3657cfb2fe7afd4daa883a0e808"
2021/04/10 05:54:05 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-7-8-iDKNamibia-20210409-192303-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 05:54:05 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-7-8-iDKNamibia-20210409-192303-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "7a6971561a1844c483c07cc3e24358447961670e"
2021/04/10 05:54:06 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-7-8-iDKNamibia-20210409-191557-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 05:54:06 ERROR : 218/iDK Namibia/2614-0-m 104/2614-0-m 104-Red-7-8-iDKNamibia-20210409-191557-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "74a91604a8e43541dfa9fbc9fa82352de1d52604"
2021/04/10 06:21:46 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Blue-1-3-iDKNamibia-20210409-202036-Blue-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/10 06:21:46 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Blue-1-3-iDKNamibia-20210409-202820-Blue-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/10 06:21:46 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Luminance-10-10-iDKNamibia-20210409-203850-Luminance-300-1.fts: Failed to copy: failed to open source object: The system cannot find the file specified.
2021/04/10 07:02:09 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-200447-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 07:02:09 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-200447-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "ae915db980a2cbb4b4c747256cba8f3c3ceb15d7"
2021/04/10 07:02:56 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-195602-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 07:02:56 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-195602-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "df7af58bff016a304feb21fdb4125bb2392c4bb3"
2021/04/10 07:04:02 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-201337-Luminance-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 07:04:02 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Luminance-7-9-iDKNamibia-20210409-201337-Luminance-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "070580a071f3922168177ddb87e719e0f2e3e8f4"
2021/04/10 07:05:03 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Red-1-3-iDKNamibia-20210409-193027-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 07:05:03 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Red-1-3-iDKNamibia-20210409-193027-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "c8ad41c91ddaa6c1f750f33060ca27b9dd137807"
2021/04/10 07:40:55 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Red-7-8-iDKNamibia-20210409-191557-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 07:40:55 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Red-7-8-iDKNamibia-20210409-191557-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "a8b2f86e10eeb7f4c72fc00abcd47270a05f8488"
2021/04/10 07:42:01 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Red-7-8-iDKNamibia-20210409-192303-Red-300-1.fts: Failed to calculate src hash: hash: failed to open: The system cannot find the file specified.
2021/04/10 07:42:01 ERROR : 218/iDK Namibia/2614-0-m 104/RAW/RAW-2614-0-m 104-Red-7-8-iDKNamibia-20210409-192303-Red-300-1.fts: corrupted on transfer: SHA-1 hash differ "" vs "65e3d430e47662f4a28f146a774bc476abf85ca5"
2021/04/10 07:42:02 ERROR : Attempt 2/3 failed with 35 errors and: corrupted on transfer: SHA-1 hash differ "" vs "65e3d430e47662f4a28f146a774bc476abf85ca5"
6

If you wonder why I didn't made it in one post: I always got the error that I am limited in posting two links. No idea what kind of link I was creating, well :wink:

Also, a screenshot here fo reference on how it looks in the B2 Backend:

Screenshot_3
Screenshot_3804×779 104 KB

7

Do you know what is causing these errors? Why is the file not found locally? Is a symlink or junction point or did it get removed during the transfer?

What is special about those files?

Can you link a complete log with -vv --retries 1

Could there be duplicated file names in the source?

8

Hi nick,

thanks for your reply.
The files are astronomical image datasets, called FITS. Size ranges from 8 to 120 MB each, but I see the problem being unrelated to the size. It looks like we end up with several versions of the same file being uploaded for some reason.

There are definitively no duplicated names in the source, as the filename is encoded with Date (YYYYmmdd) and time (HHmmss) of imaging.

I'll start a run with the verbose log and send it over once done, thanks again

9

Okay, so just started and already getting odd EOF responses...
(I removed some lines to get within the post limitation)

2021/04/11 08:46:17 DEBUG : Using config file from "C:\\Users\\Lukas\\.config\\rclone\\rclone.conf"
2021/04/11 08:46:17 INFO  : Starting bandwidth limiter at 2MBytes/s
2021/04/11 08:46:17 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "copy" "RAW Images" "B2:/skygems-userframes" "-P" "--no-traverse" "-vv" "--retries" "1" "--transfers=6" "--min-size" "5M" "--bwlimit" "2M" "--suffix" ".bak" "--exclude" "_gsdata_/*" "--log-file=c:\\AstroData\\rclone.log"]
2021/04/11 08:46:17 DEBUG : Creating backend with remote "RAW Images"
2021/04/11 08:46:17 DEBUG : fs cache: renaming cache item "RAW Images" to be canonical "//?/c:/Astronomy/RAW Images"
2021/04/11 08:46:17 DEBUG : Creating backend with remote "B2:/skygems-userframes"
2021/04/11 08:46:19 DEBUG : fs cache: renaming cache item "B2:/skygems-userframes" to be canonical "B2:skygems-userframes"
2021/04/11 08:46:20 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:20 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:20 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:20 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:20 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:20 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : 27/iDK Namibia/2552-0-M104/2552-0-M104-Red-1-3-iDKNamibia-20210410-195455-Red-300-1.fts: Size and modification time the same (differ by -929.3µs, within tolerance 1ms)
2021/04/11 08:46:21 DEBUG : 27/iDK Namibia/2552-0-M104/2552-0-M104-Red-1-3-iDKNamibia-20210410-195455-Red-300-1.fts: Unchanged skipping
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : 27/iDK Namibia/2552-0-M104/RAW/RAW-2552-0-M104-Red-1-3-iDKNamibia-20210410-195455-Red-300-1.fts: Size and modification time the same (differ by -100.9µs, within tolerance 1ms)
2021/04/11 08:46:21 DEBUG : 27/iDK Namibia/2552-0-M104/RAW/RAW-2552-0-M104-Red-1-3-iDKNamibia-20210410-195455-Red-300-1.fts: Unchanged skipping
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:21 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-025847-Blue-300-1.fts: Size and modification time the same (differ by -672.4µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-031930-Green-300-1.fts: Size and modification time the same (differ by -536.6µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-031930-Green-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-023722-Luminance-300-1.fts: Size and modification time the same (differ by -29.8µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-023722-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-022955-Luminance-300-1.fts: Size and modification time the same (differ by -754.7µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-022955-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-021549-Luminance-300-1.fts: Size and modification time the same (differ by -816.1µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-021549-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-020918-Luminance-300-1.fts: Size and modification time the same (differ by -952.8µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-020918-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-025847-Blue-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-033701-Green-300-1.fts: Size and modification time the same (differ by -27.5µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-033701-Green-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-024355-Luminance-300-1.fts: Size and modification time the same (differ by -435.5µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-024355-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-030610-Blue-300-1.fts: Size and modification time the same (differ by -642.8µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-030610-Blue-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-032834-Green-300-1.fts: Size and modification time the same (differ by -965.3µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-032834-Green-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-022251-Luminance-300-1.fts: Size and modification time the same (differ by -863.1µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-022251-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:22 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:22 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:22 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-025847-Blue-300-1.fts: Size and modification time the same (differ by -240.8µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-031930-Green-300-1.fts: Size and modification time the same (differ by -206.6µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-025847-Blue-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-022955-Luminance-300-1.fts: Size and modification time the same (differ by -71.4µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-022955-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-020918-Luminance-300-1.fts: Size and modification time the same (differ by -538.3µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-020918-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-021549-Luminance-300-1.fts: Size and modification time the same (differ by -524.5µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-021549-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-4-6-iDKNamibia-20210410-022251-Luminance-300-1.fts: Size and modification time the same (differ by -928µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Green-1-3-iDKNamibia-20210410-032834-Green-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-023722-Luminance-300-1.fts: Size and modification time the same (differ by -490.7µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-023722-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-024355-Luminance-300-1.fts: Size and modification time the same (differ by -48.3µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Luminance-1-3-iDKNamibia-20210410-024355-Luminance-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-030610-Blue-300-1.fts: Size and modification time the same (differ by -464.4µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-030610-Blue-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-025134-Blue-300-1.fts: Size and modification time the same (differ by -452.9µs, within tolerance 1ms)
2021/04/11 08:46:22 DEBUG : 51/iDK Namibia/2629-0-eso 9-10/RAW/RAW-2629-0-eso 9-10-Blue-1-3-iDKNamibia-20210410-025134-Blue-300-1.fts: Unchanged skipping
2021/04/11 08:46:22 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:22 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:22 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:23 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:24 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:25 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:25 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:25 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:25 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:25 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:26 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:27 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : Couldn't decode error response: EOF
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-201604-Red-060-1.fts: Size and modification time the same (differ by -982.9µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-201604-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-202705-Red-060-1.fts: Size and modification time the same (differ by -534.1µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-203652-Red-060-1.fts: Size and modification time the same (differ by -327.9µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-203652-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-203808-Red-060-1.fts: Size and modification time the same (differ by -481.6µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-203808-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205216-Red-060-1.fts: Size and modification time the same (differ by -344.6µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205216-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205324-Red-060-1.fts: Size and modification time the same (differ by -406.2µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205324-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205434-Red-060-1.fts: Size and modification time the same (differ by -643.7µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205434-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205542-Red-060-1.fts: Size and modification time the same (differ by -75.9µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205542-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205705-Red-060-1.fts: Size and modification time the same (differ by -169.5µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204144-Red-060-1.fts: Size and modification time the same (differ by -663.9µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204144-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-202814-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205921-Red-060-1.fts: Size and modification time the same (differ by -271.4µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205921-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204401-Red-060-1.fts: Size and modification time the same (differ by -448.4µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204401-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204510-Red-060-1.fts: Size and modification time the same (differ by -418.9µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204510-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210311-Red-060-1.fts: Size and modification time the same (differ by -337.9µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204948-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205056-Red-060-1.fts: Size and modification time the same (differ by -40.5µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205056-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210530-Red-060-1.fts: Size and modification time the same (differ by -130.7µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205705-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210652-Red-060-1.fts: Size and modification time the same (differ by -476.3µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205813-Red-060-1.fts: Size and modification time the same (differ by -804.5µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-205813-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210809-Red-060-1.fts: Size and modification time the same (differ by -461.6µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204253-Red-060-1.fts: Size and modification time the same (differ by -787.5µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-204253-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210917-Red-060-1.fts: Size and modification time the same (differ by -70.6µs, within tolerance 1ms)
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210420-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210530-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210652-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210809-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : 120/iDK Namibia/2630-1-NGTS-2b/2630-1-NGTS-2b-1-305-iDKNamibia-20210410-210311-Red-060-1.fts: Unchanged skipping
2021/04/11 08:46:28 DEBUG : B2 bucket skygems-userframes: Waiting for transfers to finish
10

They do look odd!

Can you run your sync with -vv --dump responses. This will be VERY verbose - all I need from it is an HTTP request and response that caused the EOF error. It will (I'm guessing) be an error message from a proxy or something like that.

11

Hi Nick,

alright, here is the log (please note that the previous log is also there, maybe it helps)
rclone.log (476.6 KB)

12 
2021/04/11 12:46:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/11 12:46:56 DEBUG : HTTP RESPONSE (req 0xc0001bd700)
2021/04/11 12:46:56 DEBUG : HTTP/1.1 200 
Content-Length: 680
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Sun, 11 Apr 2021 10:46:56 GMT

{
  "accountId": "807249808cd6",
  "action": "upload",
  "bucketId": "f85087b284c9a820785c0d16",
  "contentLength": 8398080,
  "contentMd5": "50ae46df2dcf0073d5323ac4ba410386",
  "contentSha1": "4cc49450e106634e4a22bcc77d0cda9a49ce590f",
  "contentType": "application/octet-stream",
  "fileId": "4_zf85087b284c9a820785c0d16_f1196f9aebe84c6a8_d20210411_m103821_c003_v0312006_t0022",
  "fileInfo": {
    "src_last_modified_millis": "1618111376006"
  },
  "fileName": "267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-032108-Luminance-090-2.fts",
  "serverSideEncryption": {
    "algorithm": null,
    "mode": null
  },
  "uploadTimestamp": 1618137501000
}
2021/04/11 12:46:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/11 12:46:56 DEBUG : 267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-032108-Luminance-090-2.fts: SHA-1 = 4cc49450e106634e4a22bcc77d0cda9a49ce590f OK
2021/04/11 12:46:56 INFO  : 267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-032108-Luminance-090-2.fts: Copied (new)
2021/04/11 12:46:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/11 12:46:56 DEBUG : HTTP REQUEST (req 0xc0006c6400)
2021/04/11 12:46:56 DEBUG : POST /b2api/v1/b2_upload_file/f85087b284c9a820785c0d16/c003_v0312006_t0022 HTTP/1.1
Host: pod-031-2006-19.backblaze.com
User-Agent: rclone/v1.55.0
Content-Length: 8398080
Authorization: XXXX
Content-Type: application/octet-stream
X-Bz-Content-Sha1: 023b7124733b0ba700e2c4890c177bd3aa60fd07
X-Bz-File-Name: 267/iDK%20Namibia/2666-1-2016%20WQ48/2666-1-2016%20WQ48-1-24-iDKNamibia-20210411-032256-Luminance-090-2.fts
X-Bz-Info-Src_last_modified_millis: 1618111482760
Accept-Encoding: gzip

2021/04/11 12:46:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/11 12:47:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/11 12:47:12 DEBUG : HTTP RESPONSE (req 0xc0008d5500)
2021/04/11 12:47:12 DEBUG : HTTP/1.1 200 
Content-Length: 680
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Sun, 11 Apr 2021 10:47:12 GMT

{
  "accountId": "807249808cd6",
  "action": "upload",
  "bucketId": "f85087b284c9a820785c0d16",
  "contentLength": 8398080,
  "contentMd5": "79f215a320ded746dafe8ee6fa4f54cb",
  "contentSha1": "a3ed62693ea2b70dc4bbedec113db9eba11c3717",
  "contentType": "application/octet-stream",
  "fileId": "4_zf85087b284c9a820785c0d16_f108445156a647a3a_d20210411_m103823_c003_v0312005_t0023",
  "fileInfo": {
    "src_last_modified_millis": "1618111166115"
  },
  "fileName": "267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-031739-Luminance-090-2.fts",
  "serverSideEncryption": {
    "algorithm": null,
    "mode": null
  },
  "uploadTimestamp": 1618137503000
}
2021/04/11 12:47:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/11 12:47:12 DEBUG : 267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-031739-Luminance-090-2.fts: SHA-1 = a3ed62693ea2b70dc4bbedec113db9eba11c3717 OK
2021/04/11 12:47:12 INFO  : 267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-031739-Luminance-090-2.fts: Copied (new)
2021/04/11 12:47:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/11 12:47:12 DEBUG : HTTP REQUEST (req 0xc0006c6900)
2021/04/11 12:47:12 DEBUG : POST /b2api/v1/b2_upload_file/f85087b284c9a820785c0d16/c003_v0312005_t0023 HTTP/1.1
Host: pod-031-2005-08.backblaze.com
User-Agent: rclone/v1.55.0
Content-Length: 8398080
Authorization: XXXX
Content-Type: application/octet-stream
X-Bz-Content-Sha1: 3bbb1c6fc2d8b36ff26ee944f9ebc265bc8ecbd6
X-Bz-File-Name: 267/iDK%20Namibia/2666-1-2016%20WQ48/2666-1-2016%20WQ48-1-24-iDKNamibia-20210411-032442-Luminance-090-2.fts
X-Bz-Info-Src_last_modified_millis: 1618111594929
Accept-Encoding: gzip

2021/04/11 12:47:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/11 12:47:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/11 12:47:37 DEBUG : HTTP RESPONSE (req 0xc0006c6300)
2021/04/11 12:47:37 DEBUG : HTTP/1.1 200 
Content-Length: 680
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Sun, 11 Apr 2021 10:47:37 GMT

{
  "accountId": "807249808cd6",
  "action": "upload",
  "bucketId": "f85087b284c9a820785c0d16",
  "contentLength": 8398080,
  "contentMd5": "7c78ca91ed6b511a9962be883b7aef7f",
  "contentSha1": "293dffd19855ec25e27d0fe4c19301ec2881ea0a",
  "contentType": "application/octet-stream",
  "fileId": "4_zf85087b284c9a820785c0d16_f110bfc4630ea9acd_d20210411_m103821_c003_v0312006_t0016",
  "fileInfo": {
    "src_last_modified_millis": "1618111268877"
  },
  "fileName": "267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-031926-Luminance-090-2.fts",
  "serverSideEncryption": {
    "algorithm": null,
    "mode": null
  },
  "uploadTimestamp": 1618137501000
}
2021/04/11 12:47:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/11 12:47:37 DEBUG : 267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-031926-Luminance-090-2.fts: SHA-1 = 293dffd19855ec25e27d0fe4c19301ec2881ea0a OK
2021/04/11 12:47:37 INFO  : 267/iDK Namibia/2666-1-2016 WQ48/2666-1-2016 WQ48-1-24-iDKNamibia-20210411-031926-Luminance-090-2.fts: Copied (new)
2021/04/11 12:47:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/11 12:47:37 DEBUG : HTTP REQUEST (req 0xc0006c6300)
2021/04/11 12:47:37 DEBUG : POST /b2api/v1/b2_upload_file/f85087b284c9a820785c0d16/c003_v0312006_t0016 HTTP/1.1
Host: pod-031-2006-10.backblaze.com
User-Agent: rclone/v1.55.0
Content-Length: 8398080
Authorization: XXXX
Content-Type: application/octet-stream
X-Bz-Content-Sha1: 65c7251b79270e621ed504525bc8e999c6edfb21
X-Bz-File-Name: 267/iDK%20Namibia/2666-1-2016%20WQ48/2666-1-2016%20WQ48-1-24-iDKNamibia-20210411-032643-Luminance-090-2.fts
X-Bz-Info-Src_last_modified_millis: 1618111711095
Accept-Encoding: gzip

2021/04/11 12:47:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
13

I think these are probably a red herring looking at the logs.

I've fixed them here

v1.56.0-beta.5386.ee87bf19c.fix-b2-error-handling on branch fix-b2-error-handling (uploaded in 15-30 mins)

Can you get me a log with -vv of the problem ocurring with that binary? It will at least have less noise in!

If you could get a listing with rclone lsl --b2-versions b2:bucket showing the problem that would be very useful too.

However looking at the logs gave me an idea - I think the empty versions might be caused by the use of --no-traverse. Can you try a sync without --no-traverse and see if it causes the empty versions.

The other possibility is using --suffix interacting strangely with

14

Hi Nick,

okay, will try and get back to you shortly.
I added --suffix only later, so the issue is not coming from that. Also, I first used move but now switched to copy to be safe :slight_smile:

--no-traverse has always been used, so it might be that.

The output of rclone lsl --b2-versions B2:bucket is in this log file (a little truncated to be able to upload it): (please note that I used the current version for this):
lsl.log (3.3 MB)

And here the first output using the beta version, although with the same commandline (with --no-traverse):
rclone.log (414.6 KB)

15

Thanks. Do you have specific names of files that you'd like me to investigate in there?

16

Hi Nick,

well, not really. All files with a version are troublesome :slight_smile:
Example:
48/iDK Namibia/2641-1-#141495_HiDK01/RAW/RAW-2641-1-#141495_HiDK01-1-30-iDKNamibia-20210409-225416-Luminance-060-1-v2021-04-10-160458-000.fts

I have to add that, since we swapped to the latest beta and added --retries 1 it seems to not show this problem. But I have to investigate more

EDIT: Okay so it looks like it is truly working now with these settings... I will investigate more and get back in a few.

17

I don't think there is anything in the new beta which affects the B2 backend so I'd guess the fix is the retries flag which is odd.

Are the files being written to at the same time rclone is running?

18

Hi Nick,
no, the files are usually being written at once and then transferred.
The original issue was that, with B2 having no versioning enabled, you could potentially end up with no file on the bucket, and still a deleted file locally (when doing the rclone move command).
It looks to me that it is connected to the handling of aborted/failed uploads, where a part had already been uploaded and it had to retry. Maybe, setting retries flag to 1, this is avoided?

Anyway, many thanks for your help. I am producing new files to be transferred continuously so i will keep an eye on how it goes. I will at some point need to do a move instead of copy to free up some local space though :slight_smile:

19

I see that file once in the lsl dump with a version.

That means it was uploaded then deleted at some point - does that make sense?

That should certainly never happen. An rclone move is an rclone copy which checks to see the destination exists properly after the transfer, followed by an rclone delete..

Retries controls the whole sync retry, so if there were failures in the first try, it tries everything again.

If you have another problem can you send a log with -vv and a note of which files you are having the problem with? I still don't think I understand what is going on!

20

Me neither :sweat_smile:
I am collecting some more logs, but the problem doesn't show up anymore, so difficult to say, honestly.

I will keep an eye on it and report back in a few days.

1 Like