Large file upload to backblaze failed?

JJB2020 · May 5, 2020, 8:06am

What is the problem you are having with rclone?

I have a ZFS snapshot that I want to upload to a backblaze B2 bucket. The upload is failing. The total dataset size is around 1.8TB. I have tried this method three times and it has failed every time. Full debug output at the bottom of this message.

zfs send -Rv zfsdata/justin@b2origin | gzip -9c | gpg -e -r <key id here> | rclone rcat b2:zfsdata/zfsdata-justin-b2origin.gz.gpg

The "zfs send" part of this will send one continuous data stream, as if it was one very large file. Am I right in thinking that the "rclone rcat" command will split this up into a multi-part upload?

On all attempts this eventually faled with this output in the terminal window ...

gpg: [stdout]: write error: Broken pipe
gpg: DBG: deflate: iobuf_write failed
2020/05/04 07:07:15 Failed to rcat with 2 errors: last error was: Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f210617c5b8edfa97_d20200503_m081333_c002_v0001132_t0009/0040": tls: use of closed connection
gpg: [stdout]: write error: Broken pipe
gpg: filter_flush failed on close: Broken pipe

The second upload failure occured after 128GB had already gone up over 20 hours or so.

What is your rclone version (output from `rclone version`)

rclone v1.51.0

os/arch: linux/amd64
go version: go1.14

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Arch Linux 64bit

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

zfs send -Rv zfsdata@b2origin | gzip -9c | gpg -e -r <key id here> | rclone rcat b2:zfsdata/zfsdata-b2origin.gz.gpg

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

Details to follow in next post ...

JJB2020 · May 5, 2020, 8:08am

[justin@IXTREME ~]$ rclone version
rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.14

[justin@IXTREME ~]$ zfs send -R zfsdata/justin@b2origin | gzip -9c | gpg -e -r 71DBCF99DCB9A5988BD179D2923C3704C7B8371F | rclone -vv rcat b2:zfsdata/zfsdata-justin-b2origin.gz.gpg
2020/05/04 15:27:26 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "-vv" "rcat" "b2:zfsdata/zfsdata-justin-b2origin.gz.gpg"]
2020/05/04 15:27:26 DEBUG : Using config file from "/home/justin/.config/rclone/rclone.conf"
2020/05/04 15:27:38 DEBUG : zfsdata-justin-b2origin.gz.gpg: File is big enough for chunked streaming
2020/05/04 15:27:38 DEBUG : zfsdata-justin-b2origin.gz.gpg: Streaming upload with --b2-chunk-size 96M allows uploads of up to 937.500G and will fail only when that limit is reached.
2020/05/04 15:27:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Starting streaming of large file (id "4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023")
2020/05/04 15:27:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 1 length 100663296
2020/05/04 15:27:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 2 length 100663296
2020/05/04 15:28:03 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 3 length 100663296
2020/05/04 15:28:23 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 4 length 100663296
2020/05/04 15:30:38 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 1
2020/05/04 15:30:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 5 length 100663296
2020/05/04 15:31:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 2
2020/05/04 15:31:25 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 6 length 100663296
2020/05/04 15:31:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 3
2020/05/04 15:31:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 4
2020/05/04 15:32:23 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 7 length 100663296
2020/05/04 15:33:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 8 length 100663296
2020/05/04 15:33:43 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 5
2020/05/04 15:34:05 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 6
2020/05/04 15:34:27 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 9 length 100663296
2020/05/04 15:34:58 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 10 length 100663296

<snip>    

2020/05/04 16:45:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 85 length 100663296
2020/05/04 16:46:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 82
2020/05/04 16:46:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 86 length 100663296
2020/05/04 16:46:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 86 (retry=true): Post "https://pod-000-1132-11.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0046": write tcp 192.168.1.2:58924->206.190.214.202:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-11.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0046", Err:(*net.OpError)(0xc000099220)}
2020/05/04 16:46:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-11.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0046": write tcp 192.168.1.2:58924->206.190.214.202:443: use of closed network connection
2020/05/04 16:46:20 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-11.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0046": write tcp 192.168.1.2:58924->206.190.214.202:443: use of closed network connection)
2020/05/04 16:46:20 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 16:46:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 86 length 100663296
2020/05/04 16:46:21 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 16:46:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 83
2020/05/04 16:46:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 87 length 100663296
2020/05/04 16:47:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 84
2020/05/04 16:47:54 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 88 length 100663296
    
<snip>

2020/05/04 16:58:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 99 length 100663296
2020/05/04 16:59:11 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 96
2020/05/04 16:59:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 100 length 100663296
2020/05/04 16:59:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 100 (retry=true): Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005": EOF: &url.Error{Op:"Post", URL:"https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005", Err:(*errors.errorString)(0xc00007c040)}
2020/05/04 16:59:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005": EOF
2020/05/04 16:59:16 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005": EOF)
2020/05/04 16:59:16 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 16:59:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 100 length 100663296
2020/05/04 16:59:17 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 17:00:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 97
2020/05/04 17:00:51 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 101 length 100663296
2020/05/04 17:01:27 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 98

<snip>

2020/05/04 17:39:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 140 length 100663296
2020/05/04 17:40:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 137
2020/05/04 17:41:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 141 length 100663296
2020/05/04 17:41:03 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 138
2020/05/04 17:41:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 142 length 100663296
2020/05/04 17:41:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 142 (retry=true): Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0059": write tcp 192.168.1.2:40574->206.190.214.206:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0059", Err:(*net.OpError)(0xc000638190)}
2020/05/04 17:41:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0059": write tcp 192.168.1.2:40574->206.190.214.206:443: use of closed network connection
2020/05/04 17:41:08 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0059": write tcp 192.168.1.2:40574->206.190.214.206:443: use of closed network connection)
2020/05/04 17:41:08 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 17:41:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 142 length 100663296
2020/05/04 17:41:09 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 17:41:44 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 139
2020/05/04 17:41:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 143 length 100663296
2020/05/04 17:41:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 143 (retry=true): Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037": write tcp 192.168.1.2:35356->206.190.214.209:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037", Err:(*net.OpError)(0xc000638f50)}
2020/05/04 17:41:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037": write tcp 192.168.1.2:35356->206.190.214.209:443: use of closed network connection
2020/05/04 17:41:49 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037": write tcp 192.168.1.2:35356->206.190.214.209:443: use of closed network connection)
2020/05/04 17:41:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 17:41:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 143 length 100663296
2020/05/04 17:41:50 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 17:43:05 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 140
2020/05/04 17:43:10 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 144 length 100663296
2020/05/04 17:44:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 141
2020/05/04 17:44:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 145 length 100663296
2020/05/04 17:44:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 145 (retry=true): Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0052": write tcp 192.168.1.2:52272->206.190.214.191:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0052", Err:(*net.OpError)(0xc000662000)}
2020/05/04 17:44:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0052": write tcp 192.168.1.2:52272->206.190.214.191:443: use of closed network connection
2020/05/04 17:44:42 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0052": write tcp 192.168.1.2:52272->206.190.214.191:443: use of closed network connection)
2020/05/04 17:44:42 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 17:44:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 145 length 100663296
2020/05/04 17:44:42 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 17:44:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 142
2020/05/04 17:44:54 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 146 length 100663296
2020/05/04 17:45:27 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 143
2020/05/04 17:45:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 147 length 100663296
2020/05/04 17:46:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 144
2020/05/04 17:46:53 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 148 length 100663296
2020/05/04 17:48:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 145
2020/05/04 17:48:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 149 length 100663296
2020/05/04 17:48:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 149 (retry=true): Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:59646->206.190.214.210:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004", Err:(*net.OpError)(0xc0004b2c30)}
2020/05/04 17:48:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:59646->206.190.214.210:443: use of closed network connection
2020/05/04 17:48:26 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:59646->206.190.214.210:443: use of closed network connection)
2020/05/04 17:48:26 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 17:48:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 149 length 100663296
2020/05/04 17:48:27 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 17:48:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 146
2020/05/04 17:48:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 150 length 100663296
2020/05/04 17:49:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 147
2020/05/04 17:49:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 151 length 100663296

<snip>

2020/05/04 18:37:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 203 length 100663296
2020/05/04 18:39:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 200
2020/05/04 18:39:25 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 204 length 100663296
2020/05/04 18:39:25 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 204 (retry=true): Post "https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0047": write tcp 192.168.1.2:50826->206.190.214.196:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0047", Err:(*net.OpError)(0xc000662000)}
2020/05/04 18:39:25 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0047": write tcp 192.168.1.2:50826->206.190.214.196:443: use of closed network connection
2020/05/04 18:39:25 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0047": write tcp 192.168.1.2:50826->206.190.214.196:443: use of closed network connection)
2020/05/04 18:39:25 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 18:39:25 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 204 length 100663296
2020/05/04 18:39:26 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 18:40:51 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 201
2020/05/04 18:40:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 205 length 100663296
2020/05/04 18:40:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 202
2020/05/04 18:41:04 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 206 length 100663296

<snip>

2020/05/04 18:52:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 217 length 100663296
2020/05/04 18:52:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 214
2020/05/04 18:52:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 218 length 100663296
2020/05/04 18:52:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 218 (retry=true): Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0051": EOF: &url.Error{Op:"Post", URL:"https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0051", Err:(*errors.errorString)(0xc00007c040)}
2020/05/04 18:52:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0051": EOF
2020/05/04 18:52:26 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0051": EOF)
2020/05/04 18:52:26 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 18:52:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 218 length 100663296
2020/05/04 18:52:27 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 18:52:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 215
2020/05/04 18:53:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 219 length 100663296

<snip>

JJB2020 · May 5, 2020, 8:10am

continued ...

2020/05/04 19:33:23 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 260 length 100663296
2020/05/04 19:35:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 257
2020/05/04 19:35:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 261 length 100663296
2020/05/04 19:35:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 261 (retry=true): Post "https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0012": EOF: &url.Error{Op:"Post", URL:"https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0012", Err:(*errors.errorString)(0xc00007c040)}
2020/05/04 19:35:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0012": EOF
2020/05/04 19:35:13 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0012": EOF)
2020/05/04 19:35:13 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 19:35:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 261 length 100663296
2020/05/04 19:35:14 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 19:35:23 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 258
2020/05/04 19:35:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 262 length 100663296
2020/05/04 19:35:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 262 (retry=true): Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": EOF: &url.Error{Op:"Post", URL:"https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010", Err:(*errors.errorString)(0xc00007c040)}
2020/05/04 19:35:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": EOF
2020/05/04 19:35:28 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": EOF)
2020/05/04 19:35:28 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 19:35:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 262 length 100663296
2020/05/04 19:35:29 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 19:35:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 259
2020/05/04 19:36:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 263 length 100663296
2020/05/04 19:36:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 263 (retry=true): Post "https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049": write tcp 192.168.1.2:58386->206.190.214.208:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049", Err:(*net.OpError)(0xc0000998b0)}
2020/05/04 19:36:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049": write tcp 192.168.1.2:58386->206.190.214.208:443: use of closed network connection
2020/05/04 19:36:01 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049": write tcp 192.168.1.2:58386->206.190.214.208:443: use of closed network connection)
2020/05/04 19:36:01 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 19:36:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 263 length 100663296
2020/05/04 19:36:01 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 19:37:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 260
2020/05/04 19:37:06 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 264 length 100663296
2020/05/04 19:38:53 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 261
2020/05/04 19:38:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 265 length 100663296
2020/05/04 19:39:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 262
2020/05/04 19:39:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 266 length 100663296
2020/05/04 19:39:41 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 263
2020/05/04 19:39:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 267 length 100663296
2020/05/04 19:40:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 264
2020/05/04 19:40:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 268 length 100663296
2020/05/04 19:42:35 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 265
2020/05/04 19:42:40 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 269 length 100663296
2020/05/04 19:42:51 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 266
2020/05/04 19:42:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 270 length 100663296
2020/05/04 19:43:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 267
2020/05/04 19:43:30 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 271 length 100663296
2020/05/04 19:44:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 268
2020/05/04 19:44:30 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 272 length 100663296
2020/05/04 19:44:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 272 (retry=true): Post "https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039": write tcp 192.168.1.2:51120->206.190.214.196:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039", Err:(*net.OpError)(0xc000099220)}
2020/05/04 19:44:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039": write tcp 192.168.1.2:51120->206.190.214.196:443: use of closed network connection
2020/05/04 19:44:31 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-05.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039": write tcp 192.168.1.2:51120->206.190.214.196:443: use of closed network connection)
2020/05/04 19:44:31 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 19:44:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 272 length 100663296
2020/05/04 19:44:31 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 19:46:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 269
2020/05/04 19:46:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 273 length 100663296
2020/05/04 19:46:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 273 (retry=true): Post "https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0053": write tcp 192.168.1.2:60082->206.190.214.195:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0053", Err:(*net.OpError)(0xc000662000)}
2020/05/04 19:46:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0053": write tcp 192.168.1.2:60082->206.190.214.195:443: use of closed network connection
2020/05/04 19:46:22 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0053": write tcp 192.168.1.2:60082->206.190.214.195:443: use of closed network connection)
2020/05/04 19:46:22 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 19:46:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 273 length 100663296
2020/05/04 19:46:23 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 19:46:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 270
2020/05/04 19:46:38 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 274 length 100663296
2020/05/04 19:47:07 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 271
2020/05/04 19:47:12 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 275 length 100663296
2020/05/04 19:48:12 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 272
2020/05/04 19:48:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 276 length 100663296
2020/05/04 19:50:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 273
2020/05/04 19:50:07 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 277 length 100663296
2020/05/04 19:50:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 274
2020/05/04 19:50:23 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 278 length 100663296
2020/05/04 19:50:50 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 275
2020/05/04 19:50:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 279 length 100663296
2020/05/04 19:51:54 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 276
2020/05/04 19:51:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 280 length 100663296
2020/05/04 19:53:44 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 277
2020/05/04 19:53:49 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 281 length 100663296
2020/05/04 19:54:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 278
2020/05/04 19:54:05 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 282 length 100663296
2020/05/04 19:54:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 279
2020/05/04 19:54:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 283 length 100663296
2020/05/04 19:55:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 280
2020/05/04 19:55:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 284 length 100663296
2020/05/04 19:57:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 281
2020/05/04 19:57:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 285 length 100663296
2020/05/04 19:57:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 285 (retry=true): Post "https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0030": write tcp 192.168.1.2:48886->206.190.214.204:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0030", Err:(*net.OpError)(0xc000638000)}
2020/05/04 19:57:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0030": write tcp 192.168.1.2:48886->206.190.214.204:443: use of closed network connection
2020/05/04 19:57:33 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0030": write tcp 192.168.1.2:48886->206.190.214.204:443: use of closed network connection)
2020/05/04 19:57:33 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 19:57:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 285 length 100663296
2020/05/04 19:57:34 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 19:57:43 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 282
2020/05/04 19:57:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 286 length 100663296
2020/05/04 19:58:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 283
2020/05/04 19:58:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 287 length 100663296
2020/05/04 19:58:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 287 (retry=true): Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0016": write tcp 192.168.1.2:41626->206.190.214.206:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0016", Err:(*net.OpError)(0xc000639090)}
2020/05/04 19:58:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0016": write tcp 192.168.1.2:41626->206.190.214.206:443: use of closed network connection
2020/05/04 19:58:21 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0016": write tcp 192.168.1.2:41626->206.190.214.206:443: use of closed network connection)
2020/05/04 19:58:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 19:58:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 287 length 100663296
2020/05/04 19:58:22 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 19:59:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 284
2020/05/04 19:59:27 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 288 length 100663296
2020/05/04 20:01:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 285
2020/05/04 20:01:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 289 length 100663296
2020/05/04 20:01:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 289 (retry=true): Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043": write tcp 192.168.1.2:35726->206.190.214.209:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043", Err:(*net.OpError)(0xc0000980a0)}
2020/05/04 20:01:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043": write tcp 192.168.1.2:35726->206.190.214.209:443: use of closed network connection
2020/05/04 20:01:20 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043": write tcp 192.168.1.2:35726->206.190.214.209:443: use of closed network connection)
2020/05/04 20:01:20 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 20:01:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 289 length 100663296
2020/05/04 20:01:21 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 20:01:27 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 286
2020/05/04 20:01:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 290 length 100663296
2020/05/04 20:02:05 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 287
2020/05/04 20:02:10 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 291 length 100663296
2020/05/04 20:03:12 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 288
2020/05/04 20:03:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 292 length 100663296
2020/05/04 20:04:58 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 289
2020/05/04 20:05:03 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 293 length 100663296
2020/05/04 20:05:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 290
2020/05/04 20:05:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 294 length 100663296
2020/05/04 20:05:51 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 291
2020/05/04 20:05:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 295 length 100663296
2020/05/04 20:06:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 292
2020/05/04 20:07:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 296 length 100663296
2020/05/04 20:08:43 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 293
2020/05/04 20:08:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 297 length 100663296
2020/05/04 20:08:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 297 (retry=true): Post "https://pod-000-1132-06.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037": write tcp 192.168.1.2:41170->206.190.214.197:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-06.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037", Err:(*net.OpError)(0xc000662000)}
2020/05/04 20:08:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-06.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037": write tcp 192.168.1.2:41170->206.190.214.197:443: use of closed network connection
2020/05/04 20:08:48 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-06.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0037": write tcp 192.168.1.2:41170->206.190.214.197:443: use of closed network connection)
2020/05/04 20:08:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 20:08:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 297 length 100663296
2020/05/04 20:08:49 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 20:08:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 294
2020/05/04 20:09:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 298 length 100663296
2020/05/04 20:09:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 298 (retry=true): Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0027": write tcp 192.168.1.2:35758->206.190.214.209:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0027", Err:(*net.OpError)(0xc0001b8140)}
2020/05/04 20:09:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0027": write tcp 192.168.1.2:35758->206.190.214.209:443: use of closed network connection
2020/05/04 20:09:00 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-18.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0027": write tcp 192.168.1.2:35758->206.190.214.209:443: use of closed network connection)
2020/05/04 20:09:00 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 20:09:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 298 length 100663296
2020/05/04 20:09:01 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 20:09:36 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 295
2020/05/04 20:09:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 299 length 100663296
2020/05/04 20:10:40 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 296
2020/05/04 20:10:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 300 length 100663296
2020/05/04 20:10:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 300 (retry=true): Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:41434->206.190.214.200:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010", Err:(*net.OpError)(0xc000662000)}
2020/05/04 20:10:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:41434->206.190.214.200:443: use of closed network connection
2020/05/04 20:10:45 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:41434->206.190.214.200:443: use of closed network connection)
2020/05/04 20:10:45 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 20:10:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 300 length 100663296
2020/05/04 20:10:46 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 20:12:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 297
2020/05/04 20:12:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 301 length 100663296
2020/05/04 20:12:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 301 (retry=true): Post "https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0029": write tcp 192.168.1.2:58522->206.190.214.208:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0029", Err:(*net.OpError)(0xc000608be0)}
2020/05/04 20:12:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0029": write tcp 192.168.1.2:58522->206.190.214.208:443: use of closed network connection
2020/05/04 20:12:31 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-17.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0029": write tcp 192.168.1.2:58522->206.190.214.208:443: use of closed network connection)
2020/05/04 20:12:31 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 20:12:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 301 length 100663296
2020/05/04 20:12:32 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 20:12:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 298

JJB2020 · May 5, 2020, 8:11am

continued ...

2020/05/04 20:12:44 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 302 length 100663296
    2020/05/04 20:12:44 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 302 (retry=true): Post "https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:48956->206.190.214.204:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010", Err:(*net.OpError)(0xc0006633b0)}
    2020/05/04 20:12:44 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:48956->206.190.214.204:443: use of closed network connection
    2020/05/04 20:12:44 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-13.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:48956->206.190.214.204:443: use of closed network connection)
    2020/05/04 20:12:44 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:12:44 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 302 length 100663296
    2020/05/04 20:12:45 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:13:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 299
    2020/05/04 20:13:25 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 303 length 100663296
    2020/05/04 20:14:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 300
    2020/05/04 20:14:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 304 length 100663296
    2020/05/04 20:14:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 304 (retry=true): Post "https://pod-000-1132-02.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:52190->206.190.214.193:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-02.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004", Err:(*net.OpError)(0xc000098500)}
    2020/05/04 20:14:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-02.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:52190->206.190.214.193:443: use of closed network connection
    2020/05/04 20:14:29 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-02.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:52190->206.190.214.193:443: use of closed network connection)
    2020/05/04 20:14:29 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:14:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 304 length 100663296
    2020/05/04 20:14:30 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:16:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 301
    2020/05/04 20:16:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 305 length 100663296
    2020/05/04 20:16:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 302
    2020/05/04 20:16:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 306 length 100663296
    2020/05/04 20:17:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 303
    2020/05/04 20:17:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 307 length 100663296
    2020/05/04 20:18:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 304
    2020/05/04 20:18:36 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 308 length 100663296
    2020/05/04 20:18:36 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 308 (retry=true): Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0048": write tcp 192.168.1.2:57734->206.190.214.201:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0048", Err:(*net.OpError)(0xc0000980a0)}
    2020/05/04 20:18:36 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0048": write tcp 192.168.1.2:57734->206.190.214.201:443: use of closed network connection
    2020/05/04 20:18:36 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0048": write tcp 192.168.1.2:57734->206.190.214.201:443: use of closed network connection)
    2020/05/04 20:18:36 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:18:36 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 308 length 100663296
    2020/05/04 20:18:37 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:20:06 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 305
    2020/05/04 20:20:11 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 309 length 100663296
    2020/05/04 20:20:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 306
    2020/05/04 20:20:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 310 length 100663296
    2020/05/04 20:21:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 307
    2020/05/04 20:21:06 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 311 length 100663296
    2020/05/04 20:22:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 308
    2020/05/04 20:22:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 312 length 100663296
    2020/05/04 20:22:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 312 (retry=true): Post "https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043": write tcp 192.168.1.2:40876->206.190.214.192:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043", Err:(*net.OpError)(0xc000638140)}
    2020/05/04 20:22:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043": write tcp 192.168.1.2:40876->206.190.214.192:443: use of closed network connection
    2020/05/04 20:22:18 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0043": write tcp 192.168.1.2:40876->206.190.214.192:443: use of closed network connection)
    2020/05/04 20:22:18 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:22:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 312 length 100663296
    2020/05/04 20:22:19 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:23:50 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 309
    2020/05/04 20:23:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 313 length 100663296
    2020/05/04 20:23:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 313 (retry=true): Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0031": EOF: &url.Error{Op:"Post", URL:"https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0031", Err:(*errors.errorString)(0xc00007c040)}
    2020/05/04 20:23:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0031": EOF
    2020/05/04 20:23:55 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0031": EOF)
    2020/05/04 20:23:55 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:23:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 313 length 100663296
    2020/05/04 20:23:55 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:24:11 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 310
    2020/05/04 20:24:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 314 length 100663296
    2020/05/04 20:24:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 314 (retry=true): Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0038": write tcp 192.168.1.2:60102->206.190.214.210:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0038", Err:(*net.OpError)(0xc0006620f0)}
    2020/05/04 20:24:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0038": write tcp 192.168.1.2:60102->206.190.214.210:443: use of closed network connection
    2020/05/04 20:24:16 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0038": write tcp 192.168.1.2:60102->206.190.214.210:443: use of closed network connection)
    2020/05/04 20:24:16 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:24:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 314 length 100663296
    2020/05/04 20:24:16 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:24:43 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 311
    2020/05/04 20:24:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 315 length 100663296
    2020/05/04 20:24:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 315 (retry=true): Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:41496->206.190.214.200:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010", Err:(*net.OpError)(0xc000608000)}
    2020/05/04 20:24:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:41496->206.190.214.200:443: use of closed network connection
    2020/05/04 20:24:48 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:41496->206.190.214.200:443: use of closed network connection)
    2020/05/04 20:24:48 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:24:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 315 length 100663296
    2020/05/04 20:24:49 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:25:57 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 312
    2020/05/04 20:26:02 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 316 length 100663296
    2020/05/04 20:27:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 313
    2020/05/04 20:27:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 317 length 100663296
    2020/05/04 20:27:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 314
    2020/05/04 20:28:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 318 length 100663296
    2020/05/04 20:28:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 318 (retry=true): Post "https://pod-000-1132-03.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0023": write tcp 192.168.1.2:34834->206.190.214.194:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-03.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0023", Err:(*net.OpError)(0xc000638000)}
    2020/05/04 20:28:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-03.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0023": write tcp 192.168.1.2:34834->206.190.214.194:443: use of closed network connection
    2020/05/04 20:28:00 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-03.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0023": write tcp 192.168.1.2:34834->206.190.214.194:443: use of closed network connection)
    2020/05/04 20:28:00 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:28:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 318 length 100663296
    2020/05/04 20:28:01 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:28:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 315
    2020/05/04 20:28:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 319 length 100663296
    2020/05/04 20:29:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 316
    2020/05/04 20:29:43 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 320 length 100663296
    2020/05/04 20:31:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 317
    2020/05/04 20:31:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 321 length 100663296
    2020/05/04 20:31:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 318
    2020/05/04 20:31:47 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 322 length 100663296
    2020/05/04 20:32:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 319
    2020/05/04 20:32:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 323 length 100663296
    2020/05/04 20:33:19 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 320
    2020/05/04 20:33:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 324 length 100663296
    2020/05/04 20:34:57 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 321
    2020/05/04 20:35:02 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 325 length 100663296
    2020/05/04 20:35:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 322
    2020/05/04 20:35:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 326 length 100663296
    2020/05/04 20:35:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 323
    2020/05/04 20:36:05 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 327 length 100663296
    2020/05/04 20:37:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 324
    2020/05/04 20:37:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 328 length 100663296
    2020/05/04 20:38:40 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 325
    2020/05/04 20:38:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 329 length 100663296
    2020/05/04 20:39:12 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 326
    2020/05/04 20:39:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 330 length 100663296
    2020/05/04 20:39:41 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 327
    2020/05/04 20:39:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 331 length 100663296
    2020/05/04 20:40:52 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 328
    2020/05/04 20:40:57 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 332 length 100663296
    2020/05/04 20:42:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 329
    2020/05/04 20:42:30 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 333 length 100663296
    2020/05/04 20:42:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 330
    2020/05/04 20:43:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 334 length 100663296
    2020/05/04 20:43:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 334 (retry=true): Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0019": write tcp 192.168.1.2:53492->206.190.214.191:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0019", Err:(*net.OpError)(0xc000608000)}
    2020/05/04 20:43:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0019": write tcp 192.168.1.2:53492->206.190.214.191:443: use of closed network connection
    2020/05/04 20:43:00 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0019": write tcp 192.168.1.2:53492->206.190.214.191:443: use of closed network connection)
    2020/05/04 20:43:00 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:43:00 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 334 length 100663296
    2020/05/04 20:43:01 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:43:26 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 331
    2020/05/04 20:43:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 335 length 100663296
    2020/05/04 20:43:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 335 (retry=true): Post "https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0040": write tcp 192.168.1.2:60302->206.190.214.195:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0040", Err:(*net.OpError)(0xc000098190)}
    2020/05/04 20:43:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0040": write tcp 192.168.1.2:60302->206.190.214.195:443: use of closed network connection
    2020/05/04 20:43:31 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-04.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0040": write tcp 192.168.1.2:60302->206.190.214.195:443: use of closed network connection)
    2020/05/04 20:43:31 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:43:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 335 length 100663296
    2020/05/04 20:43:31 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:44:35 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 332
    2020/05/04 20:44:40 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 336 length 100663296
    2020/05/04 20:46:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 333
    2020/05/04 20:46:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 337 length 100663296
    2020/05/04 20:46:38 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 334
    2020/05/04 20:46:43 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 338 length 100663296
    2020/05/04 20:47:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 335
    2020/05/04 20:47:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 339 length 100663296
    2020/05/04 20:48:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 336
    2020/05/04 20:48:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 340 length 100663296
    2020/05/04 20:48:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 340 (retry=true): Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0017": write tcp 192.168.1.2:41828->206.190.214.206:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0017", Err:(*net.OpError)(0xc000098230)}
    2020/05/04 20:48:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0017": write tcp 192.168.1.2:41828->206.190.214.206:443: use of closed network connection
    2020/05/04 20:48:22 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-15.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0017": write tcp 192.168.1.2:41828->206.190.214.206:443: use of closed network connection)
    2020/05/04 20:48:22 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2020/05/04 20:48:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 340 length 100663296
    2020/05/04 20:48:23 DEBUG : pacer: Reducing sleep to 10ms
    2020/05/04 20:49:52 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 337

JJB2020 · May 5, 2020, 8:11am

continued ...

2020/05/04 20:49:58 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 341 length 100663296
2020/05/04 20:50:27 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 338
2020/05/04 20:50:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 342 length 100663296
2020/05/04 20:50:48 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 339
2020/05/04 20:50:53 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 343 length 100663296
2020/05/04 20:52:04 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 340
2020/05/04 20:52:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 344 length 100663296
2020/05/04 20:53:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 341
2020/05/04 20:53:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 345 length 100663296
2020/05/04 20:54:10 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 342
2020/05/04 20:54:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 346 length 100663296
2020/05/04 20:54:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 346 (retry=true): Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0050": write tcp 192.168.1.2:57880->206.190.214.201:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0050", Err:(*net.OpError)(0xc0001b8050)}
2020/05/04 20:54:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0050": write tcp 192.168.1.2:57880->206.190.214.201:443: use of closed network connection
2020/05/04 20:54:15 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0050": write tcp 192.168.1.2:57880->206.190.214.201:443: use of closed network connection)
2020/05/04 20:54:15 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 20:54:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 346 length 100663296
2020/05/04 20:54:15 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 20:54:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 343
2020/05/04 20:54:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 347 length 100663296
2020/05/04 20:55:47 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 344
2020/05/04 20:55:52 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 348 length 100663296
2020/05/04 20:57:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 345
2020/05/04 20:57:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 349 length 100663296
2020/05/04 20:57:55 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 346
2020/05/04 20:58:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 350 length 100663296
2020/05/04 20:58:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 347
2020/05/04 20:58:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 351 length 100663296
2020/05/04 20:59:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 348
2020/05/04 20:59:38 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 352 length 100663296
2020/05/04 21:01:06 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 349
2020/05/04 21:01:11 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 353 length 100663296
2020/05/04 21:01:47 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 350
2020/05/04 21:01:52 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 354 length 100663296
2020/05/04 21:01:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 351
2020/05/04 21:02:01 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 355 length 100663296
2020/05/04 21:03:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 352
2020/05/04 21:03:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 356 length 100663296
2020/05/04 21:04:52 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 353
2020/05/04 21:04:58 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 357 length 100663296
2020/05/04 21:05:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 354
2020/05/04 21:05:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 358 length 100663296
2020/05/04 21:05:40 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 355
2020/05/04 21:05:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 359 length 100663296
2020/05/04 21:07:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 356
2020/05/04 21:07:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 360 length 100663296
2020/05/04 21:08:40 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 357
2020/05/04 21:08:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 361 length 100663296
2020/05/04 21:08:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 361 (retry=true): Post "https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:54814->206.190.214.203:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010", Err:(*net.OpError)(0xc000662140)}
2020/05/04 21:08:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:54814->206.190.214.203:443: use of closed network connection
2020/05/04 21:08:45 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-12.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0010": write tcp 192.168.1.2:54814->206.190.214.203:443: use of closed network connection)
2020/05/04 21:08:45 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 21:08:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 361 length 100663296
2020/05/04 21:08:46 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 21:09:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 358
2020/05/04 21:09:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 362 length 100663296
2020/05/04 21:09:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 359
2020/05/04 21:09:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 363 length 100663296
2020/05/04 21:11:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 360
2020/05/04 21:11:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 364 length 100663296
2020/05/04 21:11:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 364 (retry=true): Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039": write tcp 192.168.1.2:41818->206.190.214.200:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039", Err:(*net.OpError)(0xc000662000)}
2020/05/04 21:11:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039": write tcp 192.168.1.2:41818->206.190.214.200:443: use of closed network connection
2020/05/04 21:11:14 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-09.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0039": write tcp 192.168.1.2:41818->206.190.214.200:443: use of closed network connection)
2020/05/04 21:11:14 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 21:11:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 364 length 100663296
2020/05/04 21:11:15 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 21:12:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 361
2020/05/04 21:12:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 365 length 100663296
2020/05/04 21:13:06 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 362
2020/05/04 21:13:10 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 366 length 100663296
2020/05/04 21:13:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 363
2020/05/04 21:13:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 367 length 100663296
2020/05/04 21:15:02 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 364
2020/05/04 21:15:06 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 368 length 100663296
2020/05/04 21:16:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 365
2020/05/04 21:16:22 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 369 length 100663296
2020/05/04 21:16:54 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 366
2020/05/04 21:16:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 370 length 100663296
2020/05/04 21:17:08 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 367
2020/05/04 21:17:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 371 length 100663296
2020/05/04 21:18:53 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 368
2020/05/04 21:18:58 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 372 length 100663296
2020/05/04 21:20:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 369
2020/05/04 21:20:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 373 length 100663296
2020/05/04 21:20:41 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 370
2020/05/04 21:20:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 374 length 100663296
2020/05/04 21:20:54 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 371
2020/05/04 21:20:58 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 375 length 100663296
2020/05/04 21:22:40 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 372
2020/05/04 21:22:45 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 376 length 100663296
2020/05/04 21:23:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 373
2020/05/04 21:24:04 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 377 length 100663296
2020/05/04 21:24:32 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 374
2020/05/04 21:24:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 378 length 100663296
2020/05/04 21:24:41 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 375
2020/05/04 21:24:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 379 length 100663296
2020/05/04 21:26:25 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 376
2020/05/04 21:26:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 380 length 100663296
2020/05/04 21:27:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 377
2020/05/04 21:27:50 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 381 length 100663296
2020/05/04 21:28:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 378
2020/05/04 21:28:27 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 379
2020/05/04 21:28:28 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 382 length 100663296
2020/05/04 21:28:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 382 (retry=true): Post "https://pod-000-1132-07.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006": write tcp 192.168.1.2:59706->206.190.214.198:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-07.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006", Err:(*net.OpError)(0xc0006080f0)}
2020/05/04 21:28:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-07.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006": write tcp 192.168.1.2:59706->206.190.214.198:443: use of closed network connection
2020/05/04 21:28:29 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-07.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006": write tcp 192.168.1.2:59706->206.190.214.198:443: use of closed network connection)
2020/05/04 21:28:29 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 21:28:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 382 length 100663296
2020/05/04 21:28:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 383 length 100663296
2020/05/04 21:28:34 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 21:30:13 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 380
2020/05/04 21:30:17 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 384 length 100663296
2020/05/04 21:31:33 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 381
2020/05/04 21:31:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 385 length 100663296
2020/05/04 21:32:11 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 382
2020/05/04 21:32:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 386 length 100663296
2020/05/04 21:32:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 386 (retry=true): Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006": write tcp 192.168.1.2:60414->206.190.214.210:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006", Err:(*net.OpError)(0xc000098140)}
2020/05/04 21:32:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006": write tcp 192.168.1.2:60414->206.190.214.210:443: use of closed network connection
2020/05/04 21:32:16 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-19.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0006": write tcp 192.168.1.2:60414->206.190.214.210:443: use of closed network connection)
2020/05/04 21:32:16 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 21:32:16 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 386 length 100663296
2020/05/04 21:32:16 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 21:32:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 383
2020/05/04 21:32:24 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 387 length 100663296
    
<snip>

2020/05/04 22:44:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 462 length 100663296
2020/05/04 22:44:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 463 length 100663296
2020/05/04 22:45:29 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 460
2020/05/04 22:45:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 464 length 100663296
2020/05/04 22:45:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 464 (retry=true): Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:58154->206.190.214.201:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004", Err:(*net.OpError)(0xc0004b2000)}
2020/05/04 22:45:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:58154->206.190.214.201:443: use of closed network connection
2020/05/04 22:45:34 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-10.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0004": write tcp 192.168.1.2:58154->206.190.214.201:443: use of closed network connection)
2020/05/04 22:45:34 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 22:45:34 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 464 length 100663296
2020/05/04 22:45:35 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 22:46:54 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 461
2020/05/04 22:46:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 465 length 100663296
2020/05/04 22:48:15 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 462

<snip>

2020/05/04 23:37:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 519 length 100663296
2020/05/04 23:37:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 519 (retry=true): Post "https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049": write tcp 192.168.1.2:41530->206.190.214.192:443: use of closed network connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049", Err:(*net.OpError)(0xc000098190)}
2020/05/04 23:37:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: Post "https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049": write tcp 192.168.1.2:41530->206.190.214.192:443: use of closed network connection
2020/05/04 23:37:21 DEBUG : pacer: low level retry 1/10 (error Post "https://pod-000-1132-01.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0049": write tcp 192.168.1.2:41530->206.190.214.192:443: use of closed network connection)
2020/05/04 23:37:21 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/05/04 23:37:21 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 519 length 100663296
2020/05/04 23:37:22 DEBUG : pacer: Reducing sleep to 10ms
2020/05/04 23:38:04 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 516
2020/05/04 23:38:09 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 520 length 100663296
2020/05/04 23:39:36 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 517

<snip>

2020/05/04 23:59:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 542 length 100663296
2020/05/04 23:59:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 539
2020/05/04 23:59:51 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 543 length 100663296
2020/05/04 23:59:51 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 543 (retry=false): Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005": tls: use of closed connection: &url.Error{Op:"Post", URL:"https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005", Err:(*errors.errorString)(0xc00007c6e0)}
2020/05/04 23:59:51 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 543: Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005": tls: use of closed connection
2020/05/04 23:59:56 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 544 length 100663296
2020/05/05 00:00:30 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 540
2020/05/05 00:01:38 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 541
2020/05/05 00:02:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 542
2020/05/05 00:02:23 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 544
2020/05/05 00:02:23 DEBUG : zfsdata-justin-b2origin.gz.gpg: Cancelling large file upload due to error: Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005": tls: use of closed connection
2020/05/05 00:02:26 Failed to rcat with 2 errors: last error was: Post "https://pod-000-1132-00.backblaze.com/b2api/v1/b2_upload_part/4_z4e1c8d52fa67a3c77a110314_f218658e224cd839c_d20200504_m142824_c002_v0001132_t0023/0005": tls: use of closed connection
gpg: [stdout]: write error: Broken pipe
gpg: DBG: deflate: iobuf_write failed
gpg: [stdout]: write error: Broken pipe
gpg: filter_flush failed on close: Broken pipe
gpg: [stdout]: write error: Broken pipe
gpg: filter_flush failed on close: Broken pipe

Apologies - I had to snip some of this debug output as there were too many lines. I've kept all the hiccups in.

ncw · May 5, 2020, 9:21am

It will yes.

Note this from the log - if your upload is bigger than 937GB then you will need to increase --b2-chunk-size . This isn't the problem you are having yet though!

B2 files can be at most 10,000 parts so if you want to upload a 1.8TiB file then you'll need a chunk size of at least 188M (rclone uses MiB) - say 200M to be on the safe size. Note that rclone buffers these chunks in memory.

Hmm, this looks rather like this problem: https://github.com/golang/go/issues/3514

Which was fixed years ago, but it has the same symptoms.

A "tls: use of closed connection" is outside of rclone's control it must be a problem in the go standard library.

I can't do anything directly about that - it is probably a bug in the go standard library.

I had a look through the bugs and issues and I don't see a fix for that.

I've made a workaround here which retries that error instead of treating it as a fatal error. Can you give that a go?

https://beta.rclone.org/branch/v1.51.0-279-g3c798995-fix-tls-closed-beta/ (uploaded in 15-30 mins)

JJB2020 · May 5, 2020, 2:08pm

Hi Nick - thank you for the quick response! I'll try again with a --b2-chunk-size 200M first and see how that goes. I'll report back. After that I will also try your workaround to the "tls: use of closed connection" issue.

Cheers - Justin

ncw · May 5, 2020, 2:30pm

I think you'll need both --b2-chunk-size 200M and the code above to do a successful upload

JJB2020 · May 5, 2020, 6:04pm

I set the upload going earlier, just with the increased --b2-chunk-size. So far so good. I'm not getting the send errors that I was previously getting, and the upload it still going at full pelt. The only hiccup it has had was this, quite early on in the upload, and my guess is it's an issue on backblaze side ...

2020/05/05 15:11:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 6 length 209715200
2020/05/05 15:11:52 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 3
2020/05/05 15:12:36 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 7 length 209715200
2020/05/05 15:12:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Error sending chunk 7 (retry=true): c002_v0001126_t0037 is too busy (503 service_unavailable): &pacer.retryAfterError{error:(*api.Error)(0xc000110ea0), retryAfter:1000000000}
2020/05/05 15:12:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Clearing part upload URL because of error: c002_v0001126_t0037 is too busy (503 service_unavailable)
2020/05/05 15:12:37 DEBUG : pacer: low level retry 1/10 (error c002_v0001126_t0037 is too busy (503 service_unavailable))
2020/05/05 15:12:37 DEBUG : pacer: Rate limited, increasing sleep to 1s
2020/05/05 15:12:37 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 7 length 209715200
2020/05/05 15:12:39 DEBUG : pacer: Reducing sleep to 500ms
2020/05/05 15:14:50 DEBUG : pacer: Reducing sleep to 250ms
2020/05/05 15:14:50 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 4
2020/05/05 15:15:11 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 8 length 209715200
2020/05/05 15:17:20 DEBUG : pacer: Reducing sleep to 125ms
2020/05/05 15:17:20 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 5
2020/05/05 15:17:47 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 9 length 209715200
2020/05/05 15:18:59 DEBUG : pacer: Reducing sleep to 62.5ms
2020/05/05 15:18:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 6
2020/05/05 15:19:19 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 10 length 209715200
2020/05/05 15:19:59 DEBUG : pacer: Reducing sleep to 31.25ms
2020/05/05 15:19:59 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 7
2020/05/05 15:20:14 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 11 length 209715200
2020/05/05 15:22:39 DEBUG : pacer: Reducing sleep to 15.625ms
2020/05/05 15:22:39 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 8
2020/05/05 15:22:53 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 12 length 209715200
2020/05/05 15:25:18 DEBUG : pacer: Reducing sleep to 10ms
2020/05/05 15:25:18 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 9
2020/05/05 15:25:31 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 13 length 209715200
2020/05/05 15:26:46 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 10
2020/05/05 15:26:58 DEBUG : zfsdata-justin-b2origin.gz.gpg: Sending chunk 14 length 209715200
2020/05/05 15:27:42 DEBUG : zfsdata-justin-b2origin.gz.gpg: Done sending chunk 11

What do you make of this?

ncw · May 5, 2020, 8:30pm

Looking good! It is part of the protocol that B2 gives 503 errors. Rclone then chooses a new upload server and carries on.

JJB2020 · May 26, 2020, 9:17am

Setting the --b2-chunk-size to an appropriate value (200M in my case) solved the problem for me. The upload completed successfull, after 10 days of continuious uploading! Of course the ultimate test will be to try to restore from that backup, which I'll do in the next few days.

ncw · May 26, 2020, 10:52am

Great - glad you got it uploaded!

system · May 29, 2020, 10:52am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

Large file upload to backblaze failed?

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

What is your rclone version (output from `rclone version`)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)