Local file to minio error

Help and Support
1

What is the problem you are having with rclone?

I now have two commands to copy millions and tens of millions of files,and the target minio already has most of the data, but the previous commands did not end normally, so I adjusted the checkers relatively high, both commands are almost the same

What is your rclone version (output from rclone version)

v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

win server 2007 64 bit

Which cloud storage system are you using? (eg Google Drive)

local->minio

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy xxx xxx --bwlimit 90M --transfers 8 --checkers 60 --ignore-existing -v --log-file

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

This task has scanned more than 5 million files with 36 errors

The First

<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/85CA93C5C944FCE317D2406B178C4B22_pdf.pdf.thumb.jpg</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/85CA93C5C944FCE317D2406B178C4B22_pdf.pdf.thumb.jpg</Resource><RequestId>160E9FD94CF3FDD1</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/85CA93C5C944FCE317D2406B178C4B22_pdf.pdf: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/877925FAA1DBE6A42B5C0BD4181E0862_file.jpg.thumb.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/877925FAA1DBE6A42B5C0BD4181E0862_pdf.pdf.thumb.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/828AE938119EF691C3D2EEDDEE813AB7_file.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/87A2A530ABBA45D1928911376D14C635_file.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/877925FAA1DBE6A42B5C0BD4181E0862_file.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/87A2A530ABBA45D1928911376D14C635_file.jpg.thumb.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/87A2A530ABBA45D1928911376D14C635_pdf.pdf: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/87A2A530ABBA45D1928911376D14C635_pdf.pdf.thumb.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/880732F9DF995ABC3023C66470711EA6_file.jpg.thumb.jpg: Copied (new)
2020/05/13 23:29:34 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/877925FAA1DBE6A42B5C0BD4181E0862_pdf.pdf: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/877925FAA1DBE6A42B5C0BD4181E0862_pdf.pdf</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/877925FAA1DBE6A42B5C0BD4181E0862_pdf.pdf</Resource><RequestId>160E9FD9586E02B9</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/880732F9DF995ABC3023C66470711EA6_pdf.pdf.thumb.jpg: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/880732F9DF995ABC3023C66470711EA6_pdf.pdf: Copied (new)
2020/05/13 23:29:34 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/880732F9DF995ABC3023C66470711EA6_file.jpg: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/891F2EE8A971E01086920067D3CE28CE_file.jpg.thumb.jpg: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/891F2EE8A971E01086920067D3CE28CE_pdf.pdf.thumb.jpg: Copied (new)
2020/05/13 23:29:35 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/891F2EE8A971E01086920067D3CE28CE_file.jpg: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/891F2EE8A971E01086920067D3CE28CE_file.jpg</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/891F2EE8A971E01086920067D3CE28CE_file.jpg</Resource><RequestId>160E9FD97ECF55B0</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/891F2EE8A971E01086920067D3CE28CE_pdf.pdf: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A7715ACD2478C6DDAE432BFC81873A6_file.jpg: Copied (new)
2020/05/13 23:29:35 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A7715ACD2478C6DDAE432BFC81873A6_file.jpg.thumb.jpg: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A7715ACD2478C6DDAE432BFC81873A6_file.jpg.thumb.jpg</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A7715ACD2478C6DDAE432BFC81873A6_file.jpg.thumb.jpg</Resource><RequestId>160E9FD989FF0317</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A875E52F3B86CC15CA8CB51B98CE820_file.jpg.thumb.jpg: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A7715ACD2478C6DDAE432BFC81873A6_pdf.pdf: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A7715ACD2478C6DDAE432BFC81873A6_pdf.pdf.thumb.jpg: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A875E52F3B86CC15CA8CB51B98CE820_pdf.pdf.thumb.jpg: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A875E52F3B86CC15CA8CB51B98CE820_pdf.pdf: Copied (new)
2020/05/13 23:29:35 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A875E52F3B86CC15CA8CB51B98CE820_file.jpg: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A875E52F3B86CC15CA8CB51B98CE820_file.jpg</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8A875E52F3B86CC15CA8CB51B98CE820_file.jpg</Resource><RequestId>160E9FD9905BD699</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8AFB62A7A6D3993AD40E020521D926BB_file.jpg.thumb.jpg: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8AFB62A7A6D3993AD40E020521D926BB_file.jpg: Copied (new)
2020/05/13 23:29:35 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8AFB62A7A6D3993AD40E020521D926BB_pdf.pdf.thumb.jpg: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8AFB62A7A6D3993AD40E020521D926BB_pdf.pdf.thumb.jpg</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8AFB62A7A6D3993AD40E020521D926BB_pdf.pdf.thumb.jpg</Resource><RequestId>160E9FD9A8899E5D</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8AFB62A7A6D3993AD40E020521D926BB_pdf.pdf: Copied (new)
2020/05/13 23:29:35 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8BCE68A8ECE3287F3D1F4C25544C4C9B_file.jpg.thumb.jpg: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8BCE68A8ECE3287F3D1F4C25544C4C9B_file.jpg.thumb.jpg</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8BCE68A8ECE3287F3D1F4C25544C4C9B_file.jpg.thumb.jpg</Resource><RequestId>160E9FD9AF122C72</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:35 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8BCE68A8ECE3287F3D1F4C25544C4C9B_pdf.pdf.thumb.jpg: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>You did not provide the number of bytes specified by the Content-Length HTTP header.</Message><Key>3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8BCE68A8ECE3287F3D1F4C25544C4C9B_pdf.pdf.thumb.jpg</Key><BucketName>lnzy</BucketName><Resource>/lnzy/3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8BCE68A8ECE3287F3D1F4C25544C4C9B_pdf.pdf.thumb.jpg</Resource><RequestId>160E9FD9B248DD8E</RequestId><HostId>4bd6a9b8-d78d-4825-a8f2-0f95ee81c0d2</HostId></Error>
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8BCE68A8ECE3287F3D1F4C25544C4C9B_pdf.pdf: Copied (new)
2020/05/13 23:29:35 INFO  : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8EB4972772038758BCAFCCA645387AF7_file.jpg: Copied (new)
2020/05/13 23:29:35 ERROR : 3815/201901/20100/275E04F4AAB81BE1AB9B9F9FE3540FCF/archives/8EB4972772038758BCAFCCA645387AF7_file.jpg.thumb.jpg: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>

The second task error log is as follows



2020/05/14 06:47:14 ERROR : writ_template_mark_config/20140427: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=writ_template_mark_config%2F20140427%2F: net/http: timeout awaiting response headers
2020/05/14 06:49:09 ERROR : 2020/2020-04-30: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=2020%2F2020-04-30%2F: net/http: timeout awaiting response headers
2020/05/14 06:52:53 ERROR : 2.6.4.3/2/2/dzjz/temp: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=2.6.4.3%2F2%2F2%2Fdzjz%2Ftemp%2F: net/http: timeout awaiting response headers
2020/05/14 06:59:21 ERROR : 3779/201112/2/0C25644F45C91811F7C8E98E341ECABF/archives: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=3779%2F201112%2F2%2F0C25644F45C91811F7C8E98E341ECABF%2Farchives%2F: net/http: timeout awaiting response headers
2020/05/14 07:02:55 ERROR : writ_entity/1/20150515: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=writ_entity%2F1%2F20150515%2F: net/http: timeout awaiting response headers
2020/05/14 07:06:44 ERROR : writ_entity/1/20150519: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=writ_entity%2F1%2F20150519%2F: net/http: timeout awaiting response headers
2020/05/14 07:15:18 ERROR : 3779/201301/1/00000000000000000000000251658982/archives: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=3779%2F201301%2F1%2F00000000000000000000000251658982%2Farchives%2F: net/http: timeout awaiting response headers
2020/05/14 07:20:27 ERROR : 3778/199704/2/1EEC9CCB2ABF4D0646DE6D26A423E177/archives: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=3778%2F199704%2F2%2F1EEC9CCB2ABF4D0646DE6D26A423E177%2Farchives%2F: net/http: timeout awaiting response headers
2020/05/14 07:22:07 ERROR : 2019/2019-02-28: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=2019%2F2019-02-28%2F: net/http: timeout awaiting response headers
2020/05/14 07:22:38 ERROR : 3779/201212/2/00000000000000000000000251658566/archives: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=3779%2F201212%2F2%2F00000000000000000000000251658566%2Farchives%2F: net/http: timeout awaiting response headers
2020/05/14 07:22:43 ERROR : writ_entity/2/20150519: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=writ_entity%2F2%2F20150519%2F: net/http: timeout awaiting response headers
2020/05/14 07:25:55 ERROR : writ_entity/2/20150520: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=writ_entity%2F2%2F20150520%2F: net/http: timeout awaiting response headers
2020/05/14 07:26:16 ERROR : 3779/201208/2/3273DF39514F519113D8A4F2C392D356/archives: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=3779%2F201208%2F2%2F3273DF39514F519113D8A4F2C392D356%2Farchives%2F: net/http: timeout awaiting response headers
2020/05/14 07:27:54 ERROR : 3779/201204/1/B68239E83EF5BD60BD4933D7E276EF87/archives: error reading destination directory: RequestError: send request failed
caused by: Get http://155.1.193.146:9000/jqzy?delimiter=%2F&max-keys=1000&prefix=3779%2F201204%2F1%2FB68239E83EF5BD60BD4933D7E276EF87%2Farchives%2F: net/http: timeout awaiting response headers
2020/05/14 07:29:51 ERROR : 3779/201305/1/00000000000000000000000251660311/archives: error reading destination directory: RequestError: send request failed
2

This means that the transfer broke before the upload was completed.

That means rclone was waiting for minio to respond and it didn't in time

These both look like networking problems. Is it local the minio cluster?

Perhaps it is overloaded and responding very slowly? You could try increasing --timeout and --conntimeout.

You could also try using a --tpslimit 100 to slow it down a bit.

3

minio is not local

"transfer broke before the upload was completed",but I look like this minio error is like missing Content-Length instead of being interrupted

--tpslimit Limit HTTP transactions per second to this,how should i understand him

4

There is another question I want to know, for example, will uploading the interrupted file be the first time after scanning all the files, and then retry the failed file?

5

You did not provide the number of bytes specified by the Content-Length HTTP header

That says that the number of bytes in body of the request did not match the number of bytes supplied in the Content-Length header.

Yes it will do all the transfers then retry the sync from the start. It will probably pick up all the missing files then.

6 
2020/05/14 09:40:22 ERROR : Attempt 1/3 failed with 61 errors and: march failed with 60 error(s): first error: write tcp 10.1.1.234:36928->155.8.0.6:21: write: broken pipe
2020/05/14 10:19:33 ERROR : : error reading source directory: write tcp 10.1.1.234:54626->155.8.0.6:21: write: broken pipe
2020/05/14 10:19:33 ERROR : Attempt 2/3 failed with 1 errors and: write tcp 10.1.1.234:54626->155.8.0.6:21: write: broken pipe
2020/05/14 10:33:07 ERROR : : error reading source directory: write tcp 10.1.1.234:54609->155.8.0.6:21: write: broken pipe
2020/05/14 10:33:07 ERROR : Attempt 3/3 failed with 1 errors and: write tcp 10.1.1.234:54609->155.8.0.6:21: write: broken pipe

Can I understand this log output as retrying these 61 errors, 60 of which were successful, and retrying the last error, write: broken pipe, still failed, but this error does not seem to affect So is this command normal operation over?

7

The retries are normal for any errors. By default it will try 3 times. The errors aren't normal.

8

Can this error be retried? See the log output. Is my task ended normally?

9

If it tried 3 times on that file and still failed, it had essentially skipped that file. You can resync and it'll fix anything that was missed. You should see a summary at the end of the sync.

10

But the error I understand is just a ftp connection disconnected, so retrying has no effect

11

Is there any command to check the modification time of the directory to skip some directories, because if the number of files is millions, re-copying will be very slow, even if the checkers are adjusted to 60

12

You could just transfer the one file if you know the others were successful. If you know which directories you can exclude you can also do that with filtering.

There's also lots of list commands if you want to check dates.
There are several related list commands

  • ls to list size and path of objects only
  • lsl to list modification time, size and path of objects only
  • lsd to list directories only
  • lsf to list objects and directories in easy to parse format
  • lsjson to list objects and directories in JSON format

lsf is probably what you want though.

https://rclone.org/commands/rclone_lsf/

13

Can i set a directory such as test1 and copy it to the test1 directory under minio1: bucket1

14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.