Failed to set modification time: InvalidArgument:

pravesh.upadhyay · March 4, 2020, 9:39am

Hi Team,

I am trying to migrate data from two s3 Compliant storage. Dell EMC ECS and Cloudian Hyperstore. Encountered with memory leak issues hence using the fixed version.
C:\Users\rclone>rclone.exe version
rclone v1.50.1-048-g4005e125-fix-memory-leak-beta

os/arch: windows/amd64
go version: go1.13.4
.
When i try to sync the data between two buckets it fails with error :-
Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details

After going through lot of issues decided to use the sync command with --s3-upload-cutoff 0. Iit gives following errors:-

   status code: 400, request id: a9261568-ff28-1fff-af88-ac1f6ba4d7c3, host id:

020/03/04 09:07:22 ERROR : abc.pdf: Failed to set modification time: InvalidArgument: Range specified is not valid for source obect of size: 237057
status code: 400, request id: 4b9eb80d-c4b6-1e3e-a1ff-ac1f6ba4d32d, host id:
020/03/04 09:07:22 ERROR : xyz.pdf: Failed to set modification time: InvalidArgument: Range specified is not valid for source obect of size: 237057

the check works perfectly fine but sync always creates an issue. Please help us with the resolution for this issue.

Thanks a lot in advance.

Regards
Pravesh Upadhyay

ncw · March 4, 2020, 11:48am

This fix was released in v1.51 and is improved further in the latest beta.

We've had various problems reported with Cloudian before, eg

Are you using the latest Cloudian version?

Can you run with -vv --dump responses and post the relevant HTTP request and response for that error? Or alternatively do a small run to generate the error and post the whole log.

pravesh.upadhyay · March 4, 2020, 3:43pm

Hi Nick,
Yes we are using the latest version of Cloudian.

Please find below the HTTP request and response for your reference:-

2020/03/04 16:38:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/04 16:38:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/04 16:38:03 DEBUG : HTTP REQUEST (req 0xc000175500)
2020/03/04 16:38:03 DEBUG : GET /bucket2b?delimiter=%2F&marker=%23MA%20%23nderungsblatt%20Greifer%20Modifikation_20200114173032.pdf&max-keys=1000&prefix= HTTP/1.1
Host: s3-edc.corpintra.net
User-Agent: rclone/v1.50.2
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200304T153802Z
Accept-Encoding: gzip

2020/03/04 16:38:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/04 16:38:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/04 16:38:03 DEBUG : HTTP REQUEST (req 0xc0000ae700)
2020/03/04 16:38:03 DEBUG : GET /bucket1b?delimiter=%2F&marker=%23MA%20%23nderungsblatt%20Greifer%20Modifikation-ah_20200214123001.pdf&max-keys=1000&prefix= HTTP/1.1
Host: ecs.corpintra.net:9021
User-Agent: rclone/v1.50.2
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200304T153803Z
Accept-Encoding: gzip

2020/03/04 16:38:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/04 16:38:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/04 16:38:03 DEBUG : HTTP RESPONSE (req 0xc000175500)
2020/03/04 16:38:03 DEBUG : HTTP/1.1 200 OK
Content-Length: 374246
Content-Type: application/xml;charset=UTF-8
Date: Wed, 04 Mar 2020 15:38:03 GMT
Server: CloudianS3
X-Amz-Bucket-Region: edc
X-Amz-Request-Id: b5609c8d-b22a-1f14-a5cc-ac1f6ba4d775
X-Gmt-Policyid: 7d6bffd19c4e48552795a8eb59a6ce28

Regards
Pravesh Upadhyay

ncw · March 5, 2020, 10:54am

That doesn't appear to show the error Can you send the whole log? If you dno't want to post it publically you could email it to nick@craig-wood.com with a link to this page.

pravesh.upadhyay · March 10, 2020, 8:58am

Hi Nick,

This buckets consists of 58 millions objects the log size will be huge before it actually starts giving errors.
Interestingly for small size buckets, i dont see this error.

Please suggest an alternate way.
Thank You.

Regards
Pravesh

ncw · March 10, 2020, 11:03am

I don't need the whole log, but I do need a bit with an error in - can you cut it down a bit?

pravesh.upadhyay · March 11, 2020, 2:04pm

Hi Nick,

Please find below some error messages that i could gather from the log:-

2020/03/11 07:19:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 07:19:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 07:19:36 DEBUG : HTTP REQUEST (req 0xc0c8e7af00)
2020/03/11 07:19:36 DEBUG : PUT /abcbucket/angebot.pdf?partNumber=1&uploadId=ZWIyZTM3NDUzMTU4MzkwNzU3NjQ4OA HTTP/1.1
Host: s3-edc.corpintra.net
User-Agent: rclone/v1.51.0-082-g0c6f14c6-beta
Content-Length: 108129
Authorization: XXXX
Content-Md5: 5gTX0oU9xNaxn0UKux+AMA==
X-Amz-Content-Sha256: fee4c85a4c2c2e2fcbe2641d1b6d13edd2c8efa2ea56f9d8f180e6a6ce67bc5a
X-Amz-Date: 20200311T061936Z
Accept-Encoding: gzip

2020/03/11 07:19:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 07:19:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 07:19:36 DEBUG : HTTP RESPONSE (req 0xc0c8e7af00)
2020/03/11 07:19:36 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 356
Content-Type: application/xml
Date: Wed, 11 Mar 2020 06:19:36 GMT
Server: CloudianS3
X-Amz-Request-Id: 926ddfb7-ffff-1fff-a00a-ac1f6ba5aa2f

<?xml version="1.0" encoding="UTF-8"?>NoSuchUploadThe specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.926ddfb7-ffff-1fff-a00a-ac1f6ba5aa2f23VrgXaycPFvU_uw7QT-zzHdwYOrZ3VNssgJ4-DcHLg

2020/03/11 07:19:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 07:19:36 DEBUG : angebot.pdf: Cancelling multipart upload
2020/03/11 07:19:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

#########################################################################################################################

2020/03/11 06:58:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 06:58:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 06:58:32 DEBUG : HTTP REQUEST (req 0xc04ac06200)
2020/03/11 06:58:32 DEBUG : DELETE /abcbucket/AN000044441.pdf?uploadId=MWJlOWIzMTE1NjE1ODM5MDYzMTIwODM HTTP/1.1
Host: s3-edc.corpintra.net
User-Agent: rclone/v1.51.0-082-g0c6f14c6-beta
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200311T055832Z
Accept-Encoding: gzip

2020/03/11 06:58:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 06:58:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 06:58:32 DEBUG : HTTP RESPONSE (req 0xc04ac06200)
2020/03/11 06:58:32 DEBUG : HTTP/1.1 204 No Content
#########################################################################################################################

2020/03/11 06:59:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 06:59:49 DEBUG : HTTP REQUEST (req 0xc060202400)
2020/03/11 06:59:49 DEBUG : DELETE /abcbucket/20200310083007.pdf?uploadId=MTYyNGEyMTE1ODE1ODM5MDYzODkxMTM HTTP/1.1
User-Agent: rclone/v1.51.0-082-g0c6f14c6-beta
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200311T055949Z
Accept-Encoding: gzip

2020/03/11 06:59:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 06:59:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 06:59:49 DEBUG : HTTP RESPONSE (req 0xc05f830700)
2020/03/11 06:59:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 06:59:49 DEBUG : HTTP RESPONSE (req 0xc060202400)
2020/03/11 06:59:49 DEBUG : HTTP/1.1 204 No Content
Date: Wed, 11 Mar 2020 05:59:49 GMT
Server: CloudianS3
X-Amz-Request-Id: 2d269d74-ffbb-1fff-a7f9-ac1f6ba5d11f

2020/03/11 06:59:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 06:59:49 ERROR : 20200310083007.pdf: Failed to copy: multipart upload failed to upload part: NoSuchUpload: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.

###################################################################################################################333

<?xml version="1.0" encoding="UTF-8"?>abcbucket/RELEASE_86935431_Doc_2001071042_20200310105331.pdfabcbucketRELEASE_86935431_Doc_2001071042_20200310105331.pdf"b8a377213f75c8801fdb02f6ffdbc170-1"

2020/03/11 07:18:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 07:18:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 07:18:45 DEBUG : HTTP REQUEST (req 0xc10b8d1f00)
2020/03/11 07:18:45 DEBUG : HEAD /abcbucket/technicalspecs.pdf HTTP/1.1
User-Agent: rclone/v1.51.0-082-g0c6f14c6-beta
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200311T061845Z

2020/03/11 07:18:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/03/11 07:18:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 07:18:45 DEBUG : HTTP RESPONSE (req 0xc10be20c00)
2020/03/11 07:18:45 DEBUG : HTTP/1.1 204 No Content
Date: Wed, 11 Mar 2020 06:18:45 GMT
Server: CloudianS3
X-Amz-Request-Id: 926dbc16-ffff-1fff-a00a-ac1f6ba5aa2f

2020/03/11 07:18:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/03/11 07:18:45 ERROR : technicalspecs.pdf: Failed to copy: multipart upload failed to finalise: InvalidPart: One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag.
status code: 400, request id: 926dbc14-ffff-1fff-a00a-ac1f6ba5aa2f, host id:

Hope this helps.

Thanks and Regards
Pravesh Upadhyay

ncw · March 11, 2020, 5:06pm

What happened to that upload ID? did it get aborted or completed? Or did cloudian get confused. I need to see more log to figure it out!

If you want to figure out a way to send the full log then you could email it or a link to it to nick@craig-wood.com - put a link to this page in if you do that.

pravesh.upadhyay · March 12, 2020, 4:05pm

Hi Nick,

Shared you the logs on email.

Regards
Pravesh

ncw · March 12, 2020, 5:20pm

Thanks.

I see rclone doing this

intitiate multpart upload

2020/03/11 06:59:49 DEBUG : HTTP REQUEST (req 0xc054ddb000)
2020/03/11 06:59:49 DEBUG : POST /abcbucket/A%2019-1571_20200310083007.pdf?uploads= HTTP/1.1

response 200 OK with upload id MTYyNGEyMTE1ODE1ODM5MDYzODkxMTM

2020/03/11 06:59:49 DEBUG : HTTP RESPONSE (req 0xc054ddb000)
2020/03/11 06:59:49 DEBUG : HTTP/1.1 200 OK

The next time that upload id is used is here

2020/03/11 06:59:49 DEBUG : HTTP REQUEST (req 0xc05edd5f00)
2020/03/11 06:59:49 DEBUG : PUT /abcbucket/A%2019-1571_20200310083007.pdf?partNumber=1&uploadId=MTYyNGEyMTE1ODE1ODM5MDYzODkxMTM HTTP/1.1

And the response is

2020/03/11 06:59:49 DEBUG : HTTP RESPONSE (req 0xc05edd5f00)
2020/03/11 06:59:49 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 356
Content-Type: application/xml
Date: Wed, 11 Mar 2020 05:59:49 GMT
Server: CloudianS3
X-Amz-Request-Id: 4ca22013-ffd4-1fff-9f50-ac1f6ba5c7e1

<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchUpload</Code><Message>The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.</Message><RequestId>4ca22013-ffd4-1fff-9f50-ac1f6ba5c7e1</RequestId><HostId>RDB2Uu9NjC0PLcH4acMJVKIjrgSBbujwf0SZVJZB95o</HostId></Error>

So it looks like rclone did everything properly, but somehow Cloudian got confused about the upload ID. Something I did notice is that rclone started two uploads very close together which maybe causes the problem.

You could try --transfers 1 to see if that helps.

system · May 12, 2020, 1:20pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.