Failed to copy: Object Corrupted

Suspected Bug
1

Hey,

I was trying to sync files between my two Swift clusters ( old to new ), using this command:
rclone sync src dest
During the transfer ( Attempt 1/3 ) some of the files failed to copy, indicating “Failed to copy: Object Corrupted”.
Attempt 1 ends with:
"2019-03-11 09:52:04 ERROR : Attempt 1/3 failed with 80 errors and: Object Corrupted"
80 errors out of 228 files.

On attempt 2/3 everything works just fine and all files are copied:
"2019-03-11 09:52:07 ERROR : Attempt 2/3 succeeded"

I ran a few more tests and came up with these conclusions:

  1. It is the same number of “corrupted” files every time.
  2. Not 100% sure, but I think It only occurs with *.js and *.css files.
  3. I get the same behavior when trying to sync from Swift to a local path on a Linux machine.

Specs:
rclone v1.46

  • os/arch: linux/amd64
  • go version: go1.11.5

In the config file i only used the following fields:
type = swift
env_auth = false
user =
key =
auth =
endpoint_type = public

Could not find any reference regarding this issue, was hoping I could get some here.

Thanks in advance,
Shlomi

2

That is strange…

That is an error from the swift library rather than rclone itself.

That probably means it is a bug somewhere!

Can you try copying just one of these files with rclone copy -vv --dump bodies src:bucket/object.js - does it show the error there? If so could you post the log somewhere?

Can you try your original sync with --transfers 1 --checkers 1 to see if that makes a difference?

You could try it with --fast-list also.

3

I tried your suggestions but I get the same result.
The dump does not show any particular error:

The most weird thing is that I reduced the retries to only 1, i still get this message at the end:

2019/03/11 12:36:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/03/11 12:36:29 ERROR : file: Failed to copy: Object Corrupted
2019/03/11 12:36:29 ERROR : Attempt 1/1 failed with 1 errors and: Object Corrupted
2019/03/11 12:36:29 Failed to copy: Object Corrupted

But the file was actually downloaded!
I also checked the Swift’s logs and all I can see is 200 for the GET request.

This is why the second attempt succeeded - the file was downloaded the first time, even though it says it didn’t, and on attempt 2/3 the file was already in the destination so no need to download it again.

4

That is some strange stuff going on! What can you tell me about the source swift? How old is it? Is it openstack or CEPH or something else?

5

It is really strange indeed.
It is OpenStack Swift object storage.
I tried running this on 2 different clusters with different versions (older an newer), same result.
I really think it has something do with the file itself rather than the storage endpoint.

6

Hmm, interesting…

Any chance you could get me access to the cluster? If so private message me!

7

Yea we can do that, not sure how to private message you tho…

8

Just click on my icon, there should be a “message” button there.

9

I tried that already, did not see any message option

10

I messaged you - hopefully you can reply to that

11

@shlomiz
Have tried using --no-gzip-encoding option.?

I had the same issue several months ago when I try to download bunch of tar.gz files,
in my case using this option has solved the problem.

12

@David_Song

Thank you for the suggestion, that actually worked!

1 Like
13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.