Failed to copy: Object Corrupted


#1

Hey,

I was trying to sync files between my two Swift clusters ( old to new ), using this command:
rclone sync src dest
During the transfer ( Attempt 1/3 ) some of the files failed to copy, indicating “Failed to copy: Object Corrupted”.
Attempt 1 ends with:
"2019-03-11 09:52:04 ERROR : Attempt 1/3 failed with 80 errors and: Object Corrupted"
80 errors out of 228 files.

On attempt 2/3 everything works just fine and all files are copied:
"2019-03-11 09:52:07 ERROR : Attempt 2/3 succeeded"

I ran a few more tests and came up with these conclusions:

  1. It is the same number of “corrupted” files every time.
  2. Not 100% sure, but I think It only occurs with *.js and *.css files.
  3. I get the same behavior when trying to sync from Swift to a local path on a Linux machine.

Specs:
rclone v1.46

  • os/arch: linux/amd64
  • go version: go1.11.5

In the config file i only used the following fields:
type = swift
env_auth = false
user =
key =
auth =
endpoint_type = public

Could not find any reference regarding this issue, was hoping I could get some here.

Thanks in advance,
Shlomi


#2

That is strange…

That is an error from the swift library rather than rclone itself.

That probably means it is a bug somewhere!

Can you try copying just one of these files with rclone copy -vv --dump bodies src:bucket/object.js - does it show the error there? If so could you post the log somewhere?

Can you try your original sync with --transfers 1 --checkers 1 to see if that makes a difference?

You could try it with --fast-list also.


#3

I tried your suggestions but I get the same result.
The dump does not show any particular error:

The most weird thing is that I reduced the retries to only 1, i still get this message at the end:

2019/03/11 12:36:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/03/11 12:36:29 ERROR : file: Failed to copy: Object Corrupted
2019/03/11 12:36:29 ERROR : Attempt 1/1 failed with 1 errors and: Object Corrupted
2019/03/11 12:36:29 Failed to copy: Object Corrupted

But the file was actually downloaded!
I also checked the Swift’s logs and all I can see is 200 for the GET request.

This is why the second attempt succeeded - the file was downloaded the first time, even though it says it didn’t, and on attempt 2/3 the file was already in the destination so no need to download it again.


#4

That is some strange stuff going on! What can you tell me about the source swift? How old is it? Is it openstack or CEPH or something else?


#5

It is really strange indeed.
It is OpenStack Swift object storage.
I tried running this on 2 different clusters with different versions (older an newer), same result.
I really think it has something do with the file itself rather than the storage endpoint.


#6

Hmm, interesting…

Any chance you could get me access to the cluster? If so private message me!


#7

Yea we can do that, not sure how to private message you tho…


#8

Just click on my icon, there should be a “message” button there.


#9

I tried that already, did not see any message option


#10

I messaged you - hopefully you can reply to that


#11

@shlomiz
Have tried using --no-gzip-encoding option.?

I had the same issue several months ago when I try to download bunch of tar.gz files,
in my case using this option has solved the problem.


#12

@David_Song

Thank you for the suggestion, that actually worked!