I was trying to sync files between my two Swift clusters ( old to new ), using this command:
rclone sync src dest
During the transfer ( Attempt 1/3 ) some of the files failed to copy, indicating “Failed to copy: Object Corrupted”.
Attempt 1 ends with:
"2019-03-11 09:52:04 ERROR : Attempt 1/3 failed with 80 errors and: Object Corrupted"
80 errors out of 228 files.
On attempt 2/3 everything works just fine and all files are copied:
"2019-03-11 09:52:07 ERROR : Attempt 2/3 succeeded"
I ran a few more tests and came up with these conclusions:
- It is the same number of “corrupted” files every time.
- Not 100% sure, but I think It only occurs with *.js and *.css files.
- I get the same behavior when trying to sync from Swift to a local path on a Linux machine.
- os/arch: linux/amd64
- go version: go1.11.5
In the config file i only used the following fields:
type = swift
env_auth = false
endpoint_type = public
Could not find any reference regarding this issue, was hoping I could get some here.
Thanks in advance,
That is strange…
That is an error from the swift library rather than rclone itself.
That probably means it is a bug somewhere!
Can you try copying just one of these files with
rclone copy -vv --dump bodies src:bucket/object.js - does it show the error there? If so could you post the log somewhere?
Can you try your original sync with
--transfers 1 --checkers 1 to see if that makes a difference?
You could try it with
I tried your suggestions but I get the same result.
The dump does not show any particular error:
The most weird thing is that I reduced the retries to only 1, i still get this message at the end:
2019/03/11 12:36:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/03/11 12:36:29 ERROR : file: Failed to copy: Object Corrupted
2019/03/11 12:36:29 ERROR : Attempt 1/1 failed with 1 errors and: Object Corrupted
2019/03/11 12:36:29 Failed to copy: Object Corrupted
But the file was actually downloaded!
I also checked the Swift’s logs and all I can see is 200 for the GET request.
This is why the second attempt succeeded - the file was downloaded the first time, even though it says it didn’t, and on attempt 2/3 the file was already in the destination so no need to download it again.
That is some strange stuff going on! What can you tell me about the source swift? How old is it? Is it openstack or CEPH or something else?
It is really strange indeed.
It is OpenStack Swift object storage.
I tried running this on 2 different clusters with different versions (older an newer), same result.
I really think it has something do with the file itself rather than the storage endpoint.
Any chance you could get me access to the cluster? If so private message me!
Yea we can do that, not sure how to private message you tho…
Just click on my icon, there should be a “message” button there.
I tried that already, did not see any message option
I messaged you - hopefully you can reply to that
Have tried using --no-gzip-encoding option.?
I had the same issue several months ago when I try to download bunch of tar.gz files,
in my case using this option has solved the problem.
Thank you for the suggestion, that actually worked!
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.