Our data source is very dynamic, some objects can be created and delete at a high rate (access tokens). When we trying to back up this data, we get errors: "Failed to copy: failed to open source object". We are trying to retry the command with parameters --max-backlog 1 --transfers 1 --retries 10 --retries-sleep 500ms
But in every retry we see the same error about the same object
ERROR : sys/token/id/h4ea39e9da44c402427a94c58a1f75afd2fbc2c42603f287649e5a0c092f2ab8f: Failed to copy: failed to open source object: NoSuchKey: The specified key does not exist.
status code: 404, request id: b60f278d-01b7-43a3-93b2-e48bc687c874, host id:
ERROR : Attempt 1/10 failed with 1 errors and: failed to open source object: NoSuchKey: The specified key does not exist. status code: 404, request id: b60f278d-01b7-43a3-93b2-e48bc687c874, host id:
ERROR : sys/token/id/h4ea39e9da44c402427a94c58a1f75afd2fbc2c42603f287649e5a0c092f2ab8f: Failed to copy: failed to open source object: NoSuchKey: The specified key does not exist.
status code: 404, request id: 61722858-d96c-4b68-8fe3-8eaf62920ae6, host id:
ERROR : Attempt 2/10 failed with 1 errors and: failed to open source object: NoSuchKey: The specified key does not exist.status code: 404, request id: 61722858-d96c-4b68-8fe3-8eaf62920ae6, host id:
ERROR : sys/token/id/h4ea39e9da44c402427a94c58a1f75afd2fbc2c42603f287649e5a0c092f2ab8f: Failed to copy: failed to open source object: NoSuchKey: The specified key does not exist.
status code: 404, request id: f8a4183f-275b-434a-907e-33e8b932f0cb, host id:
ERROR : Attempt 3/10 failed with 2 errors and: failed to open source object: NoSuchKey: The specified key does not exist. status code: 404, request id: f8a4183f-275b-434a-907e-33e8b932f0cb, host id:
etc
It tries to copy the same not existing object!!
Why doesn't it start from get the next object?
How can we change this command to handle our situation?
What is your rclone version (output from rclone version)
latest
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Docker image debian:stretch-slim
Which cloud storage system are you using? (eg Google Drive)
IBM Cloud Object Storage
The command you were trying to run (eg rclone copy /tmp remote:tmp)
thanks for quick reply
if we will split it to 2 steps the issue will be much more likely because the time between get and copy will increase further.
I would expect from --max-backlog 1 --transfers 1 to work so: "take 1 and copy it, take the next, copy it"
We use command rclone --config=rclone.config copyto --no-check-dest --max-backlog 1 --transfers 1 --retries 10 --retries-sleep 500ms
Do we need to change something to make the high level retry?
No. I checked, the object doesn't exist
I see similar logs for different buckets and objects: most retries fail on the same object (with --max-backlog 32 --transfers 32 too!)
Please advise
Rclone clearly saw the object in the listing, but when it came to download it, it wasn't there.
What is suprising is when rclone did the retry the object still was there in the listing.
This could be to do with the eventual consistency of the cloud storage - they are allowed to get out of sync listing vs actual state of the object for a short while.
Thank you for your help
The issue was because of Object Storage bug and was solved.
but it still happens that backup fails with this command rclone --config=rclone.config copyto --no-check-dest --max-backlog 1 --transfers 1 --retries 10 --retries-sleep 500ms
because an object was deleted and even with 10 retries (every time another object)
How can it be that "take 1 object, send it in 1 thread" fails?
It happens immediately, doesn't it?
Please help me to understand this phenomena:
We run copyto with flags --max-backlog 32 --transfers 32 for some bucket and get errors in all 3 retries:
Attempt 1/3 failed with 2 errors...
Attempt 2/3 failed with 37 errors...
Attempt 3/3 failed with 50 errors..
All the errors are about not existing objects (that were deleted during copying)
Then we try to run copyto with flags --max-backlog 1 --transfers 1 for the same bucket in order to minimize a chance to fail. We get errors again:
Attempt 1/10 failed with 228 errors...
Attempt 2/10 failed with 326 errors...
etc
Why do we have more errors in this case? We thought this will be more likely to work
Do you think it's worth to run with --max-backlog 1 --transfers 1 or we just need to increase retries with the fist configuration (--max-backlog 32 --transfers 32) ?
Another question connected to the "object not found" error during copying
If rclone copyto command failed because some objects were removed from the source during the process, does this mean that all other objects were copied?
Can we say that the destination bucket contains all the objects existing in the source bucket at the end of the process?
@ncw Can you please advise?
Can we consider a backup with errors of type "object not found" to be successful if we don't need removed objects??
Does it contain everything except removed objects?
That's really up to you and if you feel that's a good backup, I suppose that's your use case. If a file disappears before being copied, that's a failure normally with anything else.
I personally wouldn't assume anything works and run a check after wards if you are concerned about data integrity but if the source is changing, not sure how this would work. Most backups take a snapshot of the source to make it a point in time.