How to ignore "Failed to copy: failed to open source object" errors

Help and Support
1

What is the problem you are having with rclone?

Our data source is very dynamic, some objects can be created and delete at a high rate (access tokens). When we trying to back up this data, we get errors: "Failed to copy: failed to open source object". We are OK with this. How can we continue to backup objects that are still there (the most)? Our backup process all the time fails because of such errors.

What is your rclone version (output from rclone version)

latest

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Docker image debian:stretch-slim

Which cloud storage system are you using? (eg Google Drive)

IBM Cloud Object Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config=rclone.config copyto --checksum COS_SOURCE:${bucket} COS_DESTINATION:${bucket}${BACKUP_BUCKET_SUFFIX}/${dateString}

The rclone config contents with secrets removed.

[COS_SOURCE]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = ${config.sourceCosAccessKeyId}
secret_access_key = ${config.sourceCosSecretAccessKey}
endpoint = ${config.sourceCosEndpoint}
[COS_DESTINATION]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = ${config.destinationCosAccessKeyId}
secret_access_key = ${config.destinationCosSecretAccessKey}
endpoint = ${config.destinationCosEndpoint}`

A log from the command with the -vv flag

Many errors of type

2021-01-16 17:33:42 ERROR : sys/token/id/h021e37e5dbc2ef82e5f2385f84798d2fd5a92700adbe51a77ee09b697debd645: Failed to copy: failed to open source object: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 7185d2ce-32a7-46e6-844c-ba52c9f51f59, host id:
Transferred:   	   43.040k / 1.087 MBytes, 4%, 5.183 kBytes/s, ETA 3m26s
Errors:                 1 (retrying may help)
Transferred:           73 / 1350, 5%
Elapsed time:        10.6s
Transferring:
 * sys/token/id/h00d40bbe…3f3454251f1508fa84e0f0:100% /836, 0/s, -
 * sys/token/id/h01daa12b…f61d8579afdbf58733087a:100% /836, 0/s, -
 * sys/token/id/h021088d0…c198684fa72a10dc408a17:100% /836, 0/s, -
 * sys/token/id/h021e37e5…be51a77ee09b697debd645: transferring
2021-01-16 17:33:42 INFO  : sys/token/id/h00d40bbe4924427ff25813fde0c58409bba11b66703f3454251f1508fa84e0f0: Copied (new)
Transferred:   	   43.040k / 1.086 MBytes, 4%, 5.175 kBytes/s, ETA 3m26s
Errors:                 1 (retrying may help)
...

2021-01-16 17:33:46 ERROR : sys/token/id/h50409f79892df7e1de26d0ed2bc20819a3de376c9c579b86a51b9285d348dbd7: Failed to copy: failed to open source object: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 7f37999f-d3ab-4e82-9fd6-34369ac44215, host id:
Transferred:   	  135.868k / 1.076 MBytes, 12%, 10.904 kBytes/s, ETA 1m28s
Errors:                14 (retrying may help)
Transferred:          190 / 1337, 14%
Elapsed time:        14.7s
Transferring:
 * sys/token/id/h4ce22483…17addc7fdf3a1e5a6c5520:  0% /836, 0/s, -
 * sys/token/id/h4f740972…11d0f3479b8f835be0f9f9:100% /836, 0/s, -
 * sys/token/id/h4f827fad…e2d519e810a6a87a218413:100% /836, 0/s, -
 * sys/token/id/h50409f79…579b86a51b9285d348dbd7: transferring
2

Am I understanding it that basically the issue is that you have a file, the file is about to be backed up and from the source, it's gone so you get an error since it can't find it on the source?

3

Yes. In our source, we have many temporary files (tokens)
We don't need to backup them, but we can't distinguish short time tokens from long time tokens.
If during copying some file is not found, was deleted, it's ok, it should not be copied.
We would want to ignore such type of errors.
Is it possible?

4

I'm not aware of any flag to ignore that as that's about as normal an error as they come :slight_smile:

Maybe you could try reducing the backlog of items to transfer?

      --max-backlog int                      Maximum number of objects in sync or check backlog. (default 10000)

Not sure if that would help to keep the items to copy small, but would impact performance / create more API hits.

5

That is a great idea. It will reduce the time from listing the file to transferring the file. If you make it too small it will impact performance. It will also impact the ETA times in the stats.

I think probably what you want is to set --retries 1 so rclone only tries once to do the backup, otherwise it will keep trying and keep getting a different set of errors.

6

We will try this option, thanks. We always have not more than 1000 objects, we will try to break it into chunks of 100

But it means that the action will fail in this 1 retry. There is a small chance that after retry it will take a list that won't be changed during copying. We don't change retries, it's 3 by default

7

The smaller you make it, the less time there will be between checking the object and transferring it. So you could try 10 or even 1

If there is a reasonable chance that you get a list without a failure then ignore that advice.

8

Whether should be any correlation between --max-backlog and --transfers ?
is it preferable that the backlog value be a multiple of transfers?
if I set them the same value, will this work optimally?
thanks

9

Not particularly.

That is probably a good choice if you want to make --max-backlog as small as possible.

10

I found that when I decrease the value of max-backlog backup for small buckets (with a few files) takes much more time than before, but backup for big buckets (with a lot of files) takes much less time.
When --max-backlog=1 backup of the bucket with 18 000 files takes a few seconds (!!??), but the backup of the bucket with 90 files takes 5 minutes (!!??).
Can you explain this phenomenon?

11

Can you post a log of the 90 files bucket?

12

I tried to run backup of only the small bucket and it finished in a few seconds
But when I run in parallel 20 small buckets, the last finishes in 5 min as I wrote
The big buckets run in parallel too. But in our specific case, they are not 20 but 11 buckets in parallel and the last finished in a few seconds

13

That is good

Are you maybe being rate limited by your provider?

15

We got the error even with --max-backlog=1 --transfers=1:

ERROR : Attempt 1/3 failed with 13 errors and: failed to open source object: NoSuchKey: The specified key does not exist.
ERROR : Attempt 2/3 failed with 85 errors and: failed to open source object: NoSuchKey: The specified key does not exist.
ERROR : Attempt 3/3 failed with 7 errors and: failed to open source object: NoSuchKey: The specified key does not exist.

What else can be done?
Thanks in advance

16

I think you might have to post-process the rclone log.

So do a transfer then see if there were any ERROR which weren't failed to open source if so there was a genuine error, otherwise everything is OK.

Setting --retries 1 will mean rclone just makes one, best effort pass over the data.

17

All errors are of type

 ERROR : sys/token/id/h0c6320540822dc11ded9481aa1eb5357f83fb5c92ea0bdd312b10f9f52379646: Failed to copy: failed to open source object: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 59e00cc2-3cfb-4bfc-b640-d5a78ff0be9f, host id:

I posted only the final line of every retry
I want it to retry because there is a chance that it will succeed (using default retry=3)

My questions
Why with max-backlogs=1 the error still occurs??
Can I add some other flag to avoid this type of errors?

18

It might be to do with eventual consistency of the listings. Most object storage systems are "eventually consistent" which means that the listings and the objects are allowed to get out of sync for "a while" as long as they get back in sync eventually.

I can't think of any. If you have files deleted during the transfer, then you will always get errors and cause a retry.

19

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.