Is it possible to ignore 'Object not found' errors

Help and Support
1

Hi,

We are using rclone to backup our cloud files to a local storage. We make use of the return code of rclone to determine if the backup succeeded or not in other scripts.

We noticed that it often happens that files that are being listed before it starts, no longer exist when rclone is going to process them (because the cloud storage is being used all the time, which means files can be deleted in the meantime as well).

This triggers “Failed to open: Object Not Found” errors. These errors cause rclone to return an error return code, which makes other scripts fail that depend on it.

To us, that kind of error is not really an error, and should be suppressed.

Is it possible to suppress these kind of exceptions? Or does rclone have different return codes for when errors like these happened and when really bad errors happened?

Thanks in advance!

2

Which rclone version are you using? This should be much less common with 1.37?

Are you using --retries - that is what I’d suggest, get rclone to retry until you get a clean copy or sync.

3

Hi, thanks for the swift reply.

We were using version 1.28 (an oldie), I updated it today to the latest (1.37) because I also thought that maybe a new version would fix it.

Also I did increase the --retries option from 3 default to 5.

Doing that should be enough to solve it? (besides that the actual retries number might need some trial-and-error fine tuning)

We’ll have to wait until the next backup to see if it is solved.

4

I think moving to 1.37 will make it much less likely. It could still happen, but I’m pretty sure the retries will pick it up.

5

We still see it happening, although a lot less after upgrading and increasing the maximum amount of attempts to 15, the errors are surrounded by a lot of weird HTTP exceptions:

2017/09/12 02:48:06 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:06 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:06 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:06 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:06 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:06 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:07 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:07 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:07 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:07 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>
2017/09/12 02:48:07 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h
1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>

Could these be the source of the file not found exceptions, and if so, what could be causing this or how can we try to fix it?

6

Hmm, which cloud storage provider are you using? Is it Hubic by any chance?

Any chance you could attach a log with -vv?

7

We are using Rackspace as our provider.
I will add -vv to the backup procedure and will come back to you with a more detailed log as soon as possible.

8

It seems that after increasing the amount of attempts even more it now no longer fails.

9

That is good, but kind of implies that something is not as reliable as it should be. How many attemtpts did it need?

10

14 attempts and the maximum was configured on 15, so it was very close.
Increased it to 25 now to be safe, it only happens a lot in one container (probably that one is being written a lot during the backup process).

11

Grepping the errors gives the following:

T:36:06 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:36:07 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:37:22 ERROR : Local file system at ***: not deleting files as there were IO errors
T:37:22 ERROR : Attempt 1/15 failed with 2 errors and: failed to open source object: Object Not Found
T:37:23 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:37:28 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:37:28 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:37:28 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:37:28 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:37:28 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:38:22 ERROR : Local file system at ***: not deleting files as there were IO errors
T:38:22 ERROR : Attempt 2/15 failed with 6 errors and: failed to open source object: Object Not Found
T:38:23 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:38:23 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:38:23 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:39:08 ERROR : Local file system at ***: not deleting files as there were IO errors
T:39:08 ERROR : Attempt 3/15 failed with 3 errors and: failed to open source object: Object Not Found
T:39:08 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:39:08 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:39:08 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:39:42 ERROR : Local file system at ***: not deleting files as there were IO errors
T:39:42 ERROR : Attempt 4/15 failed with 3 errors and: failed to open source object: Object Not Found
T:39:43 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:39:43 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:39:43 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:40:19 ERROR : Local file system at ***: not deleting files as there were IO errors
T:40:19 ERROR : Attempt 5/15 failed with 3 errors and: failed to open source object: Object Not Found
T:40:20 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:40:48 ERROR : Local file system at ***: not deleting files as there were IO errors
T:40:48 ERROR : Attempt 6/15 failed with 1 errors and: failed to open source object: Object Not Found
T:40:49 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:40:49 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:40:49 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:40:49 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:11 ERROR : Local file system at ***: not deleting files as there were IO errors
T:41:11 ERROR : Attempt 7/15 failed with 4 errors and: failed to open source object: Object Not Found
T:41:12 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:12 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:12 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:34 ERROR : Local file system at ***: not deleting files as there were IO errors
T:41:34 ERROR : Attempt 8/15 failed with 3 errors and: failed to open source object: Object Not Found
T:41:35 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:35 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:35 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:35 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:35 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:41:56 ERROR : Local file system at ***: not deleting files as there were IO errors
T:41:56 ERROR : Attempt 9/15 failed with 5 errors and: failed to open source object: Object Not Found
T:41:57 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:42:17 ERROR : Local file system at ***: not deleting files as there were IO errors
T:42:17 ERROR : Attempt 10/15 failed with 1 errors and: failed to open source object: Object Not Found
T:42:17 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:42:17 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:42:38 ERROR : Local file system at ***: not deleting files as there were IO errors
T:42:38 ERROR : Attempt 11/15 failed with 2 errors and: failed to open source object: Object Not Found
T:42:39 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:42:39 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:42:58 ERROR : Local file system at ***: not deleting files as there were IO errors
T:42:58 ERROR : Attempt 12/15 failed with 2 errors and: failed to open source object: Object Not Found
T:42:59 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:42:59 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:42:59 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T:43:19 ERROR : Local file system at ***: not deleting files as there were IO errors
T:43:19 ERROR : Attempt 13/15 failed with 3 errors and: failed to open source object: Object Not Found
T:43:42 ERROR : Attempt 14/15 succeeded
T+1:25:39 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T+1:26:14 ERROR : Local file system at ***: not deleting files as there were IO errors
T+1:26:14 ERROR : Attempt 1/15 failed with 1 errors and: failed to open source object: Object Not Found
T+1:26:15 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T+1:26:41 ERROR : Local file system at ***: not deleting files as there were IO errors
T+1:26:41 ERROR : Attempt 2/15 failed with 1 errors and: failed to open source object: Object Not Found
T+1:26:42 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T+1:27:03 ERROR : Local file system at ***: not deleting files as there were IO errors
T+1:27:03 ERROR : Attempt 3/15 failed with 1 errors and: failed to open source object: Object Not Found
T+1:27:04 ERROR : ***: Failed to copy: failed to open source object: Object Not Found
T+1:27:16 ERROR : Local file system at ***: not deleting files as there were IO errors
T+1:27:16 ERROR : Attempt 4/15 failed with 1 errors and: failed to open source object: Object Not Found
T+1:27:28 ERROR : Attempt 5/15 succeeded
12

OK that doesn’t look too bad! I hope that works OK.

13

It is still happening, even with the max amount of attempts set to 50.
It fails after a while with:

2017/09/18 02:26:14 ERROR : Attempt 50/50 failed with 6 errors and: failed to open source object: Object Not Found

14

This must surely be a problem with your provider?