[webdav] [fastweb] ignore 504 Gateway Time-out


#1

After make it works with fastweb[1] a problem arise (it happened also with curl)

If upload time is higher than 1 minute you get a 504 Gateway Time-out
Fact is file is uploaded correctly.
Is there a way to ignore this errore and have rclone doing is stuff.

DEBUG : rclone: Version "v1.45-072-g0ec6dd9f-beta" starting with parameters ["./rclone" "-vv" "copy" "FILE.mkv" "sec:Anime/"]
...
DEBUG : pacer: Rate limited, increasing sleep to 342.432864ms
DEBUG : pacer: low level retry 1/1 (error 504 Gateway Time-out)
DEBUG : pacer: Reducing sleep to 256.824648ms
ERROR : FILE.mkv: Failed to copy: 504 Gateway Time-out
ERROR : Attempt 3/3 failed with 1 errors and: 504 Gateway Time-out

maybe a flag that skip this check?

thanks

[1] [nextcloud] can't connect because webview (saml)


#2

Hi I build from source with some minor edit to webdav.go

 // retryErrorCodes is a slice of error codes that we will retry
 156 var retryErrorCodes = []int{
 157     429, // Too Many Requests.
 158     500, // Internal Server Error
 159     502, // Bad Gateway
 160     503, // Service Unavailable
 161     //504, // Gateway Timeout
 162     509, // Bandwidth Limit Exceeded
 163 }
 164 
 165 // shouldRetry returns a boolean as to whether this resp and err
 166 // deserve to be retried.  It returns the err as a convenience
 167 func shouldRetry(resp *http.Response, err error) (bool, error) {
 168         if resp.StatusCode == 504 {                                                                                                                                                                                                                                                                                    
 169         err = nil
 170         }
 171     return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
 172 }

and after a build:

$./rclone --low-level-retries 1 --retries 1 -vv copy -P FILE.mkv sec:Anime/
2019/01/10 15:07:07 DEBUG : rclone: Version "v1.45-DEV" starting with parameters ["./rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "-P" "FILE.mkv" "sec:Anime/"]
2019/01/10 15:07:07 DEBUG : Using config file from "/home/ME/.config/rclone/rclone.conf"
2019-01-10 15:07:08 DEBUG : FILE.mkv: Couldn't find file - need to transfer
2019-01-10 15:19:35 INFO  : FILE.mkv: Copied (new)
Transferred:   	    7.424G / 7.424 GBytes, 100%, 10.166 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:    12m27.7s
2019/01/10 15:19:35 INFO  : 
Transferred:   	    7.424G / 7.424 GBytes, 100%, 10.166 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:    12m27.7s

2019/01/10 15:19:35 DEBUG : 4 go routines active
2019/01/10 15:19:35 DEBUG : rclone: Version "v1.45-DEV" finishing with parameters ["./rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "-P" "FILE.mkv" "sec:Anime/"]

Now I have same questions:

It did works? Is there a way to perform a check without downloading the remote file and md5sum both files?

What do you think about a
--ignore-http-statuscode list_of_status_code_comma_separated
flag?


#3

That gateway timeout is being sent by the remote server. Unfortunately there is nothing rclone can do to fix it :frowning:


#4

well, you can ignore it (see my dirty workaround in second post).
Do you think it will be worth add a --ignore-http-statuscode flag?


#5

Ok, I think you can implement something like this:
In every backend where shouldRetry and retryErrorCodes is defined
you should replace shouldRetry with this:

func shouldRetry(resp *http.Response, err error) (bool, error) {
    if resp != nil {
            for _, i := range IgnoreStatusCode {
                    if resp.StatusCode == i {
                            return false, nil
                    }
            } 
    }
	return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}

Now we need to populate IgnoreStatusCode from the command line --ignore-http-status-code 504,505

Then if I understood cobra a little (I didn’t). you should add somewhere:
RootCmd.PersistentFlags().String("ignorehttp","","Ignore http status code and keep going")

and then

IgnoreHttpStatusCode := cmd.Flag("ignorehttp").Value.String()
tmp := strings.Split(IgnoreHttpStatusCode, ",")
var IgnoreStatusCode []int{}
for _, i in range(tmp) {
	statuscode, err := strconv.Atoi(i)
 	if err != nil {
		MEGAERROR_STOPEXECUTION()
	}
	IgnoreStatusCode = append(IgnoreStatusCode ,statuscode)
}

#6

Ah I see what you mean by ignoring the 504 now! You mean treat the 504 as success.

Let me see if I understand

  1. you upload a large file
  2. the upload takes longer than 1 minute - say 10 minutes
  3. after 10 minutes the file is uploaded correctly
  4. but the return code is 504 not 20x

Is that correct?

Webdav doesn’t provide a content hash (or not a standard one anyway). So rclone will be left checking the size of the file is OK after the upload.

The way to do that would be to add some backend configuration to the webdav backend. This will automatically make a --webdav-XXX flag, eg https://github.com/ncw/rclone/blob/master/backend/webdav/webdav.go#L89 These automatically get put into the Options struct just below

This could be made advanced config to hide it normally.

Do you really need a list of codes? Have you seen one other than 504?

Want to send a pull request?


#7

yes.

so a simple

{
	Name:       "ignore504",
	Help:       "treat http status 504 as success.",
},

is enough? Onestly I am lost here.

nope, only seen 504 on this server, but because you use retryErrorCodes on other backends I thought maybe a global flag to disable some of them could be useful.
Either way I will try to write something in golang.


#8

Yes, lets try that to start with.

Excellent!


#9

Hi,

I’m not sure, but it seems that after 504 the modtime is not updated with the original modtime.
I’m performing some experiment using moveto (to a Crypt over Webdav) and for large files I’m experiencing this issue. Not sure the root cause.

ciao


#10

Ah… that will be something to do with the server I think.

It is some kind of timeout - maybe you can increase the timeouts in the webdav server?


#11

I cannot change any Webdav server configuration. It’s provided by my ISP.

ciao


#12

OK. I suspect that it is time based before getting the 504.

I don’t think I can fix this in rclone :frowning:


#13

Can you add a “touch -t xxx” at end of copy/move? A flag like “–force-modification” could be useful.