I'm using rclone to download files via an http remote, and when a particular file throws a 502 error (relatively common), it doesn't do any low-level retries or fail in any way. This can cause files to be deleted when running an rsync operation. It does this for some remotes (e.g. 1Fichier), but I couldn't find any errors it'll automatically retry on in the HTTP code
Run the command 'rclone version' and share the full output of the command.
rclone v1.66.0
os/version: Microsoft Windows 11 Home 23H2 (64 bit)
os/kernel: 10.0.22631.3593 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.22.1
go/linking: static
go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
HTTP
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Thanks for the response. I'm not sure what you mean by other copy tools? I tried rclone copy rather than rclone sync and ran into the same things. wget works but isn't granular enough for the kind of sync I'm trying to do here, which does a lot of filtering. It also occasionally fails on the downloads as well. I've noticed that other remote types have error messages that they'll automatically retry on, I think the HTTP remote should be able to do the same thing as well?
I agree, the server is pretty flaky but I thought rclone had some features that would help that out. You're also right about the --fast-list, I've got it in there for completeness but it doesn't actually do anything. The full log looks much the same:
INFO: 2024/05/22 16:52:32 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "sync" "--transfers=5" "remote_http:path" out_dir" "-vv" "--filter" "+ {{(file1.zip|file2.zip)}}" "--filter" "- *"]
INFO: 2024/05/22 16:52:32 DEBUG : Creating backend with remote "remote_http:path"
INFO: 2024/05/22 16:52:32 DEBUG : Using config file from "XXX\\rclone\\rclone.conf"
INFO: 2024/05/22 16:52:32 DEBUG : Root: remote_http:path
INFO: 2024/05/22 16:52:32 DEBUG : Creating backend with remote "out_dir"
INFO: 2024/05/22 16:52:32 DEBUG : fs cache: renaming cache item "out_dir" to be canonical "//?/out_dir"
INFO: 2024/05/22 16:52:36 DEBUG : fileX.zip: skipping because of error: failed to stat: HTTP Error: 502 Bad Gateway
[Way more 502s here, inconsistent from run to run]
INFO: 2024/05/22 16:53:26 DEBUG : file1.zip: Size and modification time the same (differ by 0s, within tolerance 1s)
INFO: 2024/05/22 16:53:26 DEBUG : file1.zip: Unchanged skipping
INFO: 2024/05/22 16:53:26 DEBUG : Local file system at //?/out_dir: Waiting for transfers to finish
INFO: 2024/05/22 16:53:26 DEBUG : Waiting for deletions to finish
INFO: 2024/05/22 16:53:26 INFO : There was nothing to transfer
INFO: 2024/05/22 16:53:26 INFO :
INFO: Transferred: 0 B / 0 B, -, 0 B/s, ETA -
INFO: Checks: 1/1, 100%
INFO: Elapsed time: 54.2s
INFO:
INFO: 2024/05/22 16:53:26 DEBUG : 3 go routines active
the real issue is if the 502 happens on the file that's in the include filter, it won't get transferred at the end, but without -vv it won't even show that that's because of this 502 error
i do not use http remotes much, so sure why rclone is not doing retries.
maybe it is not supported for http remotes.
DEBUG : fileX.zip: skipping because of error: failed to stat: HTTP Error: 502 Bad Gateway
i would think rclone should output ERROR, not DEBUG
maybe that is the reason, rclone is not retrying?
I agree, I was poking in the code a little earlier and see the HTTP remotes have no codes they'll retry on, which I think they should have? That or the debug should be raised to an error, one or the other. Unless the HTTP protocol doesn't support that? Not an expert here