STOP and READ USE THIS TEMPLATE NO EXCEPTIONS
I promise I have read the template, my question is for advice on parsing the Rclone outputs rather than an issue running particular code. Currently we save the stderr and stdout from rclone copy to a text file where it is formatted very nicely. We also want to specifically catch errors and show these elsewhere.
We have added the --use-json-log to the rclone copy command and am parsing with the python code below. Essentially this reformats the json output back to the string output, and captures (for the stdout only, called with capture_errors) the errors in a list.
My main question I guess is whether this will indeed capture all relevant errors from the stdout, or there is some case I am missing. And more generally, if there are any failure modes I’ve not considered. I’ve done a lot of testing but maybe there are some connection errors types or json content I’ve not covered. I don’t know Go but happy to look through the relevant parts of the RClone code if anyone would link, and apologies if I have missed this in the docs.
def reformat_rclone_copy_output(
stream: bytes, capture_errors: bool = False
) -> tuple[str, list[str]]:
""""""
split_stream = stream.decode("utf-8").split("\n")
errors = []
for idx, line in enumerate(split_stream):
try:
line_json = json.loads(line)
except json.JSONDecodeError:
continue
if capture_errors:
if line_json["level"] == "error":
if "object" in line_json:
errors.append(
f"The file {line_json['object']} failed to transfer. Reason: {line_json['msg']}"
)
else:
errors.append(f"ERROR : {line_json['msg']}")
split_stream[idx] = (
f"{line_json['time'][:19]} {line_json['level'].upper()} : {line_json['msg']}"
)
format_stream = "\n".join(split_stream)
return format_stream, errors