Rclone log file template lines for success or skipping

What is the problem you are having with rclone?

I'm working on writing a script that will interact with the files that have been successfully copied or skipped using Rclone. To do this, I'm reading the log-file after rclone has run. Does anyone know what the output line looks like in each of those cases? Or can you point me to the file in github that I can read the template for the logs?

What is your rclone version (output from rclone version)

rclone v1.56.0

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "${SOURCE_DIR}" "${DESTINATION_DIR}" --update --use-server-modtime --fast-list --log-file=out.log --log-level DEBUG

hi,
i do the same thing with a python script.

at first, i read the log files, found the text entries i want.
then i created a set of regular expressions.

the script will read each line, try to match regular expression.
if match then do something...

with rclone, for a file copied, there are multiple text to match

for example,
2021/09/07 08:48:06 INFO : xxx.xxx: Copied (new)
2021/09/07 08:48:06 INFO : xxx.xxx: Copied (server-side copy)

note i am not an expert at reg ex and perhaps these two rules can be combined into one.
this is works well for me.
^.* INFO :.*: Copied .new.$
^.* INFO :.*: Copied .server-side copy.$

it is possible to use one rule, but might cause a false positive
^.* INFO.*:.*: Copied

as for skipped files,
2021/09/07 08:48:06 DEBUG : xxx.xxx: Unchanged skipping

Hi,
Thank you for your response! The regex is very helpful.

I looked through my log file, and for skipped files I found:

2021/09/02 16:45:30 DEBUG : xxx.txt: Destination mod time is within 1ns of source and files identical, skipping

I'm just still wondering if there is a way to look at all the possible combinations that could be outputted to the log file. I want to make the script as robust as possible and I want to account for all the possible ways the log file could say that it skipped a file or that a file was successful.

maybe someone else has an idea or can point you to the source code.

You will find many of the relevant messages here:

1 Like

Thank you! This is exactly what I was looking for.

Always numerous alternatives to achieve similar result with regex, I will just throw in two variants of combined rules:

^.* INFO : .*: Copied \(.*\)$

^.* INFO : .*: Copied \(new|server-side copy\)$

1 Like

thanks for sharing.

actually, i had forgotten i was already doing that.
i have a python script that takes a rclone debug log, strips out stuff and emails to me a rclone.short.log
^.* DEBUG :.*Waiting for (checks|transfers|deletions) to finish$

regex is not my thing,
What's too painful to remember; We simply choose to forget
https://www.youtube.com/watch?v=5m1rUGV8R3s

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.