How to ONLY print files successfully copied or errors?

driib · July 16, 2023, 5:38pm

What is the problem you are having with rclone?

I want to get cron emails from rclone when files were copied or when an error occurred. No emails otherwise. I found How to print a list of files synced/copied? but it's not enough because I get extra output printed:

$ rclone --transfers 3 --tpslimit 12 --stats-one-line -v move ...
2023/07/16 19:26:56 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1
2023/07/16 19:26:59 INFO  : There was nothing to transfer
2023/07/16 19:26:59 INFO  :           0 B / 0 B, -, 0 B/s, ETA -
2023/07/16 19:26:59 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1
2023/07/16 19:27:02 INFO  : There was nothing to transfer
2023/07/16 19:27:02 INFO  :           0 B / 0 B, -, 0 B/s, ETA -
2023/07/16 19:27:02 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1
2023/07/16 19:27:06 INFO  : There was nothing to transfer
2023/07/16 19:27:06 INFO  :           0 B / 0 B, -, 0 B/s, ETA -
2023/07/16 19:27:06 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1
2023/07/16 19:27:09 INFO  : There was nothing to transfer
2023/07/16 19:27:09 INFO  :           0 B / 0 B, -, 0 B/s, ETA -
2023/07/16 19:27:09 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1
2023/07/16 19:27:13 INFO  : There was nothing to transfer
2023/07/16 19:27:13 INFO  :           0 B / 0 B, -, 0 B/s, ETA -
2023/07/16 19:27:14 INFO  : Starting transaction limiter: max 12 transactions/s with burst 1
2023/07/16 19:27:22 INFO  : x$d1828-Playing With Prolog-Guest Lecture for Intro to SWI-Prolog by Dr. Jan W-5raU9tHeKzo.m4a: Copied (new)
2023/07/16 19:27:22 INFO  : x$d1828-Playing With Prolog-Guest Lecture for Intro to SWI-Prolog by Dr. Jan W-5raU9tHeKzo.m4a: Deleted
2023/07/16 19:27:22 INFO  :    54.371 MiB / 54.371 MiB, 100%, 7.766 MiB/s, ETA 0s

What I want is the output of the above command but as if 2>&1 | grep -v -e "Starting transaction limiter" -e "There was nothing to transfer" -e "0 B / 0 B, -, 0 B/s, ETA -" was applied (without having to use grep and most importantly, to mess with stderr redirection):

$ rclone-archive.sh 2>&1 | grep -v -e "Starting transaction limiter" -e "There was nothing to transfer" -e "0 B / 0 B, -, 0 B/s, ETA -"
# no output, cron does not email
$ rclone-archive.sh 2>&1 | grep -v -e "Starting transaction limiter" -e "There was nothing to transfer" -e "0 B / 0 B, -, 0 B/s, ETA -"
2023/07/16 19:28:27 INFO  : x$d2111-TwoSetViolin-Ling Ling Workout Ft. @ChloeChuaviolinist-blHEB0iYoIo.m4a: Copied (new)
2023/07/16 19:28:27 INFO  : x$d2111-TwoSetViolin-Ling Ling Workout Ft. @ChloeChuaviolinist-blHEB0iYoIo.m4a: Deleted
2023/07/16 19:28:27 INFO  :    15.977 MiB / 15.977 MiB, 100%, 5.312 MiB/s, ETA 0s

Is it possible? Thank you in advance!

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-76-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Should not matter.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

See above.

The rclone config contents with secrets removed.

Should not matter, only remote secrets are there.

A log from the command with the `-vv` flag

Should not matter

kapitainsky · July 16, 2023, 5:42pm

rclone does not send any emails. it generates logs you have to process yourself and apply whatever logic needed to trigger some action e.g. email.

driib · July 16, 2023, 5:44pm

I apologize for not being clear. I run a script (that invokes rclone) as a cron job. I am looking for flags to rclone to reduce the output to only important notices without having to process the log by hand so that cron only emails me when rclone moved some files or failed.

kapitainsky · July 16, 2023, 5:51pm

I get you - rclone logs output at the moment is not super scripts friendly (it is more human readable)... you can change debug level (--log-level) but still then you have to do your shell kung-fu to get out of this what you need.

I think you could propose new better logs output structure or even implement it and post PR. Many people would be happy:)

Animosity022 · July 16, 2023, 5:57pm

You can use:

      --use-json-log                                Use json log format

and parse out the logs for the details you want. The logs generally include 'more' than your use case.

driib · July 16, 2023, 6:06pm

Thank you @Animosity022, I will see what I can whip up with the JSON output and jq. For now, I have this in my scripts:

rclone_q() {
   rclone $OPTS "$@" 2>&1 | { grep -v -e "Starting transaction limiter" -e "There was nothing to transfer" -e "0 B / 0 B, -, 0 B/s, ETA -" || true ; }
}

And if --use-json-log + jq don't help much either, I will use your advice @kapitainsky and see what I can suggest. Though it would be the first time for me writing Go

Thanks for the quick help!

kapitainsky · July 16, 2023, 6:08pm

Of course it would be perfect world if you come with working solution but good and clever ideas are also super welcomed.

I agree with you that logs at the moment are not easy to use for automated parsing.

driib · July 16, 2023, 6:40pm

All right, I am ~~truly lost~~ a tiny bit confused with the JSON output:

For some reason, JSON is piped into stderr. The idea with stderr is to put there all messages that are not valid output (in this case, parseable JSON).
I seem to get no output without dry-run. I get most of the output back by using -v but its JSON has a different structure than the --dry-run JSON output, which makes it impossible to debug the jq command without doing non-dry run.

Here is a minimally usable JSON wrapper function:

rclone_q() {
   rclone -v --use-json-log "$@" 2>&1 | jq -r 'select(.msg == "Deleted" or .msg == "Copied (new)") | "\(.object) \(.msg)"'
}

Compared to the grep, I am not sure how much simpler it is. Also, I am a bit more worried about error handling (nearly completely relying on set -euo pipefail here). Perhaps, select(.msg == "Deleted" or .msg == "Copied (new)" or .level == "error") could help.

I will check the rclone source code when I get some free time and see what I can suggest. But I think --use-json-log should certainly write JSON to stdout.

kapitainsky · July 16, 2023, 8:12pm

This is definitely area where rclone can become better - there is more and more people using it for serious data processing. For long it was films//plex/data hoarders tool of choice and a lot of implemented features were impromptu response to this audience requirement. There is nothing wrong with this as it is open source project and should serve everybody. But there is definitely need to make it more friendly to a bit more complicated jobs. And here rclone does not have to be something what it is not. What only is needed is more structured approach e.g. to logging.

driib · July 17, 2023, 2:39pm

Thank you, @kapitainsky. Both for quick replies and your work on this great tool! Indeed, as you say, the tool itself does the job perfectly. I will see how can I contribute to better logging.

system · August 16, 2023, 2:40pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.