Transfer stats on non-0 exit

Question / Suggestion -

I do a couple of sync-jobs between providers and accounts throughout the day and generally rely on this post-transfer set of stats which is logged after each run:

Transferred:       26.810G / 26.810 GBytes, 100%, 8.007 MBytes/s, ETA 0s
Checks:            935334 / 935334, 100%
Deleted:                3
Transferred:          215 / 215, 100%
Elapsed time:      57m8.7s

After each such run I parse and add the last transfer's use together so I have an idea of what amount of upload I have used in a 24h period (for gdrive specifically but also for my monitoring system for upload / # files copied and deleted so I have an idea of the lay of the land).

The introduction of the --drive-stop-on-upload-limit and -max-duration commands are of course welcomed in this scenario, however I notice that a non-zero exit code (as you will run into when using these flags) ensures that the above transfer stats are not logged on exit.

Therefore,

  • Is it possible / would it be possible to log these transfer stats on non-zero exits?
  • Longer term, -max-[upload|download]-per-period=<time>,<size> would be super - possibly maintain state in rclone.conf but sounds like a nightmare to implement (pairs of providers, etc).

Thoughts?

Thanks!

Possible to implement? sure.

Available now? I am almost certain the answer is no. Not unless you make your own script to parse the logs to extract the information from log-files.

@ncw I guess you could make a note of this request.

Sorry if I was not clear @thestigma

The scripts I use that does the parsing, etc is already in place and fine. It's merely that a non-zero exit code does not log the transfer stats, whilst a zero-exit does.

That seems like a much easier change then..
I don't suppose it works if you force an exit 0 (or equivalent) at the script exit?

If not then we need a code-change in rclone, but that sounds like a fairly small and doable thing, so NCW can probably help you in this.

The exit code of a shell script does not change the behaviour of rclone - therefore it is a rclone change yes.

Hopefully NCW can help out :slight_smile:

That sounds like a sensible request... I had a look at that bit of code and it was a bit of a mess so I've tidied it up - it will always output stats before ending now.

What do you think?

https://beta.rclone.org/branch/v1.51.0-009-g6cb5a5fa-fix-exit-stats-beta/ (uploaded in 15-30 mins)

Well there is always --bwlimit which if you set it correctly you don't need to maintain state!

But, yes, that sounds quite hard.

Thanks @ncw I'll give that beta a spin as it'll solve my transfer accounting issue

I appreciate --bwlimit however that assumes the quantity to copy/move/sync is quite static and not very variable (size and bandwidth wise) and can therefore get a bit messy in other scenarios.

The use of --drive-stop-on-upload-limit and the parsing I'm already doing of the transfer stats hopefully now even on failure solves that dilemma, thank you!

1 Like

Just to report back @ncw - the beta works perfectly and reports stats on a non-zero exit code as expected now.

Thanks a ton!

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52

1 Like