While exploring the RC API (especially via
librclone), I've seen @ncw mention several times that
librclone really needs a way to stream results back to a caller on-demand.
Would this approach be useful?
rclone RC accepts a
true, rclone stores a running list of every log message that it WOULD have written to
stderrfor a given operation (copy, sync, etc.)
A new API endpoint gets added:
core/log. This would function like
core/stats; it simply returns all the log messages that have been cached up to that point. Then, it clears the cache.
core/reset-logendpoint could be added to clear the log message cache without sending the messages. And perhaps a
--rc-cache-log-limitparameter to specify the maximum number of cached entries with a reasonable default, such as 1,000.
The logging system already supports JSON output, which is what the RC API requires. So the return of
core/logwould be an array of JSON objects, each of which is a log message exactly as it would have appeared in a Terminal when
The on-demand nature of the
core/logoperation ensures that rclone does not waste time/resources broadcasting messages when the client doesn't care (maybe the UI isn't on screen, so we don't need to update a live list of log messages; we can delay getting them until the app is in the foreground again—an important consideration because 90% of a backup app's life is spent running in the background with no UI.)
Because the behavior is opt-in, there is no effect on existing users or configs.
To be useful, any operation launched via
librclonewould have to be
_async. But that's recommended anyway.
Memory, obviously. Caching the log messages requires RAM. I don't know how quickly the footprint would grow because I don't know how much overhead is involved in the logging system. I also realize Go is a garbage-collected language and if, say, 10,000 messages all get marked for collection at once, I don't know what that performance hit is—I'm not very familiar with Go.
Function Pointers. It looks like Go supports them, so I think the
func RPC(method string, input string) (output string, status int)API in
librclonecould be expanded to accept a pointer to a function to be called with log output. But you give up the "on-demand" advantage and use resources at all times. If the client developer is dumb, you could also have log-handler functions that lag and tie up rclone in the process. You keep the memory footprint low, though.
Websockets. If the
RPC(...)method accepted a websocket address, you could chuck all the output to that. There's some security concerns, but I suppose you could upgrade to
wss://to solve that. Websockets would be pretty reliable if both ends are on the local machine; but this approach involves more overhead and developer work than function pointers or the original proposal. (Websockets are a pain to work with from C and
librcloneis quite pleasant in C currently.)
Anyone have thoughts on this before I take a swing in one direction or another?