Transfers: Suggested Change in RC API

bdkjones · May 28, 2022, 5:47am

Currently, when using the rclone rc core/transferred API, rclone returns the last 100 completed transfers.

This is not a particularly useful approach for an API. What I'd really like to know is: "Which transfers have completed since I last polled rclone?" This is far more useful because:

I don't have to worry about setting a polling interval too short that I'm wasting CPU or too long that I "drop" transfers because rclone exceeds the limit.
I don't have to constantly de-dupe the results to ignore transfers I already saw the last time I called the API.
rclone won't run out of RAM because it will still prune completed transfers after each call I make—the new approach could be placed behind a flag so that the user knows they're opting-into a contract: "call to get transfer information periodically or rclone will eventually seize on large operations."

bdkjones · May 28, 2022, 5:50am

I do see where pruneTransfers() is implemented and where the 100 limit is set, but I can't figure out where the core/transferred RC command is handled to actually get the completed transfers. (I was attempting to put together a PR for the feature.)

Animosity022 · May 29, 2022, 7:37pm

Feel free to submit a PR.

Ole · May 30, 2022, 8:23am

You may be looking for this:

github.com

rclone/rclone/blob/7361c98b2d15744f820668333a0bfc803c440e2a/fs/accounting/stats_groups.go#L127-L182

      
        
            func rcTransferredStats(ctx context.Context, in rc.Params) (rc.Params, error) {
            	// Check to see if we should filter by group.
            	group, err := in.GetString("group")
            	if rc.NotErrParamNotFound(err) {
            		return rc.Params{}, err
            	}
            
            
	out := make(rc.Params)
            	if group != "" {
            		out["transferred"] = StatsGroup(ctx, group).Transferred()
            	} else {
            		out["transferred"] = groups.sum(ctx).Transferred()
            	}
            
            
	return out, nil
            }
            
            
func init() {
            	rc.Add(rc.Call{
            		Path:  "core/transferred",

This file has been truncated. show original

I found it by searching for “core/transferred” in VS Code with “files to include” set to “**/*.go”.

Hoping it is the right place, I haven’t verified/debugged.

system · July 29, 2022, 8:24am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.