OK, I've had a chance to look at this. I tried both suggestions. Not exactly as they were typed in the comments suggesting them, but I think good enough to get their intents.
First, rclone copy ./blah Proton:blah --no-traverse --max-age 37d -P --immutable --combined=~/combined.txt --log-level=DEBUG --log-file=~/log.txt:
The summary for rclone of this command said 93,585 listed, with 2,068 checks and zero transfers.
combined.txthas exactly one line per "check" and no other lines at all. All of them indicate that the "path was found in source and destination and was identical"
log.txt had 190,803 lines, those being:
- 182,512 saying things were excluded
- 8,272 saying things were being skipped
- 19 miscellaneous startup/shutdown messages
Exactly half of the 182,512 (i.e. 91,256) "excluded" messages ended with "Excluded (ModTime Filter)". The other (exact) half ended with just "Excluded".
From a brief look at a few small portions of the file, it looks like those two messages were probably always paired with each other - i.e. there were 91,256 things that were "excluded", and each of those things produced an "Excluded (ModTime Filter)" message immediately followed by an "Excluded" message.
Similarly, the 8,272 "skipped" messages were of four types, each comprising exactly a quarter of them. Note that's 2,068, i.e. the same as "checked". The four types were:
- "Size and modification time the same"
- "size = [0-9]+ OK"
- "Unchanged skipping"
- "Sync Logger : Match: ="
I was guessing that the "excluded" count would line up with "listed minus checks". It does not, but it's roughly close (i.e. 93,585 - 2,068 = 91,517 > 91,256)". It's short by 261. Looking a little closer, I see that 261 is exactly the number of subdirectories involved.
So, again, I don't really know what I'm talking about, but from this first command, it seems like rclone is at the very least correctly:
- distinguishing between those files older than
--max-ageand those younger, and - determining that all of those younger are the same on my local drive and on Proton.
That doesn't necessarily imply anything about the API requests, and I didn't see any additional information resulting from this command that might do so. However, the other suggested command does:
rclone copy ./blah Proton:blah --no-traverse --max-age 37d -P --immutable -vv --dump=headers > ~/vvout 2> ~/vverr
The rclone summary has the same numbers as the previous command (i.e. 93,585 listed, with 2,068 checks and zero transfers).
The vvout file is completely empty. The vverr file has a whole bunch of lines (1,477,772). Most of these are individual lines from the bodies of individual HTTP requests or replies.
There were 12,615 "HTTP REQUEST" lines and 12,615 "HTTP RESPONSE" lines.
The 12,615 "HTTP REQUEST" lines were associated with 10,286 GET and 2,329 POST requests.
The 12,615 "HTTP RESPONSE" lines were all associated with "200 OK" responses.
For the GET requests, I noticed that virtually all of them (all but five) were either for something containing the string "folders" or something containing the string "links" (and, to be clear, there were none at all that were for something containing both "folders" and "links"). The five that had neither "folders" nor "links" were for what look to me like higher-level things in the organizational hierarchy:
- /core/v4/users
- /core/v4/addresses
- /drive/volumes
- /drive/shares/ followed by something that looks like a Base64-encoded string, which I'm assuming it might not be wise of me to share here
- /drive/shares?ShowAll=1
Note that the number of POST requests (i.e. 2,329) is exactly equal to 2,068 + 261 (i.e. the number of checks plus the number of subdirectories).
I did not look at any of the request/reply stuff in more detail, as I don't know what I'm looking for. If anyone wants more detail, please let me know what you want me to search for, preferably as specifically as possible.
So, again with the caveat that I don't know what I'm talking about, it seems to me like nothing here rules out the possibility that rclone is asking Proton to do something that causes it to look up metadata for the full ~100,000 files, but with that said, it's definitely not making 100,000 API requests.
Finally, just a little note about what I now intend to do about all this when I next get a chance (hopefully pretty soon):
In my personal usage, the files I put on Proton using rclone are intended to be uploaded once and thereafter never changed or deleted. So, in the hopes that uploading an entirely new file will not cause Proton to go wild looking for metadata on a bazillion things, I intend to make a script to:
- Build the equivalent of the
--max-age-filtered file list from the shell rather than from rclone; - Tell rclone to upload the things specifically on the list, with nothing else, not bothering to check anything like if they're already present on Proton;
- Remember the age of the newest thing thus uploaded, for use the next time I want to build the equivalent of the
--max-age-filtered file list.
Again assuming that this will not cause metadata and/or API-rate issues on Proton's end that might lead to me being considered "abusive", then it seems to me like I should be able to do this frequently (several times a day or whatever) without worry.
And then, I don't know, maybe like once a week or so, do a full rclone copy (like the ones I do several times a day now) to make sure everything's really up there, and really correct.