This sounds quite similar to what I reported here:
Rclone mount repeatedly uploading the same unchanged files
One idea I can suggest is to add --read-only to mount in read-only mode, since you do not want the mount to do any uploading. This doesn't really solve the underlying issue, but it might at least prevent the main symptom for you.