rclone needs to download a complete directory listing with meta data (date, size, hash,...) every time you do a sync to check the directory against a similar list generated for your local folder.
You can get an impression of the size and data by performing this command:
rclone lsjson --recursive --hash Scaleway: > Dir_Scaleway.txt
and then checking the size and content of Dir_Scaleway.txt (using e.g. Windows Explorer and Notepad)
There will typically be some overhead to transfer data, so I would guess you see a file with a size around 20MB (3GB/4/24/1.5) equivalent to 200 Bytes(characters) per directory entry (20MB/100,000). That sounds as expected to me, especially because you are using (standard) encryption which increases the length of paths and filenames (as you can see in Dir_Scaleway.txt).
The simplest way to decrease the downloads is to reduce the sync frequency, as an example you could reduce directory downloads by a factor 4 by just syncing once every hour.
You may also want to consider the more advanced approach using top-up syncs every 15 minutes and a full sync every 24 hours, the approach is described in these posts:
Copy to S3 incrementally - #2 by ncw
--no-traverse too slow with lot of files - #2 by ncw