It sounds like you are being throttled by OneDrive. This typically happens when you suddenly increase your activity - or do extremely large up/downloads.
You can check by looking after "pacer" entries when starting with the -vv flag. The pacer basicly adjusts the speed based on the feedback/responses it gets from OneDrive. You can see an (extreme) example here:
2021/06/26 14:03:00 DEBUG : Too many requests. Trying again in 120296 seconds.
2021/06/26 14:03:00 DEBUG : pacer: low level retry 1/10 (error accessDenied: throttledRequest: Too Many Requests)
2021/06/26 14:03:00 DEBUG : pacer: Rate limited, increasing sleep to 33h24m56s
2021/06/26 14:03:00 DEBUG : pacer: Reducing sleep to 25h3m42s
2021/06/26 14:03:00 DEBUG : pacer: Reducing sleep to 18h47m46.5s
2021/06/26 14:03:00 DEBUG : pacer: Reducing sleep to 14h5m49.875s