Apparently this issue was already reported back in October 2018
To quote ncw himself:
The --drive-shared-with-me hides the non shared with me stuff so rclone doesn’t see the existing folder and creates a duplicate
Which BTW is what was puzzling me in the OP.
Until it is fixed, a workaround seems to be adding the shared folder to your drive and copy from there. This will avoid using the flag --drive-shared-with-me, which is what is causing the issue.