This is a known issue
Which I haven't got round to fixing yet (mostly because I can't figure out how!)
You can work around it I think by creating a duplicate drive, one with the shared_with_me = true parameter, then when you want to copy between them copy from shared_drive to normal_drive and you won't get the duplicate. You'll need to set --drive-server-side-across-configs
Perhaps a better solution would be for --drive-shared-with-me to show all the shared files and all of you own files?