After a bunch of testing I found as long as I'm using version v1.60.0 adding --drive-upload-cutoff 1000T fixes it
(doesn't fix it on v1.55.1, I didn't try the versions in between)
--drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8Mi)
Setting it to 1000T effectively disables the chunked uploading