I am trying to pull down two large files from Drive on behalf of a couple users in my G Suite domain. They are about 700GB and 850GB respectively, and I should emphasize each one is a single file, not a directory.
Nonetheless, I am now using trusty rclone to retrieve the files and copy them to a mounted network folder. I was trying to copy both files at once with two separate commands as they are in very divergent directories, but both copy jobs have given the same error for the two massive individual files:
Removing partially written file on error: unexpected EOF
At first I was like oh well, that's a fluke. Then after making another 100GB+ of progress, it happened again.
How can I mitigate this? I am fine with doing one file at a time if that helps, or including any necessary options for the command, whether it's copy or sync.
I personally wouldn't copy to a mount to move data and would use rclone copy/move. Even more so if they are larger files.
You'd need to see the debug info around that message to get more information as from that message, it seems it stopped writing and not sure you can tell much else.
Appreciate the advice! I am using rclone copy if that wasn't clear in the OP, my bad. Could it be because the destination is the mount and not on the local FS? I am looking to avoid using massive space on the headless system where rclone is running even if it's temporary.
The logs gave me nothing else, but I think it may have been related to some unnecessary bandwidth limiting options that stalled the transfer long enough for a connection issue of some kind. I just re-ran one of the file copies with unrestricted bandwidth and it downloaded the 700GB file in approximately an hour!
Unfortunately, now the larger file is throwing another classic error:
2019/06/26 14:52:00 Failed to copy: failed to open source object: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
I forget what the M.O. for that is, but it sounds like I need to let it cool off for 24hrs, right?
No worries! Sorry I am a bit all over the place today so did not clarify.
I am rclone copy -ing from GDrive to a local destination (mounted folder on an internally-networked file server). I know there is a 750GB daily upload limit, but I am downloading from Drive so did not anticipate hitting any quotas. I vaguely recall a 10TB daily download limit but that was for accounts, not individual files -- and this isn't 10TB.
My command is pretty vanilla, but is basically as follows:
Update: the faster bandwidth outpaced what I assume were network changes that caused the EOF. Managed to move both files in a matter of a few hours all told. All is well.