I've been using https://rclone.org/docs/#max-transfer-size to upload my initial backup to Google Drive. I'm noticing, however, that any file(s) that are terminated due to the max-transfer-size will not be picked up on any subsequent runs of the sync command. So I get these folders with either a partial upload or no upload of the content, and when I re-run the sync command, rclone skips those folders.
What is your rclone version (output from rclone version )
rclone v1.50.2
os/arch: linux/amd64
go version: go1.13.4
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Unraid, Linux 64 bit
Which cloud storage system are you using? (eg Google Drive)
gsuite
The command you were trying to run (eg rclone copy /tmp remote:tmp )
Where SOURCE is a local directory and DESTINATION is an encrypted google team drive. I saw this issue happening in regular Drive, before I switched to using a service account on a team drive.
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp )
You seem to be limiting the transfer to 80MB. Is that your intent?
You'd have to post a log with -vv as if you ran even syncs, it would pick everything and keep them in sync. I'd guess the problem you are having is it may take many, m any runs depending on the size of the source to finish the initial sync with those settings and it's a bit random how it picks what to sync.
Interesting, where is rclone storing the data on the size of a file? For example, I see this log:
2019/12/29 18:36:51 DEBUG : Christmas 2019 Family Opening (2019)/Christmas 2019 Family Opening (2019).mkv: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/29 18:36:51 DEBUG : Christmas 2019 Family Opening (2019)/Christmas 2019 Family Opening (2019).mkv: Unchanged skipping
But I can confirm in the Google Drive that the file is not in there. Makes me think it's recording somewhere the file when it began the transfer, but the transfer terminated due to the max-transfer limit but that fail to upload isn't being recognized.
Interestingly.. I may be speaking too soon. Let me let this run for a bit and see. I did look via rclone command line at the folder in question and the file is in there. The Finder (Mac OS) mounted GDrive wasn't showing it. There could be lag, or I could be crazy. Let me look around before I use more of your time. Thank you.
Double check the destination - does it have a : in it? If not then you are storing it to a local directory somewhere. It sounds obvious but it is an easy mistake to make.
Alright, I let everything finish. Spot checking appears everything is in there. The repository on GDrive, however, is about 400GB smaller than my local (4.04 TB):
Total objects: 2804
Total size: 3.673 TBytes (4038502673716 Bytes)
Is that normal given encryption potentially reducing space? Though I do see that the Bytes in rclone's size command response matches the 4.04TB in my disk size locally..