Using max-transfer-size and unable to sync failed files

Hi All,

What is the problem you are having with rclone?

I've been using https://rclone.org/docs/#max-transfer-size to upload my initial backup to Google Drive. I'm noticing, however, that any file(s) that are terminated due to the max-transfer-size will not be picked up on any subsequent runs of the sync command. So I get these folders with either a partial upload or no upload of the content, and when I re-run the sync command, rclone skips those folders.

What is your rclone version (output from rclone version )

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Unraid, Linux 64 bit

Which cloud storage system are you using? (eg Google Drive)

gsuite

The command you were trying to run (eg rclone copy /tmp remote:tmp )

  rclone sync --verbose -P --stats=10m \
  --backup-dir team-google-drive-encrypted:/OLD/ \
  --no-update-modtime --fast-list --bwlimit 80M \
  --checkers=16 --transfers=8 --max-transfer 80M \
  --drive-service-account-file=$JSON_LOC/company-store.json \
  $SOURCE $DESTINATION

Where SOURCE is a local directory and DESTINATION is an encrypted google team drive. I saw this issue happening in regular Drive, before I switched to using a service account on a team drive.

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp )

No mention of the files that it is skipping.

You seem to be limiting the transfer to 80MB. Is that your intent?

You'd have to post a log with -vv as if you ran even syncs, it would pick everything and keep them in sync. I'd guess the problem you are having is it may take many, m any runs depending on the size of the source to finish the initial sync with those settings and it's a bit random how it picks what to sync.

Okay, thank you. I just updated my original post to follow more of the way others post. I will let the full sync finish and then see what happens.

Interesting, where is rclone storing the data on the size of a file? For example, I see this log:

2019/12/29 18:36:51 DEBUG : Christmas 2019 Family Opening (2019)/Christmas 2019 Family Opening (2019).mkv: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/29 18:36:51 DEBUG : Christmas 2019 Family Opening (2019)/Christmas 2019 Family Opening (2019).mkv: Unchanged skipping

But I can confirm in the Google Drive that the file is not in there. Makes me think it's recording somewhere the file when it began the transfer, but the transfer terminated due to the max-transfer limit but that fail to upload isn't being recognized.

Google doesn't partially upload things. It's either there or it isn't.

That log looks like it is there based on the output.

Do you see it locally?

What's the local size?
Can you a rclone ls on the remote and show the directory as to where it should be?

Interestingly.. I may be speaking too soon. Let me let this run for a bit and see. I did look via rclone command line at the folder in question and the file is in there. The Finder (Mac OS) mounted GDrive wasn't showing it. There could be lag, or I could be crazy. Let me look around before I use more of your time. Thank you.

Double check the destination - does it have a : in it? If not then you are storing it to a local directory somewhere. It sounds obvious but it is an easy mistake to make.

Alright, I let everything finish. Spot checking appears everything is in there. The repository on GDrive, however, is about 400GB smaller than my local (4.04 TB):

Total objects: 2804
Total size: 3.673 TBytes (4038502673716 Bytes)

Is that normal given encryption potentially reducing space? Though I do see that the Bytes in rclone's size command response matches the 4.04TB in my disk size locally..

There's no size reduction in encryption but without seeing any logs or anything, it's really just guess work.

It could be hard links on the source size or something along those lines.

You'd need to share a log of the sync command if you want to dig around it and see if something is incorrect.

1 Like

This probably looks like the difference between TiB & TB: 3.673 TiB = 4.038 TB

1 Like

Wow, the mysterious TiB explained. Found a helpful article that made this super clear and it looks like that was the answer https://searchstorage.techtarget.com/definition/tebibyte-TiB

Thank you. All is well in the world again and rclone is working nicely!

1 Like