Google Drive Picture Upload Starts fast(ish) then immediately slows

Help and Support
1

What is the problem you are having with rclone?

I'm uploading thousands of images (~220Gb) to my google drive (unlimited storage account) with my own ID and Secret. I've tried a lot of different flags, with different numbers of transfers, checkers, and whenever I first run a command the upload speed starts around 1.5Mb/s, but after about 10 seconds it drops to 300Kb/s and wont go any higher. I've been googling and reading for hours, but my I still don't understand what is happening.

I'm running it on a Raspberry Pi 4b (because its the only computer I can get close enough to the router to hardwire in). And my bandwidth floats between 1Mb/s and 10Mb/s.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1
- os/version: raspbian 10.12
- os/kernel: 5.10.103-v7l+ (armv7l)
- os/type: linux
- os/arch: arm
- go/version: go1.17.9
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive, University provided with Unlimited Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "/media/pi/SSD/Hitch3" GDrive_Davis:drone-imagery/20r22/ --verbose --stats 1s --fast-list --transfers=40 --checkers=40 --tpslimit=10 --drive-chunk-size=1M

The rclone config contents with secrets removed.

[GDrive_Davis]
type = drive
scope = drive
token = ***
team_drive = 
client_id = ***
client_secret = ***

A log from the command with the -vv flag

2022/07/21 14:55:20 INFO  : Starting transaction limiter: max 10 transactions/s with burst 1
2022/07/21 14:55:20 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "copy" "/media/pi/LatYoungSSD/Hitch3" "GDrive_Davis:drone-imagery/2022/" "--verbose" "--fast-list" "--transfers=40" "--checkers=40" "--tpslimit=10" "--drive-chunk-size=1M" "-vv"]
2022/07/21 14:55:20 DEBUG : Creating backend with remote "/media/pi/LatYoungSSD/Hitch3"
2022/07/21 14:55:20 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2022/07/21 14:55:20 DEBUG : Creating backend with remote "GDrive_Davis:drone-imagery/2022/"
2022/07/21 14:55:20 DEBUG : GDrive_Davis: detected overridden config - adding "{apV6B}" suffix to name
2022/07/21 14:55:20 DEBUG : Google drive root 'drone-imagery/2022': 'root_folder_id = 0AJZBfUzVzJ9hUk9PVA' - save this in the config to speed up startup
2022/07/21 14:55:21 DEBUG : fs cache: renaming cache item "GDrive_Davis:drone-imagery/2022/" to be canonical "GDrive_Davis{apV6B}:drone-imagery/2022"
2022/07/21 14:55:25 DEBUG : pacer: low level retry 1/10 (error unexpected EOF)
2022/07/21 14:55:25 DEBUG : pacer: Rate limited, increasing sleep to 1.739345664s
2022/07/21 14:55:25 DEBUG : pacer: Reducing sleep to 0s
2

Google is rate limiting the uploads. Not a lot we can do about it I think.

Google photos has absolutely masses of rate limiting unfortunately.

3

I figured that was the case, but also I had it running at over 700Kb/s - 1.3Mb/s last week at a friends house, almost 3-4x what I'm getting here. If I could even get that far it would make a huge difference.

4

Remove that and use the defaults and you'll get a better experience.

5

changed that, and speeds are about the same

6

Can you share the log file showing what you changed and more than a few seconds of output?

7

Since I last changed it has run 10 minutes. I didn't put a log-file flag on it though, so ill rerun it for a bit with the log file flag

Edit: My grammar was terrible

8

davisLog.txt (78.7 KB)
Here's a log file showing about 17 minutes of runtime

9

Update: I let it run overnight and it dropped to the 40-60Kb/s range. At this rate it would take me months to transfer my images. Any help is greatly appreciated!

10

Ok. Seems like the rate limiting is gone.

I'd check out this one and see if that helps at all.

Many folks were complaining about slow speeds with the Google API endpoints.

Rclone mount random slow speeds - Help and Support - rclone forum

11

Thanks for the reply! I read through that thread, and it looks like most people are talking about download speeds, and they say it was resolved by google in the last 4 days. Not sure I fully understand how that applies? Sorry for my ignorance

12

They are having slow speeds and slowdowns.
You are having slow speeds and slowdowns as well.

Give it a whirl and see if the fixes they are talking about help.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.