Google Drive Latency Improvements

[N_P]

What is the problem you are having with rclone?

I'm trying to read the first 200 characters from a bunch of small (1-20MB) files sequentially to get some metadata and am trying to improve my latency to be better than 0.25s. I know there are limits to how fast I can get data over the internet with gdrive but didn't know if I was missing any known improvements. For testing, I've been using
time -p head -c 200 file.txt
or running 100 to get an average with
for i in {1..100}; do time -p head -c 200 file.txt ; done

I've tried lowering the data to smaller chunks (256k) and increasing and speed limits such as the pacer but it seems like I normally bottleneck at the 0.25s minimum. I have Google Fiber at home so I don't think that's the problem.

Am I missing any obvious configs or are any of my current ones counterproductive?
Do any of the flags set in the mount need to be in the config itself or do they just override?
I'm open to any suggestions as I've already tried about 50+ combinations moving down from about 0.5s to 0.25s.

I've also tried searching the forums the best I could, a lot of other posts relate to VFS, and since I'm only reading files once I haven't seemed to get any benefits out of it.

What is your rclone version (output from rclone version)

rclone v1.55.0

• os/type: linux
• os/arch: amd64
• go/version: go1.16.2
• go/linking: static
• go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu, 64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount meta: /mnt/meta --allow-other --buffer-size 256k --dir-cache-time 1000h --poll-interval 15s --read-only --no-checksum --no-modtime --drive-pacer-min-sleep 10ms --tpslimit 100 --timeout 1h --max-read-ahead 4k --attr-timeout 1m --vfs-read-chunk-size 256k --drive-chunk-size 256k --transfers 50

The rclone config contents with secrets removed.

[meta]
type = drive
scope = drive.readonly
team_drive = *
token = *
client_id = *
client_secret = *
pacer_min_sleep = 10ms
pacer_burst = 100

A log from the command with the -vv flag

I can add logs if needed, however, I think it's mostly a config question.

[ncw (Nick Craig-Wood)]

That seems to be about how long it takes unfortunately

You could try turning the buffering off --buffer 0 but I'm not optimistic it will help.

[asdffdsa (jojothehumanmonkey)]

hi,
i did a test using wasabi, a s3 rclone known, for hot storage

this is the mount command

source="wasabi01"
mountpoint="/home/ubuntu/rclone/mountpoint/$source"
rclone mount $source:timehead200 $mountpoint

and if i run

file="/home/ubuntu/rclone/mountpoint/wasabi01/file.txt"
time -p head -c 200 $file

the output is

real 0.13
user 0.00
sys 0.00

and if i run

file="/home/ubuntu/rclone/mountpoint/wasabi01/file.txt"
for i in {1..100}; do time -p head -c 200 $file ; done

this is the output
as i am learning linux, i was not able to copy all 100 runs from the terminal but this is what it all looked like

real 0.13
user 0.00
sys 0.00
real 0.13
user 0.00
sys 0.00
real 0.13
user 0.00
sys 0.00
real 0.11
user 0.00
sys 0.00
real 0.12
user 0.00
sys 0.00
real 0.12
user 0.00
sys 0.00
real 0.12
user 0.00
sys 0.00
real 0.13
user 0.00
sys 0.00
real 0.12
user 0.00
sys 0.00
real 0.13
user 0.00
sys 0.00
real 0.11
user 0.00
sys 0.00
real 0.13
user 0.00
sys 0.00
real 0.13
user 0.00
sys 0.00

[N_P]

Woops, looks like I copied an old one. One like this should do the total time for 100

time -p for i in {1..100}; do head -c 200 file.txt ; done

[asdffdsa (jojothehumanmonkey)]

file="/home/ubuntu/rclone/mountpoint/wasabi01/file.txt"
time -p for i in {1..100}; do head -c 200 $file ; done

real 13.10
user 0.07
sys 0.74

[N_P]

Old 1000:
real 235.75
user 1.39
sys 0.71

0 Buffer 1000:
real 231.75
user 1.39
sys 0.71

Difference seems to be within the margin for error.
I'm getting closer to 0.4s using the google drive API directly with curl for a partial reads so I have to commend you for being faster than that somehow.

[left1000]

I'd like to mention about...... a month? or so? ago google's api request limit started throttling me about twice as hard as it ever had before.

I might make a separate thread about it, but I'm not really sure there's anything to do about it, or any solution. (My guess is that they're intentionally making my service worse because I'm grandfathered in to the plan where I drastically underpay them, and they're hoping to scare me off before having to actually deal with me/us? who knows.)