Poor read speed from Google Drive

What is the problem you are having with rclone?

My home download is about 500mbps/62.5MB/s. When downloading files from a Google Drive share I'm getting reasonable speeds, but nothing near the theoretical max.

Run the command 'rclone version' and share the full output of the command.

Docker container so:

rclone v1.57.0
- os/version: alpine 3.14.2 (64 bit)
- os/kernel: 5.4.0-91-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

mount gcrypt: /data/gcrypt --allow-other --dir-cache-time 1000h --log-level INFO --poll-interval 15s --umask 000 --user-agent oop --rc --rc-addr :5572 --cache-dir=/cache --vfs-cache-mode full --vfs-cache-max-size 250G --vfs-cache-max-age 2000h --vfs-read-ahead 2G --vfs-read-chunk-size-limit 0 --vfs-cache-poll-interval 5m --allow-non-empty

The rclone config contents with secrets removed.

I think this is pretty standard, but here it is:

[gcrypt]
type = crypt
remote = gdrive:media
filename_encryption = standard
directory_name_encryption = true
password = 
password2 = 

[gdrive]
type = drive
scope = drive
server_side_across_configs = true
client_id = 
client_secret = 
team_drive = 
token = 

A log from the command with the -vv flag

Can't do this at the moment - I have it on INFO and there's just stuff about vfs cache cleaned.

Extra Info

I tested the read speed of the drive mount with dd if={a rather large file} of=/dev/null bs=1G count=1 and the output is:

1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 55.2374 s, 19.4 MB/s

I tested the speed of the drive the cache is on with dd if=/dev/zero of=test.img bs=1G count=1 oflag=direct and the output is:

1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.65512 s, 161 MB/s

Testing reads with dd if=test.img of=/dev/null bs=1G yields:

1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.65512 s, 161 MB/s

My cache drive is a HDD, but with these figures I wouldn't have thought it would be an issue as cache writing should also be sequential like dd?

I ran the first command with a few files and it always seemed to cap out at 20MB/s with typical figures around 15MB/s. I never run it on the same file twice due to files being cached.

Could this be a crypt problem, with my CPU not being able to handle it fast enough? I've looked at htop when reading and the CPU seems to spike to a max of 40% with it being quite periodic (because of the chunks?). I know there's a --drive-chunk-size, but I believe this only applies to uploads?

The HDD tests are run on the host, not within the docker container. When I run dd in alpine it seems to miss the rate output.

Any ideas? I may be missing something here.

hello and welcome to the forum,

imho, test using rclone copy gdrive: to establish a base line.

and really need to use a debug log, looks for issues such as pacer

imho, test using rclone copy gdrive: to establish a base line.

This doesn't show any sort of rate output, just outputs all the files being copied. What do we expect to see?

and really need to use a debug log, looks for issues such as pacer

Okay will do, I assume loglevel=DEBUG is good?

add --progress

keep in mind that with gdrive, rclone can only copy 2/3 files per second.

--log-level=DEBUG

I've uploaded the logs of rclone mount with this. They're quite long, so I had to upload to Google Drive, as I can't upload here. Is this okay? I ran dd if=mybigfile.mkv of=/dev/null bs=1G count=1.

Will do this now.

well, as i suggested,
if you want to test download speeds from gdrive,
then download direct from gdrive,
using a command like rclone copy gdrive: /path/to/local/folder --progress,

Running rclone --config rclone.conf copy --progress gdrive:media/60b5c9pvhntomp3n5j5g6bs308/5b9v453aaniu9dep40u82d12hggt1b5eike1820qgvlomij3pf20 .

Transferred:        1.338G / 27.257 GBytes, 5%, 57.624 MBytes/s, ETA 7m40s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 2, 50%
Elapsed time:       23.7s
Transferring:
 * pq647li0mag1706lpbrkdo…apsjcicor8hs39r7qatcio:  4% /27.257G, 57.730M/s, 7m39s

I didn't let it finish, as it qould take quite a while. I guess this implies that the limiting factor is crypt? The CPU on this machine isn't ancient but it's not great, it's an i3-2120. Unless the file I'm testing here is some sort of anomaly.

no need to guess, just download a file from the gcrypt. something like

rclone --config rclone.conf --progress copy gcrypt:path/to/folder/file .

Hmm, no it's not.

rclone --config rclone.conf copy --progress gdrive:media/60b5c9pvhntomp3n5j5g6bs308/ .
2022-01-24 21:16:45 NOTICE: 410t28k1vekq04k1k20a1fjai0/vlikrraqd9eqh5t2r2ocrmtfsn3ac1q0ekvmregpk69gqkfidh7tbdepgf6ecm4bi9cod3dqd8l0gv9s58kcc6fv80h8hojgpb9cue8: Duplicate object found in source - ignoring
2022-01-24 21:16:56 NOTICE: biu1e2v7v15c4at2jfua8n1g2gsipr3600vqctkcj4pgamg726jg/jmr8to4hfbjam9lrim97522rh4f54278tvjfne1hdnpeqql0vmagt2tnrvgn28uoedbln6ga9moqmt1nd77kb8h4uu7idpnhjv3smeo7qvolp31ss4pt0440f26los0s: Duplicate object found in source - ignoring
2022-01-24 21:17:07 NOTICE: k9rqurha28re07ms464gkbv8tuihmqeko6td0ohrnj8rncs0hakjlb9t4lpt1fq3dp9p328qhoomk/1doo0sqndq33rnpd8knavcdhg2t0md4k33amtfb433uvag64cd740t9dcr0scest980u3pdlv96n5tn5vlishub7s0qkadi2tl4qcgjalun45jvnbu3k48us2m6ocpg2bgjchb4fgtqngmumtsjbp9ccubji80rf03si3ej7nahc3t7jpfenujh2jbqsd7vg2q36mql69jluo: Duplicate object found in source - ignoring
2022-01-24 21:17:07 NOTICE: k9rqurha28re07ms464gkbv8tuihmqeko6td0ohrnj8rncs0hakjlb9t4lpt1fq3dp9p328qhoomk/1doo0sqndq33rnpd8knavcdhg2t0md4k33amtfb433uvag64cd740t9dcr0scest980u3pdlv96n5tn5vlishub7s0qkadi2tl4qcgjalun45jvnbu3k48us2m6ocpg2bgjchb4fgtqngmumtsjbp9ccubji80rf03si3ej7nahc3t7jpfenujh2jbqsd7vg2q36mql69jluo: Duplicate object found in source - ignoring
2022-01-24 21:17:12 NOTICE: mvkcs7d6vde3q0bnicjf0bvlf99qfl24ijop82b65ta156lvitg0/ncqvp33sh8eq0ipr4df5up3hj5uki7cntlsd9qvd6tgrtht2qkk9o0k3a643ol3pqc4vl606bpqqdnt2obsa6a5dtnknliocfe3vr3g: Duplicate object found in source - ignoring
2022-01-24 21:17:12 NOTICE: p8u7nt7gbhj6j46md91q1l37vsa0brrh8uvk6vafa06ai3t8gout08l2nfpjqjctl8i0boscg6rm0/nog1h0btk9n8qq25tu4u4q7a1qv8dtbdum59p24cpl7qnl137skvob96ng3je2h7nj01otqrc8grus8oflsviq46bivokd9h3v7g8g8l244b4tt6br9sl9tpe8083aet: Duplicate object found in source - ignoring
Transferred:        1.760G / 3.475 TBytes, 0%, 58.672 MBytes/s, ETA 17h14m37s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1255, 0%
Elapsed time:       30.7s
Transferring:
 * 0g8hhsdp1u0cr0roseio8c…hb8342dv69ek1r6834580m:  5% /8.747G, 16.127M/s, 8m43s
 * 0g8hhsdp1u0cr0roseio8c…ollc1t1rqcpd7tfrosp5jg: 21% /2.301G, 16.806M/s, 1m50s
 * 0qggsqnrlrbaua3oemopsl…7387m88csc9tmuvhrpcvgh:  1% /26.674G, 16.825M/s, 26m34s
 * l4umhpek2ivfgiinlooll7…epl6j636otle43reb6q1jg: 99% /303.074M, 9.796M/s, 0s^C

did you see my last post?

Yes, running now. Just need to convert the crypted filenames.

OP, you're downloading at max speeds (57MB/s) according to your ISP's download speed (500mbps).

`rclone --config rclone.conf copy -P gcrypt:file.mkv .
Transferred:      698.367M / 27.250 GBytes, 3%, 47.422 MBytes/s, ETA 9m33s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 2, 50%
Elapsed time:       14.7s
Transferring:
 *file.mkv:  2% /27.250G, 46.746M/s, 9m41s^C

So it seems with the rclone commands, but not in my original mount situation (see original post).

Yes, downloading/uploading from mounts is not recommended, as it's always slower.

1 Like

So using rclone copy seems to yield a high transfer speed, but using mount doesn't. Also, to do this copy I shut down other applications that were also using the mount which I suppose may also affect the transfer speeds.

Bummer. Is this a pretty hard limitation, or something that can be tweaked at all?

As far as I'm aware, there are no flags that will make this faster. For small, individual uploads/downloads or syncs, a mount might work just fine, but for large, continuous transfers you want to use move/copy, ideally with a larger than default --drive-chunk-size (if supported).

if you want to test rclone mount download speeds, use a file manager and download a large file.
rclone mount will always be much slower

for example, rclone copy wasabi01: . can easily saturate my 1Gbps internet connection.
but with rclone mount
image

1 Like

Unfortunately, using move/copy isn't possible here. Does --drive-chunk-size work with mount?