Very slow copying from Raspberry PI to Google drive

Dew · July 13, 2020, 9:34pm

What is the problem you are having with rclone?

I am getting around 4-5 MBytes/s when copying from raspberry pi 4 to google drive but I am getting around 60 MBytes/s when copying from windows to google drive

What is your rclone version (output from `rclone version`)

rclone v1.52.2

os/arch: linux/arm
go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

"Raspbian GNU/Linux 10 (buster), 32bit

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy --verbose --transfers 4 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s --stats-file-name-length 0 --fast-list /srv/dev-disk-by-label-Data1/downloads/completed/ubuntu-20.04-desktop-amd64.iso crypt:

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = XXX

[crypt]
type = crypt
remote = gdrive:/crypt
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

A log from the command with the `-vv` flag

2020/07/14 05:31:57 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "copy" "-vv" "--log-file" "/srv/dev-disk-by-label-Data/appdata/rclone/rclone.log" "--verbose" "--transfers" "4" "--checkers" "8" "--contimeout" "60s" "--timeout" "300s" "--retries" "3" "--low-level-retries" "10" "--stats" "1s" "--stats-file-name-length" "0" "--fast-list" "/srv/dev-disk-by-label-Data1/downloads/completed/ubuntu-20.04-desktop-amd64.iso" "crypt:"]
2020/07/14 05:31:57 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2020/07/14 05:31:57 DEBUG : fs cache: renaming cache item "/srv/dev-disk-by-label-Data1/downloads/completed/ubuntu-20.04-desktop-amd64.iso" to be canonical "/srv/dev-disk-by-label-Data1/downloads/completed"
2020/07/14 05:32:00 DEBUG : ubuntu-20.04-desktop-amd64.iso: Need to transfer - File not found at Destination
2020/07/14 05:32:00 INFO  : 
Transferred:   	         0 / 2.529 GBytes, 0%, 0 Bytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:         0.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 0/s, -

2020/07/14 05:32:01 DEBUG : h39g744r4mekhgab4djeruccskh62fug7t216sgbqm5d8p2u883g: Sending chunk 0 length 8388608
2020/07/14 05:32:01 INFO  : 
Transferred:   	  640.172k / 2.529 GBytes, 0%, 410.782 kBytes/s, ETA 1h47m33s
Transferred:            0 / 1, 0%
Elapsed time:         1.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 0/s, -

2020/07/14 05:32:02 INFO  : 
Transferred:   	    2.938M / 2.529 GBytes, 0%, 1.148 MBytes/s, ETA 37m32s
Transferred:            0 / 1, 0%
Elapsed time:         2.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 928.272k/s, 47m33s

2020/07/14 05:32:03 INFO  : 
Transferred:   	    5.470M / 2.529 GBytes, 0%, 1.537 MBytes/s, ETA 28m
Transferred:            0 / 1, 0%
Elapsed time:         3.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 1.386M/s, 31m4s

2020/07/14 05:32:04 INFO  : 
Transferred:   	        8M / 2.529 GBytes, 0%, 1.755 MBytes/s, ETA 24m30s
Transferred:            0 / 1, 0%
Elapsed time:         4.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 1.727M/s, 24m54s

2020/07/14 05:32:04 DEBUG : h39g744r4mekhgab4djeruccskh62fug7t216sgbqm5d8p2u883g: Sending chunk 8388608 length 8388608
2020/07/14 05:32:05 INFO  : 
Transferred:   	   11.096M / 2.529 GBytes, 0%, 1.991 MBytes/s, ETA 21m35s
Transferred:            0 / 1, 0%
Elapsed time:         5.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 1.899M/s, 22m37s

2020/07/14 05:32:06 INFO  : 
Transferred:   	   13.972M / 2.529 GBytes, 1%, 2.130 MBytes/s, ETA 20m8s
Transferred:            0 / 1, 0%
Elapsed time:         6.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 2.047M/s, 20m57s

2020/07/14 05:32:07 DEBUG : h39g744r4mekhgab4djeruccskh62fug7t216sgbqm5d8p2u883g: Sending chunk 16777216 length 8388608
2020/07/14 05:32:07 INFO  : 
Transferred:   	   16.629M / 2.529 GBytes, 1%, 2.200 MBytes/s, ETA 19m29s
Transferred:            0 / 1, 0%
Elapsed time:         7.5s
Transferring:
 * ubuntu-20.04-desktop-amd64.iso:  0% /2.529G, 2.161M/s, 19m50s

asdffdsa · July 13, 2020, 9:36pm

hello and welcome to the forum

have you done this?
https://rclone.org/drive/#making-your-own-client-id

what is the rclone command you are using on windows?

Dew · July 13, 2020, 9:48pm

Hi,

Thanks.

Yes i am using my own client ID and I use the same command in windows.

  C:\Users\Dew\AppData\Local\Programs\Rclone Browser\rclone-v1.52.2-windows-amd64\rclone.exe copy --verbose --transfers 4 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s --stats-file-name-length 0 --fast-list E:\Videos\Movies\XXX.mkv crypt:OMV

asdffdsa · July 13, 2020, 9:50pm

figure out the max upload speed using pi4.
is the pi4 using a lan or wifi?
can you do internet speed tests for both computers.

rclone copy --verbose --transfers 1 --checkers 1 /srv/dev-disk-by-label-Data1/downloads/completed/ubuntu-20.04-desktop-amd64.iso crypt:

Dew · July 13, 2020, 10:18pm

How do i figure out the max upload speed?
Pi is on LAN
PI speed

Download: 23.62 Mbit/s Upload: 43.95 Mbit/s

Windows

Around 1 GB Mbps

asdffdsa · July 13, 2020, 10:29pm

on the pi4, for /srv/dev-disk-by-label-Data1/
is that usb drive, sdcard, or what?

that does does not look correct?
perhaps 1Gbps which is 1000Mbps

windowz=1000Mbps
pi4=44Mbps

is that correct?

Dew · July 13, 2020, 10:32pm

Yes.. that's a USB drive connected to the USB 3 port.

Windows = 1000Mbps

asdffdsa · July 13, 2020, 10:35pm

so the pi4 internet upload speed is just 4.4% as compared to windoz?

are the pi4 and the windoz on the same lan?

about that usb3 drive,

is it using UASP or not?
ssd or not?
can you test its performance?

ncw · July 15, 2020, 2:13pm

An upload speed of 43 Mbit/s is about 4.5 Mbyte/s so that is probably the thing to work on.

Dew · July 15, 2020, 2:29pm

thanks.. i am asking for help in the other forum regarding the ethernet speed. I have already tried to disconnect the wifi so i'm not sure why it is getting such a slow speed.

Dew · July 17, 2020, 12:51am

It is indeed a network issue. Thanks for the help and please close this!

ncw · July 17, 2020, 8:15am

What was the problem in the end?

asdffdsa · July 17, 2020, 1:47pm

hi @Dew,

i just purchased a pi4, and was getting slow upload speeds.

using pip, i updated speedtest-cli and got better speeds.
i was going to post about here this morning, but you beat me to it.

but no where near my windows laptop speeds.

could you share what you learned?
and what is your upload/download speeds using rclone and speedtest-cli .

thanks

Dew · July 18, 2020, 6:05am

I had a docker image running openVPN but I set it to use host network instead of bridge network. This affected the host network speed as it was going through the VPN too. Hope it help!

I am now getting close to windows speed with speedtest on the PI4 and having around 30-40 MB transfer speed using rclone.

asdffdsa · July 18, 2020, 1:00pm

me too, i get good speeds using speedtest.
with rclone, 30-40 MB, that is all i can get with rclone.
on my windows laptop, i can easily saturate my 1Gbps fiber optic connection.

Dew · July 18, 2020, 1:13pm

I got around 60-70MB on windows anyway i am not too particular as there is a 750GB upload limit to gdrive.

system · September 17, 2020, 9:13am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.