Upload between team drive and google drive using my bandwith

Help and Support
#1

What is the problem you are having with rclone?

Its using the speed from my server to move between team drives and google drive

What is your rclone version (output from rclone version)

v1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64 bits

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy bentancur:/encriptado sdrive:/encriptado --progress -vv

The rclone config contents with secrets removed.


[sdrive]
type = drive
scope = drive
token = {"access_token":"ya29.a0A_YtgrbQyq24Jui4vMfqCMl-OqvXdDFjC8PgsNLwNSsq8S49c","expiry":"2021-07-17T14:01:41.0614089-03:00"}
root_folder_id = 0ALRjm0Zd_dlCUk9PVA
client_id = 34311510110-3jvakjs94lric55rcontent.com
client_secret = PwRK

[bentancur]
type = drive
scope = drive
token = {"access_token":"ya29.a0AYBoMsVlDvJIqEhvG61gKXjaBRPskeFMXgsgNFZMQAgw2IJ9PsYZmwbziTEw3tGZaKyXND_BsYt2HZpwfA_rsqxD2shjyedYUn9jMwIlfD782m4","expiry":"2021-07-17T14:01:39.7321771-03:00"}
team_drive = 0AIOKIMNbUPVA
root_folder_id =

A log from the command with the -vv flag

Microsoft Windows [Versión 10.0.17134.1792]
(c) 2018 Microsoft Corporation. Todos los derechos reservados.

C:\Users\RODRIGO>cd desktop

C:\Users\RODRIGO\Desktop>cd googledrive

C:\Users\RODRIGO\Desktop\GoogleDrive>rclone config
Current remotes:

Name                 Type
====                 ====
1                    drive
10                   drive
11                   drive
12                   drive
125                  drive
126                  drive
129                  drive
13                   drive
131                  drive
132                  drive
134                  drive
137                  drive
139                  drive
14                   drive
140                  drive
2                    drive
3                    drive
4                    drive
5                    drive
6                    drive
7                    drive
8                    drive
9                    drive
acache               cache
adrive               drive
ateam                drive
ateamcache           cache
bentancur            drive
gcache               cache
gdrive               drive
gnomo                drive
local                local
prodrigo             drive
prueba               drive
rclone               cache
remux                drive
sdrive               drive
team                 drive
uhd                  drive
ycache               cache
ydrive               drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

C:\Users\RODRIGO\Desktop\GoogleDrive>rclone copy bentancur:/encriptado sdrive:/encriptado --progress -vv
2021/07/17 13:01:40 DEBUG : Using config file from "C:\\Users\\RODRIGO\\Desktop\\GoogleDrive\\rclone.conf"
2021/07/17 13:01:40 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "bentancur:/encriptado" "sdrive:/encriptado" "--progress" "-vv"]
2021/07/17 13:01:40 DEBUG : Creating backend with remote "bentancur:/encriptado"
2021/07/17 13:01:40 DEBUG : bentancur: Loaded invalid token from config file - ignoring
2021/07/17 13:01:40 DEBUG : Saving config "token" = "{\"access_token\":\"ya29.a0ARrdaM8vsX0ZYjAkqBlwJURkigHuDTlNLCX-gSH6M3fi0\",\"token_type\":\"Bearer\",\"refresh_token\":\"1//099vWsOPcur" of the config file
2021/07/17 13:01:40 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2021/07/17 13:01:41 DEBUG : bentancur: Saved new token in config file
2021/07/17 13:01:41 DEBUG : fs cache: renaming cache item "bentancur:/encriptado" to be canonical "bentancur:encriptado"
2021/07/17 13:01:41 DEBUG : Creating backend with remote "sdrive:/encriptado"
2021/07/17 13:01:41 DEBUG : sdrive: Loaded invalid token from config file - ignoring
2021/07/17 13:01:42 DEBUG : Saving config "token" = "{\"access_token\":\"ya29.a0ARrdGBESNwF-L9Irgtc6R4ULEEMU2dlaUBjWQ47QttoQc-WuKGr356mKUn_G8lz4UIsvOp49c\",\"expiry\":\"2021-07-17T14:01:41.0614089-03:00\"}" in section "sdrive" of the config file
2021/07/17 13:01:42 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2021/07/17 13:01:42 DEBUG : sdrive: Saved new token in config file
2021/07/17 13:01:42 DEBUG : fs cache: renaming cache item "sdrive:/encriptado" to be canonical "sdrive:encriptado"
2021-07-17 13:01:43 DEBUG : Google drive root 'encriptado': Waiting for checks to finish
2021-07-17 13:01:43 DEBUG : Google drive root 'encriptado': Waiting for transfers to finish
2021-07-17 13:01:46 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 0 length 8388608
2021-07-17 13:01:49 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 8388608 length 8388608
2021-07-17 13:01:52 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 16777216 length 8388608
2021-07-17 13:01:55 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 25165824 length 8388608
2021-07-17 13:01:58 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 33554432 length 8388608
2021-07-17 13:02:01 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 41943040 length 8388608
2021-07-17 13:02:04 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 50331648 length 8388608
2021-07-17 13:02:07 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 58720256 length 8388608
2021-07-17 13:02:09 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 67108864 length 8388608
2021-07-17 13:02:12 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 75497472 length 8388608
2021-07-17 13:02:15 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 83886080 length 8388608
2021-07-17 13:02:18 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 92274688 length 8388608
2021-07-17 13:02:20 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 100663296 length 8388608
2021-07-17 13:02:23 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 109051904 length 8388608
2021-07-17 13:02:26 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 117440512 length 8388608
2021-07-17 13:03:04 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 125829120 length 8388608
2021-07-17 13:03:07 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 134217728 length 8388608
2021-07-17 13:03:10 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 142606336 length 8388608
2021-07-17 13:03:13 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 150994944 length 8388608
2021-07-17 13:03:15 DEBUG : Breaking.Bad.S05E15.2160p.WEB-DL.3xRus.2xEng.TrollUHD-ULTRAHDCLUB.mkv: Sending chunk 159383552 length 8388608
#2

thanks for re-posting the requested info.
if you do not want to use your local bandwidth, use
https://rclone.org/drive/#drive-server-side-across-configs

#3

I never needed to use that before and I copied over 400 Tb, but im gonna try that now

Edit: It works like a charm, really weird I never had to use it before

#4

If Im trying to copy it to a new crypt folder should the command works? Because is using again my bandwith

#5

from non-crypt remote to crypt remote, that flag will not work, as that is not a simple file copy operation.
rclone has to download the non-crypted file, crypt it and then upload it, tho rclone does that a chunk at a time.

one workaround is to use google cloud virtual machine and run rclone on that.
they offer a free and cheap virtual machines.
and if done correctly, there is no cost for ingress/egress of data between two gdrives.

#6

Thanks, Im gonna take a look on that

#7

I created a virtual machine on google cloude and I try the copy to the crypto but Im getting 12Mbps transfer speed, if I copy to another uncrypted folder then it moves like at 27Gbps

Any ideas what can I do?

#8

what was the exact command?

without knowing the exact details on the virtual machine you are using, i would guess the limiting factor is CPU. the processing power required to crypt the file.

#9

rclone copy 2:/encriptado encriptado:/p --drive-chunk-size 1024M --drive-stop-on-upload-limit --drive-server-side-across-configs --progress

I guess the problem is the CPU then. I choose the e2-medium 4gb ram

#10

using that flag, sure, the transfer speed is so much higher.
and since that flag does not use the vm network, --drive-chunk-size 1024M should have no effect

#11

So the only way to do it is with a high end server?

#12

the vm you are currently using works fine at 12Mbps.
is that is too slow, perhaps get a vm with more power

#13

I was getting 110 Mbps on my dedicated server. I have around 600 Tb

#14

rclone has to crypt the data using local computing resources, virtual machine or physical machine.

sorry, there is no --magic workaround for that.

#15

Ok thanks anyway for the help, I learned a few new things

#16

learn and return...