Best procedures for uploading large files to Google Drive?

Help and Support
1

What is the problem you are having with rclone?

My upload speed seems to be slow. I have done some research and seen that I should increase the chunk size when dealing with large files. The files I am uploading are 100GB - 140GB in size (high quality / bitrate edited videos). There will no be many files uploaded daily due to the 750GB GDrive limit, so around 6-7 files only uploaded per day.

My question is, is there anything I can do on rclone / the pc's end in order to increase efficiency of uploading a single large file?

I have also already generated a clientID and clientSecret, so I am not using the default ones from rclone. These were generated from another google account (not the google drive's account), but it should be fine as per the docs.

What is your rclone version (output from rclone version)

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04.2 LTS x86_64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy 2021_08_03_BeachSunset.mp4 /mnt/gdrive/1/ -P --drive-chunk-size 2048M --buffer-size=8192M

The rclone config contents with secrets removed.

[drive1]
type = drive
client_id = xxxxxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxxxxx
scope = drive
token = {"access_token":"ya29.a0ARrdaM-sCT9YjkiSWDTxnsCdacuwphem0KYTcqUCd6HcPoA_n-6Nn5X58FhEnoJ7AiKNm55>
root_folder_id = 0ALPHeeCXztPLUk9PVA

A log from the command with the -vv flag

rclone copy 2021_08_03_BeachSunset.mp4  /mnt/gdrive/1/ -P --drive-chunk-size 512M -vv
2021/08/09 00:28:17 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "copy" "2021_08_03_BeachSunset.mp4" "/mnt/gdrive/1/" "-P" "--drive-chunk-size" "512M" "-vv"]
2021/08/09 00:28:17 DEBUG : Using config file from "/home/chushie/.config/rclone/rclone.conf"
2021-08-09 00:28:17 DEBUG : 2021_08_03_BeachSunset.mp4: Need to transfer - File not found at Destination
2021-08-09 00:28:17 DEBUG : preAllocate: got error on fallocate, trying combination 1/2: operation not supported
2021-08-09 00:28:17 DEBUG : preAllocate: got error on fallocate, trying combination 2/2: operation not supported
Transferred:      392.027M / 101.337 GBytes, 0%, 11.702 MBytes/s, ETA 2h27m13s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 1, 0%
Elapsed time:       33.5s
Transferring:
 * 2021_08_03_BeachSunset.mp4 :  0% /101.337G, 11.881M/s, 2h25m1s

*Do note the following log is with only 512 drive chunk size and no buffer size, however changing those to what I showed on the command above doesn't lead to any improved speeds.

Thank you very much for your time!

2

You have an old version of rclone so you'd want to update that.

Changing the drive-chunk-size and copying to the mount wouldn't have any impact. To upload faster, it's better to upload directly to your remote rather than using a mount.

If you want to use the mount still, you'd want to add the chunk size parameter to your remote in your rclone.conf. I use 1GB for me.

chunk_size = 1024M
3

Thanks for the reply! What do you mean uploading directly to the remote rather than using the mount? How would I do that?

atm, I have a crypt mounted, and I'm uploading to that. Also is 1024M for chunk_size best practice for 100GB files?

4

Run:

rclone copy /tmp/some/localfile remotename:

As you don't need a mount to copy files.

5

Perfect, thank you! works fine on my crypt drive.

Quick question, since my files are around 120GB in storage, should I increase the chunk_size to be higher? and /or anything else I can do to increase speeds? atm only getting ~52 Mi/s and I should be getting more in theory.

Lastly, I reckon I can wget directly into drive as well this way? (have to use wget bc multi-threaded solutions such as axel doesn't work)

wget http://127.0.0.1:4067/file.txt -P 2:

Is that right?

Thanks once again Animosity

6

I only use rclone to interact with my drive so not sure on wget.

Chunk size can be whatever works best for you so you have to fiddle to see what you get and what performs best. I use 1G for my setup as that seems to work ok for me.

1 Like
7

@chushie you never told us what your Internet speeds are. I've been using the following flags for all my transfers for a long time, with a couple of recent additions:

--ignore-existing --verbose --transfers 4 --checkers 8 --bwlimit 95M --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --drive-chunk-size 512M --stats 1s --stats-file-name-length 0 --drive-pacer-burst 2000 --drive-pacer-min-sleep 5ms --user-agent *******

This will copy/move 4 files at a time, maxing out my gig/gig connection. Fewer files will slow things down, and in my experience, increasing drive-chunk-size beyond 512M/1024M won't make a difference. You should try to find your sweet spot, though.

Note that a lot of my flags are RcloneBrowser defaults, which is what I use for all my uploads.

8

@VBB my internet speeds are about 1000 /1000 Mbps .

would i still need to use arguments if i'm moving files the way @Animosity022 stated? since it will be uploading directly to the remote? moreover, does changing chunk_size in the rclone .config change anything either, since i'm uploading directly to the remote?

I'm getting an average of ~50 Mbps upload, roughly 50%. Speedtest at the moment is giving a speed of 800, so it's still bottlenecked it seems..

9

Those flags apply regardless of how you upload. Some of them might be Rclone defaults anyway, but you'd have to check which ones. Whatever you set in the config file applies to any and all Rclone actions. So, if you put drive-chunk-size in there, you're good to go. I don't use any global flags, so I put them all in the individual commands.

10

Alright. Thanks!

11

Do you knwo if there is a way to wget / alternative directly to the remote? It seems like wget can only do it to the mount, such as /mnt/drive/1/. I believe this may lower upload speeds?

12

Sorry, I have no experience with wget.

13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.