Optimize speed for Google Drive

I have a server at Contabo, the bandwidth is 200M, and I did the following, I have a Nextcloud running on a docker on my server, and the data directory where the files are, I want to leave them mounted on the Drive.

I did a test by sending a 2G file to the mounted Drive directory on my Contabo server and found it a little slow. I use Duplicati to do bkp on the same drive and it takes 15 minutes to send about 60G of data.

With the rclone it took a little more than 2 minutes to send 2G.

This is my current configuration ...

[Unit]
Description=rClone Mount RondonEmbyMain
AssertPathIsDirectory=/home/tales/Drive
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone mount \
   --config=/home/tales/.config/rclone/rclone.conf \
   --fast-list \
   --drive-skip-gdocs \
   --vfs-read-chunk-size=128M \
   --vfs-read-chunk-size-limit=2048M \
   --buffer-size=128M \
   --max-read-ahead=256M \
   --poll-interval=1m \
   --dir-cache-time=12h \
   --timeout=10m \
   --transfers=16 \
   --checkers=8 \
   --drive-chunk-size=128M \
   --allow-non-empty \
   --allow-other \
   -vP \
   drive:/ /home/tales/Drive
ExecStop=/usr/bin/fusermount -u /home/tales/drive
Restart=on-failure
RestartSec=10
User=tales

[Install]
WantedBy=multi-user.target

Version:

    rclone v1.54.1
    - os/arch: linux/amd64
    - go version: go1.16

rclone.conf

[drive]
type = drive
client_id = XXXXXXXXXXXXXXXXXXXXXXXXXXXX
client_secret = XXXXXXXXXX
scope = drive
token = {"access_token":"XXXXXXXXT22:56:24.745806456-03:00"}
team_drive = 0AXXXXXXXX
root_folder_id = 

You seemed to delete the help and support template and lots of the information to help you out is missing.

If you can add those details, we can give you a hand.

Do you have enough information now? I believe that you don't need all this, just optimize the part of the service where the command is executed in the assembly.

Missing a log file as that’s the most critical piece.

It's working, but I think it's a little slow, it took about 2 minutes for me to send a 2G file to the mounted drive. I want to drop that time to less than 1 minute.

One other thing .. I don't know if you understand, I'll do the following ..

the "data" directory of the nextcloud where the files are located i'm going to mount my docker in the drive folder that is on my server already mounted, so there are some options like this (--dir-cache-time = 12h) that I believe need a better configuration.

there is some wrong parameter that I did not find .. I changed the configuration a little for that ..

[Unit]
Description=rClone Mount RondonEmbyMain
AssertPathIsDirectory=/home/tales/Drive
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone mount \
   --config=/home/tales/.config/rclone/rclone.conf \
   --fast-list \
   --drive-skip-gdocs \
--drive-skip-shortcuts \
   --vfs-read-chunk-size=128M \
   --vfs-read-chunk-size-limit=2048M \
   --buffer-size=128M \
   --max-read-ahead=256M \
   --poll-interval=1m \
   --dir-cache-time=12h \
   --timeout=10m \
   --transfers=10 \
   --checkers=8 \
   --drive-chunk-size=128M \
   --allow-non-empty \
   --allow-other \
   --log-file /home/tales/logs/rclone.log \
   -vP \
   drive:/ /home/tales/Drive
ExecStop=/usr/bin/fusermount -u /home/tales/drive
Restart=on-failure
RestartSec=10
User=tales

[Install]
WantedBy=multi-user.target

The log has nothing, I moved a file into the mounted folder and the log remained empty

See the duplicati backup on the same drive that the rclone is mounted on:
Captura de tela de 2021-05-24 23-07-26

Backup: 68,45GB It took 45 minutes

As part of the help template, we ask for a log file to help diagnose the issue and provide assistance. It explains to run the command with -vv and share the log file.

The log file in debug contains the details to show why something may or may not be working.

Stop the service.
Add -vv to the mount command.
Start the service.

Share the full log file.

An example log file with -vv will have all the info.

felix@gemini:~$ rclone copy /etc/hosts GD: -vv
2021/05/24 23:12:23 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/05/24 23:12:23 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]
2021/05/24 23:12:23 DEBUG : Creating backend with remote "/etc/hosts"
2021/05/24 23:12:23 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2021/05/24 23:12:23 DEBUG : Creating backend with remote "GD:"
2021/05/24 23:12:23 DEBUG : hosts: Size and modification time the same (differ by -517.966µs, within tolerance 1ms)
2021/05/24 23:12:23 DEBUG : hosts: Unchanged skipping
2021/05/24 23:12:23 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.5s

2021/05/24 23:12:23 DEBUG : 4 go routines activefelix@gemini:~$ rclone copy /etc/hosts GD: -vv
2021/05/24 23:12:23 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/05/24 23:12:23 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]
2021/05/24 23:12:23 DEBUG : Creating backend with remote "/etc/hosts"
2021/05/24 23:12:23 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2021/05/24 23:12:23 DEBUG : Creating backend with remote "GD:"
2021/05/24 23:12:23 DEBUG : hosts: Size and modification time the same (differ by -517.966µs, within tolerance 1ms)
2021/05/24 23:12:23 DEBUG : hosts: Unchanged skipping
2021/05/24 23:12:23 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.5s

2021/05/24 23:12:23 DEBUG : 4 go routines active

hi,

  • --fast-list does nothing on a mount.
  • rclone needs to be updated.

What do you mean you don't do anything?
The server is archlinux, I have the version of the repository

--fast-list - you can remove that, does not work with rclone mount

the only way to keep rclone up to date is here

1 Like

Follow the log file, I downloaded and uploaded a 2G manjaro iso

My server has 8G of RAM and I currently do not use or 2G

update for 1.55 version

Based on your log, you are still using the old version:

2021/05/25 12:59:55 DEBUG : rclone: Version "v1.54.1"

If you use something like rsync -P, you can see the time/speed.

felix@gemini:~$ rsync -P ubuntu-20.04.2-live-server-amd64.iso /home/felix/test
ubuntu-20.04.2-live-server-amd64.iso
  1,215,168,512 100%   71.36MB/s    0:00:16 (xfr#1, to-chk=0/1)

That's with a mount with no options and just one test file copy:

felix@gemini:~$ rclone mount GD: /home/felix/test -vv
2021/05/25 12:24:59 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/05/25 12:24:59 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "mount" "GD:" "/home/felix/test" "-vv"]

Yes, I updated after generating the log, but I am at 1.55 now. There are no errors, but I want to know if you can improve the transfer of files to the drive. As I said, I will mount the "data" directory of the nextcloud and all the files will be on the drive.

Nextcloud does a lot of synchronization, removing update files.

I am not sure what Nextcloud is/does as I am only here for rclone.

I have a rclone mount that I shared that maxes my gigabit line to my Google Drive with the defaults so there is not really much more optimization as rclone is generally very fast.

You may want to retest with the latest and validate your results are the same and share a new log with the updated testing.

There are many things that impact speed like geolocation and your ISP peering. I'm in the US, have fiber and have a lot of control of my bandwidth/QOS so I can be certain of a lot of things along the pipe so I get very consistent results.

My VPS is in the USA, Contabo. Nextcloud is a file server.

Did you see my options up there, you who have more knowledge, can you tell me if they are good, if I can optimize something else?

I'd use defaults and only add options if they have specific reason to be added.

I'd remove other things and validate testing with the basic of tools before adding layers on top.

I use rsync to test file copy since it gives some nice output on the performance or you can even just use rclone copy with -P.

Baseline without the mount and see what overhead it shows.

felix@gemini:~$ rclone copy ubuntu-20.04.2-live-server-amd64.iso GD: -P
Transferred:   	    1.132G / 1.132 GBytes, 100%, 74.655 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        15.7s

I get just about the same performance on a mount and without a mount.

[tales@t4l35 ~]$ rclone copy manjaro-xfce-21.0.5-210519-linux510.iso drive: -P
Transferred:        2.399G / 2.399 GBytes, 100%, 21.128 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      1m57.3s

My network is 200Mbps

so getting 21.128 MBytes/s for the transfers of a single file, that looks ok.
what if you transfer multiple files and tweak --drive-chunk-size?

Do you know Nextcloud?
I don't know how it will work with several files, when I mount the volume of the google drive, using it without any options is better? no flag?

I searched and I didn't find what the "-v" option is for, can you tell me?

I left it like this now ..

  GNU nano 5.7                                                           /etc/systemd/system/drive.service                                                                      
[Unit]
Description=rClone Mount RondonEmbyMain
AssertPathIsDirectory=/home/tales/Drive
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone mount \
   --config=/home/tales/.config/rclone/rclone.conf \
   --log-file /home/tales/logs/rclone.log \
   -P \
   drive: /home/tales/Drive
ExecStop=/usr/bin/fusermount -u /home/tales/drive
Restart=on-failure
RestartSec=10
User=tales

[Install]
WantedBy=multi-user.target

never used nextcloud.

21.128 MBytes/s from a 200Mbps connection using a vps, for a single file seems ok to me.
what more do you expect?

this is the goto guide here at the forum for gdrive + systemd
best to use that as is
homescripts/rclone.service at master · animosity22/homescripts · GitHub