Performance issues

Hello,

we cant' get it faster than 15KB/s with GSuite Drive. We created our own ID and using it.

The best transfer rate we achieved is with:
--fast-list --transfers=40 --checkers=40 --drive-chunk-size=1M

Here is what we get with RClone (v1.48.0):

Transferred: 834.834k / 3.681 GBytes, 0%, 13.705 kBytes/s, ETA 78h12m27s
Errors: 0
Checks: 0 / 0, -
Transferred: 78 / 10538, 1%
Elapsed time: 1m0.9s
Transferring:

This speed makes it unpractical to use. Anybody has any other suggestion in order to increase performance?

Here is also the googleapis stats

-- UPDATE --
The speed is as expected if I transfer big files however if I send a directory with small files (like a git repo) the speed makes crawl down to unusable levels
I also notice the %errors on oauth and lists

With Google, they only let you create 2-3 files per second.
You can only only do ~10 transactions per second and you have it configured for roughly 80 with those settings.

If you run with -vv, you'll see quite the number of retries along the way majorly slowing you down.

I run with a very small set of transfers and checkers, but my use case is moving larger media files per night:

Transferred:      212.989G / 212.989 GBytes, 100%, 80.682 MBytes/s, ETA 0s
Errors:                 0
Checks:                36 / 36, 100%
Transferred:           18 / 18, 100%
Elapsed time:     45m3.2s

with majority of the defaults:

/usr/bin/rclone move /data/local/ gcrypt: --log-file /opt/rclone/logs/upload.log -v --drive-chunk-size 64M --exclude-from /opt/rclone/scripts/excludes --delete-empty-src-dirs --fast-list

unfortunately using --transfers=5 --checkers=5 doesn't change the performance and -vv doesn't show any errors/retries.
Top speed achieved 65 KB/s.
I would like to reach 60 MB/s (or even 6 MB/s) but not 0.06 MB/s (the link speed is 100 MB/s)

With Google, you can only create 2-3 files per second via the API. There is no magic to fix that other than if Google changes the policy.

You can work around it by zipping things up or tarring up things to make them larger files rather than a lot of small ones.

I think you mean 100 Mb/s (megabits per second) rather than MB/s (Megabytes per second). I'm assuming you have 100Mb link so you'd expect to see at most ~8MB/s.

I really meant MB and not Mb as I have a Gb link (up and down).
And with the speed of rclone I really meant 65 KB/s as in Kilo Bytes.
I tried now with the Mega service and there the top speed is 150 KB/s, also too slow.
I will try some zip workaround, thanks for the tip.

The link speed isn't 100MB/s though at Gigabit as it's more like 125MB/s. I'd just refer to it as a gigabit link if that's what you got.

I normally push close to my max gigabit FIOS link with overhead.

Transferred:   	  184.781G / 184.781 GBytes, 100%, 81.827 MBytes/s, ETA 0s
Transferred:   	  102.842G / 102.842 GBytes, 100%, 88.843 MBytes/s, ETA 0s
Transferred:   	  291.600G / 291.600 GBytes, 100%, 90.464 MBytes/s, ETA 0s
Transferred:   	  294.829G / 294.829 GBytes, 100%, 88.661 MBytes/s, ETA 0s
Transferred:   	  212.953G / 212.977 GBytes, 100%, 80.731 MBytes/s, ETA 0s
Transferred:   	  212.989G / 212.989 GBytes, 100%, 80.682 MBytes/s, ETA 0s
Transferred:   	  561.735G / 564.324 GBytes, 100%, 85.589 MBytes/s, ETA 30s
Transferred:   	  564.214G / 564.324 GBytes, 100%, 85.206 MBytes/s, ETA 1s
Transferred:   	  564.338G / 564.338 GBytes, 100%, 85.174 MBytes/s, ETA 0s
Transferred:   	  350.591G / 350.901 GBytes, 100%, 87.945 MBytes/s, ETA 3s
Transferred:   	  350.905G / 350.905 GBytes, 100%, 87.763 MBytes/s, ETA 0s

Those are a list of my daily uploads but again, they are mostly video files and larger where you don't hit the 2-3 file creation limit.

I use rcat to do it quite easily. Maybe that will help you.

tar -zcvf - / --exclude-from=/tmp/exc | rclone rcat robgs-cryptp:/vpsbackups/data_date "+%Y-%m-%d_%H:%M:%S".tar.gz -v

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.