Advice on cloud provider for syncing small files

Help and Support
1

What is the problem you are having with rclone?

I started using rclone for a corporate backup on Google Drive of roughly 300GB made of a ton of small files. The problem regarding upload speed is not new in the community; I already identified several topics discussing the issue like this, this, this, and tried accordingly to:

  1. provide client_id and client_secret;
  2. increase the number of --transfers, --checkers, --tpslimit and --bwlimit;

As a result for these, speed has increased in the upload of large files, but the problem of small files is still painfully there, showing years in ETA.

Further suggestions like increasing --max-backlog and setting --fast-list don't seem to apply to my case, as I have a problem in pushing a single file of a few bytes in less than 30 seconds.

Therefore, I'm asking if anyone could recommend any specific cloud provider that can increase the thoughput of files in upload. The only pointer to this issue seems calisro's post (@calisro I summon you!), but I'm unsure whether the problem is addressed properly.

What is your rclone version (output from rclone version)

rclone v1.54.0 - os/arch: windows/amd64 - go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bits

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync Z:/path/to/local/folder remote:/backup/folder --transfers=40 --checkers=40 --tpslimit=10 --bwlimit=9M
2

hello,
i use wasabi, a s3 rclone, known for hot storage.
wasabi does not have all those gdrive limits in terms up upload, throttliing and api limits.
you have to pay for it tho.

1 Like
3

Generally if you don't want to be throttled on API calls / second or IOs / second then you need to choose a provider which charges you for it!

Of all the providers I've tried S3 is probably the fastest. Also pretty expensive. These two things may be correlated :wink:

B2 works well for this kind of load and isn't too expensive.

As far as I know drive is the only provider to have an uploads per second quota.

1 Like
4

Thanks for your suggestions; I'll go through a test in this weekend and I'll come back with some benchmarks :slight_smile:

5

TLDR: wasabi addresses the problem of slow upload for small files

At the moment, I'm using @asdffdsa's solution.
It appears that it's a cost effective solution, GDPR compliant (at least, for me that I'm an European citizen), provides with the common "security by default" features, and I'm not reading any strong reasons why not to chose it.

@ncw you might want to check it out, given that ingress and egress quotas are free. I can't really make a true comparison of network speed due my limited bandwidth.

But hey, I solved the problem with small files throughput! Thanks for the support you both.

1 Like
6

good to know.

about the network speed, i have a 1Gbps fiber optic internet connection.
rclone can easily saturate that with wasabi.

1 Like
7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.