I need tips for improving read speed of multiple small files

Help and Support
1

What is the problem you are having with rclone?

I recently started storing a couple of comics/manga and quickly realized that their reading responsiveness was quite bad, needing to open a new small file (<10mb .cbz files) every couple of minutes could take up to 20 seconds each, up to a certain point is understandable since this is a cloud storage but I'm still wondering if there is any room for improvement.

I have no problem with media consuming which is the main focus of this, bd remuxes usually start playing within 4 to 8 seconds and they are very responsive in general.

All of this under a gigabit connection, google drive as cloud service, linux mint as OS and an encrypted drive.

I'm not entirely sure where I got the parameters and values I'm using but it probably was copy and paste from somewhere until I got it working, there might be something that can be improved/fixed.

Run the command 'rclone version' and share the full output of the command.

  • rclone v1.54.0
  • os/arch: linux/amd64
  • go version: go1.15.7

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount *: * --allow-other --log-level INFO --log-file *.log --multi-thread-cutoff=250M --multi-thread-streams 8 --tpslimit 20 --tpslimit-burst 20 --poll-interval 15s --umask 002 --user-agent * --rc --rc-addr :5572 --rc-no-auth

The rclone config contents with secrets removed.

[Vault]
type = drive
client_id = *
client_secret = *
scope = drive
token = {"access_token":"*"}

[VaultCrypt]
type = crypt
remote = Vault:Vault
filename_encryption = standard
directory_name_encryption = true
password = *
password2 = *

A log from the command with the -vv flag

it's a normal mount and it work's fine, no logs of any error
2

Check out:

Rclone mount random slow speeds - Help and Support - rclone forum

3

I will red it in detail and try a couple of things that already caught my attention, but also wanted to ask, you mentioned a transition from gdrive to dropbox, ¿do you find it better in general or are there compromises? when I started my google workspace account I didn't consider dropbox, so, I'm kind of curious

4

Dropbox has no upload or download quotas.

5

I was in the same boat as you up until recently when I got fed up with google and their limits & price increases. A few months ago I switched over to dropbox and I haven't regret that decision. It's probably one of the best things I've done in a long time. As @Animosity022 said, dropbox has no limits or quotas on bandwidth. I'm dealing with 616TB storage. I've uploaded around 10TB in a day and have also downloaded about the same all in a 24 hour period without worrying about being cut off like google does with their daily limits. It's really refreshing I can finally get things done without those restrictions.

6

As far as I could find, Dropbox requires you to have at least 3 accounts for the unlimited storage plan, is that right?

Appart from the download/upload limits, do you notice a better performance in term of responsiveness with Dropbox?

7

Yes, they do require the 3 minimum and it's paid up front, unlike google where if you have 1 account, you pay for one. So on Dropbox if you only register one account they still bill you for 3.

Honestly, I noticed quite a bit better performance with dropbox and that's what also influenced my decision to switch away from google, besides the daily quotas they enforced. Now, as of a few weeks ago, I logged into my google to transfer more files out and I'm noticing a great improvement over there now. As of this writing, this evening I appear to be getting great speed and performance with both dropbox and google using rclone+encrypt.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.