Slow reads from dropbox mount

Help and Support
1

What is the problem you are having with rclone?

Slow reads from dropbox mount. rclone copy speeds are fine.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-76-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone \
    -v mount m2: /rclonmnts/m2 \
    --rc --rc-no-auth \
    --rc-addr="127.0.0.1:16459" \
    --allow-other \
    --vfs-fast-fingerprint \
    --no-checksum \
    --uid 0 \
    --gid 1077 \
    --umask 007

Env:

RCLONE_CONFIG="/rclonfigs/rclone.conf"
RCLONE_VFS_CACHE_MODE=full
RCLONE_VFS_CACHE_MAX_SIZE=16G
RCLONE_VFS_CACHE_MAX_AGE=1h
RCLONE_VFS_CACHE_POLL_INTERVAL=1m
RCLONE_CACHE_DIR="/mnt/nvme/rcloncache/3"
RCLONE_POLL_INTERVAL=1m
RCLONE_DIR_CACHE_TIME=72h
RCLONE_VFS_READ_CHUNK_SIZE=16M
RCLONE_VFS_READ_CHUNK_SIZE_LIMIT=2G
RCLONE_BUFFER_SIZE=32M
RCLONE_VFS_READ_AHEAD=100M
RCLONE_PASSWORD_COMMAND="/usr/bin/op read op://secrets/rclone/password"

The rclone config contents with secrets removed.

[db]
type = dropbox
client_id = xxxx
client_secret = xxxx
token = xxxx

[mrawunion2]
type = union
upstreams = db:z01:nc db:z03:nc db:z04:nc db:z05:nc db:z06

[m2]
type = crypt
remote = mrawunion2:
filename_encryption = standard
filename_encoding = base64
password = xxxx

Is there anything I can change to improve reads? I don't upload using the mount, only rename and move, hence uploads are not important.

2

Could you be more precise and try to quantify it a bit?

Clean cache before:

umount /rclonmnts/m2
rm -rf /mnt/nvme/rcloncache/3/*

and, e.g.:

time rclone copy m2:path/to/file .

time cp /rclonmnts/m2/path/to/file .

You are using union and it comes at the cost of many operations slowdown as 5 remotes (even when like in your case it is the same one) have to be accessed. It should not affect copy though.

3

Be mindful cleaning the cache as that would cause data loss for anything not uploaded.

What was the reason to change this? If you change it from the default, it'll cause some delay in sequentially reading larger files as you have to 'ramp' up from 16M requests higher which means more API calls, more latency, etc.

What's the reason for this change?

Why did you change these values as well?

4

Thanks for catching that - I was thinking about copy from remote and completely forgot that writes are cached too.

1 Like
5

26 MiB/s if I copy anything from the mount to /dev/null, 70 MiB/s if I rclone -vP copy

6

times:

cp:
0.03s user 1.05s system 1% cpu 1:21.82 total

rclone copy:
8.34s user 5.09s system 52% cpu 25.744 total

1.6 GiB file

7

I removed the buffer flag and all vfs flags except RCLONE_VFS_CACHE_MODE=full. Speed does not improve at all

8

This is specifically why we ask for logs as you are comparing two things that shouldn't be compared.

A copy uses a multi threaded download.
A mount does not.

So that makes sense.

rclone copy DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv /data -vv
2023/07/17 20:11:02 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/07/17 20:11:02 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "copy" "DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv" "/data" "-vv"]
2023/07/17 20:11:02 DEBUG : Creating backend with remote "DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv"
2023/07/17 20:11:02 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/07/17 20:11:02 DEBUG : fs cache: adding new entry for parent of "DB:jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv", "DB:"
2023/07/17 20:11:02 DEBUG : Creating backend with remote "/data"
2023/07/17 20:11:02 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Need to transfer - File not found at Destination
2023/07/17 20:11:02 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: Starting multi-thread copy with 4 parts of size 358.812Mi
2023/07/17 20:11:02 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 1/4 (0-376242176) size 358.812Mi starting
2023/07/17 20:11:02 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 4/4 (1128726528-1504953150) size 358.798Mi starting
2023/07/17 20:11:02 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 3/4 (752484352-1128726528) size 358.812Mi starting
2023/07/17 20:11:02 DEBUG : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv: multi-thread copy: stream 2/4 (376242176-752484352) size 358.812Mi starting
^C2023/07/17 20:11:03 INFO  : Signal received: interrupt
2023/07/17 20:11:03 INFO  : Dropbox root '': Committing uploads - please wait...
2023/07/17 20:11:03 INFO  : jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv.zunimin3.partial: Removing failed copy
2023/07/17 20:11:03 INFO  : Exiting...

You are comparing 4 threads in a copy vs a single thread in a mount.

9

Sorry about the lack of logs. So in a nutshell, there is nothing wrong with my flags? Speed cannot be improved further?

10

If 4 threads is giving you 70, a single thread on 26ish seems quite good.

I've never noticed any problems on my mount streaming as 26MiB/s is quite good.

11

I see, it's just weird because I moved from gdrive to dropbox, and gdrive would do 70-80 MiB/s without any problems. I assumed there are some dropbox specific tweaks for mounts. Thanks for the help

12

I always had far better single threaded performance on GDrive compared to Dropbox but zero problems ever streaming anything.

1 Like
13

I'm reading UHD remux bluray from my dropbox mount, no problem. No problem with single threaded download either, about the same speed as gdrive (i've a 2gb download fibre, 900mb upload). The main diff for me is the single upload speed, where drive was 2 to 3 time faster. Which can make sense since // chunck upload (for a single file) is not in the dropbox backend (yet).

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.