Slow speeds downloading and uploading

Help and Support
1

What is the problem you are having with rclone?

I have started suffering slow speeds, is like is limited to 10-11Mbps.
I have a mount and if I use rsync it starts transferring ok but after a few second it goes down to 11Mbps.

rsync /mount/Movie.mkv /home/cere/descargas/ -P
Movie].mkv
1,126,203,392 9% 11.22MB/s 0:15:52

I have even tried removing the flag --bwlimit-file and rebooting without success and when doing an speed test I get the full gig.

What is your rclone version (output from rclone version)

v1.53.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The rclone config contents with secrets removed.

 ExecStart=/usr/bin/rclone mount --allow-other  --dir-cache-time 1000h --poll-interval 15s --umask 002 --cache-dir=/mnt/pool/cache --vfs-cache-mode full --vfs-cache-max-size 400G --log-level INFO  --log-file=/mnt/rclonemount.log --vfs-cache-max-age=336h --bwlimit-file 30M --config /home/username/.config/rclone/rclone.conf ```
2

You need to include a debug log.

also, just to be clear you are running rsync against your rclone mounted google drive? Why not use rclone sync instead?

3

I was trying to copy a file with rsync from my GD mount to a local folder to check the speeds on a single.

It starts a 30M (is my current bwlimit) but after a few seconds goes down to 10/11

When starts;

rsync /mount/Movie.mkv /home/cere/ -P
Movie.mkv
  1,175,748,608   9%  280.52MB/s    0:00:37

After 6 or 7 seconds

rsync /mount/Movie.mkv /home/cere/ -P
Movie.mkv
  3,711,991,808  30%   11.01MB/s    0:12:21

Re the logs, I have changed my mount --log-level to DEBUG but there isn't anything new a part from the INFO logs for vfs cache cleaned

2020/09/29 20:54:02 INFO  : vfs cache: cleaned: objects 2088 (was 2088) in use 0, to upload 0, uploading 0, total size 283.042G (was 283.042G)
2020/09/29 20:55:02 INFO  : vfs cache: cleaned: objects 2088 (was 2088) in use 0, to upload 0, uploading 0, total size 283.042G (was 283.042G)
2020/09/29 20:56:02 INFO  : vfs cache: cleaned: objects 2088 (was 2088) in use 1, to upload 0, uploading 0, total size 283.042G (was 283.042G)
2020/09/29 20:57:02 INFO  : vfs cache: cleaned: objects 2088 (was 2088) in use 0, to upload 0, uploading 0, total size 283.115G (was 283.042G)
2020/09/29 20:58:02 INFO  : vfs cache: cleaned: objects 2088 (was 2088) in use 0, to upload 0, uploading 0, total size 283.115G (was 283.115G)
2020/09/29 20:59:02 INFO  : vfs cache: cleaned: objects 2088 (was 2088) in use 0, to upload 0, uploading 0, total size 283.115G (was 283.115G)
4

I think you are pulling this file directly from your VFS cache and not from Gdrive. What storage are you using for your cache volume? /mnt/pool/cache? is it a thumb drive or something?

You can manually delete the file from the VFS cache location and try again.

5

Cheers for the advise but I get same behaviour deleting the file from cache and re-trying.

/mnt/pool/ is a secondary HDD of 1TB I use for the rclone cache and downloads.

Is there any way/command for clearing the whole cache?

6

@ookla-ariel-ride I did some more testing with my old mount settings and I don't have the issue so it has to be something to do with the cache or VFS mode as the HDD is capable of much more than 10M

/dev/sdb:
 Timing buffered disk reads: 550 MB in  3.01 seconds = 182.84 MB/sec

This is my old mount but I'd rather go back to VFS full, performance was much better when it worked

rclone mount --allow-other --allow-non-empty --tpslimit 16 --cache-tmp-wait-time=5m --buffer-size=64M --dir-cache-time=72h --umask 002 --log-level INFO --vfs-read-chunk-size-limit=1G --vfs-read-chunk-size=128M --log-file=/mnt/rclonemount.log --size-only --vfs-cache-max-age=675h --vfs-cache-mode writes --config /home/cere/.config/rclone/rclone.conf remote: /mnt/rclone/

With that mount goes smooth all the way at 1G

rsync /mnt/rclone/Movie.mkv /home/cere/descargas/ -P

Movie.mkv
  7,161,282,560  59%  112.09MB/s    0:00:42
7

I've gone back to the VFS full config and I still get the limitation to 10M but this time I've been able to grab the [DEBUG LOGS].(https://mega.nz/file/YpFT1CxK#iA8MGOFlhEk81AvfUJR8_YBT4z3yzZLw2yQ2m9kOaDk)
Would you be able to have a look to see if you can see anything obvious?

Cheers

8

I think we will need @ncw or other team members to help with this, but I will take a look.

Curious, what kernel are you using on your system?

9

Thanks @ookla-ariel-ride

Linux version 4.19.0-10-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.132-1 (2020-07-24)

Edit:
Same issue changing it to...

5.4.0-0.bpo.3-amd64
10

Is there a reason you need to use this option. As far as I know it's bad since it can copies files underneath the mount.

With the uploading part: Check what your server is outputting when it is still uploading that file. What the Mb/s are while uploading?

11

hi,
he was using that on the old mount, which did perform well.

based on the debug file, the new mount settings does not use it.

12

i know this is basic advice, that when i have a problem with rclone,
i test using the simplest possible command and then add flags.

rclone mount akira:UN /mnt/rclone

13

Yikes. I should of read everything beforehand. My bad!

Though --bwlimit-file 30M should look like this --bwlimit-file=30M

For some odd reason I tried to use the same function but it did not work unless I put the = sign there.

14

just tested, works for me.

15

it did work for me until recently, I used most of @Animosity022 settings and just changed the 16M for 30M.

16

Thanks for the advise @asdffdsa
I'm freeing up some space on my main HDD to set the cache there and if I'm still facing the issue I will start like that.

Edit;

This is very odd.... After doing the above the issue was still there so I've done another test and realised that the issue is just with the mount...
If I do the following I pass the 10M mark no problem....

rclone copy -P "remote:Movie.mkv" ~/
Transferred:   	    1.506G / 2.332 GBytes, 65%, 36.065 MBytes/s, ETA 23s

I give up for today :cry:

17

I don't see anything obvious there.

Could it be that the HD you are using to store the cache on is really slow?

How is it formatted? It needs to support sparse files so can't be FAT for example.

18

thanks for the response @ncw

I don't think the issue are the HD's, I've tried using both (system HD and Download HD) for cache with same results and disks can read/write at much better speeds than 10MB/sec

/dev/sda:
 Timing cached reads:   27560 MB in  1.99 seconds = 13858.22 MB/sec
 Timing buffered disk reads: 558 MB in  3.01 seconds = 185.43 MB/sec
/dev/sdb:
 Timing cached reads:   28744 MB in  1.99 seconds = 14458.70 MB/sec
 Timing buffered disk reads: 550 MB in  3.01 seconds = 182.93 MB/sec

In fact the issue is only with the mount. If I use rclone copy/move I get much better speeds so could it be the Kernel?
I had 4.19.0-10-amd64 when all this started and I've tried 5.4.0, 5.6.0 & 5.7.0 but I can't appreciate any improvement.

19

What is the disk partition format?

20

The format is EXT4 on both hard drives