Do you have a RAM disk that you could try setting the cache to? And copy the file to /dev/null
?
Will help to figure out if the problem is with the disk or rclone.
Do you have a RAM disk that you could try setting the cache to? And copy the file to /dev/null
?
Will help to figure out if the problem is with the disk or rclone.
Ok, so I've created a RAM disk and setup cache there, same results (even a bit worse) and write/read show much higher results.
- Test trying to copy from mount directly
sudo rsync --progress Movie.mkv /dev/null -P
Movie.mkv
2,503,611,382 100% 7.91MB/s 0:05:01 (xfr#1, to-chk=0/1)
Same file with rclone copy
Transferred: 2.332G / 2.332 GBytes, 100%, 32.423 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1m20.9s
Write speeds on RAM disk
sudo dd if=/dev/zero of=/tmp/ramdisk/zero bs=4k count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB, 391 MiB) copied, 0.186891 s, 2.2 GB/s
Read speeds on RAM disk
sudo dd if=/tmp/ramdisk/zero of=/dev/null bs=4k count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB, 391 MiB) copied, 0.144923 s, 2.8 GB/s
Ok, as a catch-up, can you show the latest mount settings being used?
Also, reading online, it looks like ramdisk doesn't support sparse files, so you may need to create a block device from RAM and format it with ext4. Something like (adjust accordingly for your requirements):
mount -t tmpfs -o size=200M tmpfs temp/
cd temp/
dd if=/dev/zero of=disk.img bs=1M count=199
losetup /dev/loop0 disk.img
mkfs.ext4 /dev/loop0
mount /dev/loop0 temp2/
Also, paste the DEBUG logs again obtained from performing this test.
Ok, I did create the loop
tmpfs 10G 69M 10G 1% /temp
/dev/loop0 9.6G 37M 9.1G 1% /temp2
Then mounted on /temp2 and ran the same test to go back to the beginning, not going beyond 10/11M.
sudo rsync --progress /mnt/rclone/Movie.mkv /dev/null -P
Movie.mkv
2,503,611,382 100% 10.54MB/s 0:03:46 (xfr#1, to-chk=0/1)
The mount settings for this test and the last one are the same, just changing the cache location and size of the original mount I normally use.
rclone: Version "v1.53.1" starting with parameters;
--user-agent=cere
--allow-other
--dir-cache-time 1000h
--poll-interval 15s
--umask 002
-cache-dir=/temp2/
--vfs-cache-mode full
--vfs-cache-max-size 10G
--log-level INFO
--log-file=/home/cere/rclonemount.log
--vfs-cache-max-age=336h
--config /home/cere/.config/rclone/rclone.conf
akira:UN /mnt/rclone/
I'd test without the mount all together and see your speeds.
Just run a rclone copy with a single file and download it and see if that gets you decent speeds as it may not be the mount or anything else other than you have bad peering to Google's API.
I think that experiment has already been done earlier in the thread:
Oops, I see that now. Thanks!
Yip, I have tested with rclone copy and it doesn't happen the issue is just with the mount.
I have even tried this docker but issue still persists
Can you do this test with rclone copy just to test like for like?
I did the test with the same file and rclone copy, is on the same comment. I don't have the issue there, see below;
Is that from the mount or remote? I believe @ncw is asking you test using rclone copy from the mount instead of directly from the remote.
Oh my bad I misunderstood @ncw request (that was from the remote)
Here is the output of doing both again
From Remote
rclone copy remote:Movie.mkv /home/cere/ -P
Transferred: 2.332G / 2.332 GBytes, 100%, 31.100 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1m24.3s
From Mount
rclone copy /mnt/rclone/Movie.mkv ~/descargas/ -P
Transferred: 2.332G / 2.332 GBytes, 100%, 10.464 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3m48.1s
Edit: I am currently on Kernel 5.7.0 and I have installed;
linux-image-4.19.0-10-amd64
linux-image-4.19.0-11-amd64
linux-image-5.7.0-0.bpo.2-amd64
I've tried changing my GRUB_DEFAULT from 0 to 1 and 2 (then update the grub) and reboot and no matter what it boots on 5.7.0
Thank you.
The problem is very clear there...
If you go to --vfs-cache-mode writes
then the problem disappears - is that correct?
I'm assuming for all the tests you are doing this with an uncached file. I would expect if you do the test twice in a row that it should be much quicker as the second should use the cached version with --vfs-cache-mode full
If it isn't much quicker then that is pointing to your disks being slower than your network which is really odd!
So can you try the rclone copy from the mount twice in a row without clearing the cache?
Can you also try it with --multi-thread-streams 1
on the rclone copy
command to see if that makes any difference?
It would be worth collecting IO statistics while it is running - you can run vmstat 5
in another terminal window for example
Thanks for your response @ncw
If you go to
--vfs-cache-mode writes
then the problem disappears - is that correct?
No, I talked to fast... when I did the test with the test I didn't remove the old file so it started at 100+MB/s but then went down (I've done it again with --vfs-cache-mode writes and is not going beyond 10/11MB/s)
I would expect if you do the test twice in a row that it should be much quicker as the second should use the cached version with
--vfs-cache-mode full
rclone copy /mnt/rclone/Movie.mkv ~/ -P
Transferred: 2.332G / 2.332 GBytes, 100%, 10.440 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3m55.3s
rclone copy /mnt/rclone/Movie.mkv ~/ -P
Transferred: 2.332G / 2.332 GBytes, 100%, 364.509 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 6.6s
Can you also try it with
--multi-thread-streams 1
on therclone copy
command to see if that makes any difference?
No difference when using flag --multi-thread-streams 1
If it isn't much quicker then that is pointing to your disks being slower than your network which is really odd!
If it was the disks I guess I could not download via web browser or torrents at solid 90-95MB/s or use rclone copy from a remote and get above 30MB/s for a single file no?
The only thing I can think of by reading other users problems is the Kernel. I've been able to go back to 4.19.0-11-amd64 but the issue still remains. Is there any specific Kernel for Debian I can try?
Correct.
Hmm if it is a kernel problem with FUSE then the Second time (file cached)
test would be slow too. So I don't think it can be that.
If it was a kernel networking problem then the use rclone copy from a remote and get above 30MB/s for a single file
would be slow, so it can't be that.
So what can it be???
Can you try your mount without the --bwlimit-file 30M
flag? I know you said you tried that above, but I'd like to see the speed measurements if possible. That is the only flag which is the slightest bit unusual.
UPDATE: FYI @ncw , I have tested with plexdrive and speeds when accessing files from the mount are the same 10/11MB/s. So could it be FUSE? (I don't know if they share anything else apart from that)
Can you try your mount without the
--bwlimit-file 30M
flag? I know you said you tried that above, but I'd like to see the speed measurements if possible. That is the only flag which is the slightest bit unusual.
Same file uncached from mount without --bwlimit-file flag
rclone copy Movie.mkv ~/ -P
Transferred: 2.332G / 2.332 GBytes, 100%, 7.692 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 5m10.4s
eno1 / traffic statistics
rx | tx
--------------------------------------+------------------
bytes 2.45 GiB | 21.01 MiB
--------------------------------------+------------------
max 101.41 Mbit/s | 873 kbit/s
average 65.65 Mbit/s | 550.73 kbit/s
min 60 kbit/s | 6 kbit/s
--------------------------------------+------------------
packets 1802883 | 297669
--------------------------------------+------------------
max 8656 p/s | 1476 p/s
average 5634 p/s | 930 p/s
min 111 p/s | 6 p/s
--------------------------------------+------------------
time 5.33 minutes
Same file but from remote
rclone copy -P "akira:Movie.mkv" ~/
Transferred: 2.332G / 2.332 GBytes, 100%, 27.859 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1m34.0s
eno1 / traffic statistics
rx | tx
--------------------------------------+------------------
bytes 2.45 GiB | 25.05 MiB
--------------------------------------+------------------
max 350.54 Mbit/s | 3.89 Mbit/s
average 219.32 Mbit/s | 2.19 Mbit/s
min 50 kbit/s | 16 kbit/s
--------------------------------------+------------------
packets 1780063 | 370426
--------------------------------------+------------------
max 29534 p/s | 6923 p/s
average 18542 p/s | 3858 p/s
min 91 p/s | 11 p/s
--------------------------------------+------------------
time 1.60 minutes
With rclone v1.50.2 and --vfs-cache-mode full the limitation of 10MB/s disappears (but the issue is to have to wait until the file is fully downloaded to open)
rclone copy Movie.mkv ~/ -P
Transferred: 2.332G / 2.332 GBytes, 100%, 27.202 MBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 1m27.7s
rx | tx
--------------------------------------+------------------
bytes 1.56 GiB | 14.72 MiB
--------------------------------------+------------------
max 396.54 Mbit/s | 4.18 Mbit/s
average 191.51 Mbit/s | 1.76 Mbit/s
min 20 kbit/s | 1 kbit/s
--------------------------------------+------------------
packets 1132738 | 217436
It could be...
Rclone had some problems with the kernel fuse implementation recently. I thought it was all fixed up...
You could try adding --async-read=false
to your rclone mount
- this will replicate the 1.50.2 FUSE system better.
it seems like v1.50.2 don't like --async-read=false flag
2020/10/06 21:55:34 Fatal error: unknown flag: --async-read
Use that for 1.53 and test as that parameter didn't exist in the lower versions.