Rclone and Yandex.Disk (2022-09 summary)

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2

  • os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
  • os/kernel: 10.0.19044.2075 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.18.6
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Yandex.Disk

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe mount YandexDisk: Y: --config "C:\Users\User\AppData\Roaming\rclone\rclone.conf" --vfs-cache-mode full --cache-dir "E:\rclone cache\Yandex" --no-check-certificate --no-console --log-file C:\SW\rclone\rclone_yDisk.log

The rclone config contents with secrets removed.

[YandexDisk]
type = yandex
token = {"access_token":"xxxxx","token_type":"OAuth","refresh_token":"xxxxx","expiry":"2023-09-15T14:10:30.1832003+03:00"}
client_id = xxxxx
client_secret = xxxxx

Previously created topics regarding the low speed of Yandex.Disk if it is mounted as a local disk using rclone

Rclone 25-50 Mbps slow transfer speed

Rclone copy large files to the cloud Yandex.Disk

Coping a lot of small files significantly slows down the uploading pace

Yandex-disk slow upload speed

Conclusions:

  1. most likely, there is always throttling from the server side. Throttling conditions are unknown, and Yandex does not provide an opportunity to find out them either in general or in relation to a specific personal user key;
  2. Yandex uses multithreading in its native applications, but did not specify it in the API documentation. As a result, it cannot be implemented in third-party applications;
  3. Yandex in its native applications uses a comparison of the hash sum of the uploaded file with the files in the server cache. For third-party applications, this feature is also missing;
  4. netsh int tcp set global autotuninglevel=normal can help someone. But not to me;
  5. maybe using --tpslimit can help someone. But in any case, this is not suitable for searching through tens of thousands of files and it will definitely be slower than the native client.

I asked a question to the support team to find out the reasons for the slow data transfer speed:

After mounting Yandex-Disk on one physical machine with Microsoft Windows 10.0.19044.2075 using rclone (the --vfs-cache-mode full key is used) with its own token (Yandex ID API — Yandex Technologies), almost immediately throttling up to hundreds of kilobytes per second and one element per second. With an actual Internet connection speed in the region of 80-90 Mbps. It's impossible to work.

The log contains errors like

2022/09/22 16:07:29 ERROR : /: Dir.Stat error: Get "
https://cloud-api.yandex.com/v1/disk/resources?limit=1000&path=disk%3A%2F
": dial tcp: lookup cloud-api.yandex.com: no such host
2022/09/22 16:07:29 ERROR : IO error: Get "
https://cloud-api.yandex.com/v1/disk/resources?limit=1000&path=disk%3A%2F
": dial tcp: lookup cloud-api.yandex.com: no such host

and

2022/09/20 02:24:58 ERROR : 2022 (...).jpg: vfs cache: failed to upload try #3, will retry in 40s: vfs cache: failed to transfer file from cache to remote: Put "
https://cloud-api.yandex.com/v1/disk/resources?path=%2F2022+%D0%9A%D0%BE%D0%BD%D1%82%D0%B5(...)
": dial tcp: lookup cloud-api.yandex.com: no such host
2022/09/20 02:24:58 ERROR : 2022 (...).png: Failed to copy: Put "
https://cloud-api.yandex.com/v1/disk/resources?path=%2F2022+%D0%9A%D0%BE%D0%BD(...)
": dial tcp: lookup cloud-api.yandex.com: no such host

and

2022/09/16 13:00:02 ERROR : Unsorted from Nextcloud/Content Department/Backups/phonerecs/MToxMDA1OTcwMjo2MzY0ODYxODAxOjA=.mp3: Failed to copy: [423 - DiskResourceLockedError] Resource is locked. (Ресурс заблокирован. Возможно, над ним выполняется другая операция.)
2022/09/16 13:00:02 ERROR : Unsorted from Nextcloud/Content Department/Backups/phonerecs/MToxMDA1OTcwMjo2MzY0ODYxODAxOjA=.mp3: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: [423 - DiskResourceLockedError] Resource is locked. (Ресурс заблокирован. Возможно, над ним выполняется другая операция.)

Our team planned to use Yandex.Disk to store together several hundred gigabytes of predominantly recognized PDF documents with an average size of 1.5 MB and images up to 20-30 MB in size with full-text search.

Support answer:

We guarantee the stable operation of only official Yandex solutions, as we do not take part in the development, configuration and support of third-party applications. Unfortunately, if you're having trouble using another app, we can't help you because we don't know how it works. You can use any official Yandex application.
We have clients for Windows, macOS and Linux, apps for Google Android and iOS, and of course a web version.
If none of the official solutions work for you, please tell us why. We will definitely take your comments into account when working on updates!

1 Like

That's a nice summary, thank you.

This error looks more like a DNS lookup problem than a server side throttling problem. Have you investigated that?

No, can you provide me with comprehensive instructions on this?

On you local machine, it's not getting a DNS result when it looks up cloud-api.yandex.com

Troubleshooting DNS clients | Microsoft Learn

I changed the DNS servers of my wired network connection to the following:

1.1.1.1
1.0.0.1
8.8.8.8
8.8.4.4
9.9.9.9
185.228.168.9
185.228.169.9
76.76.2.2
76.76.10.2
94.140.14.14
94.140.15.15
94.140.14.140
94.140.14.141

I cleared the local DNS cache, launched rclone, and again tried to transfer several files from one Yandex.Disk folder to another. I didn't notice any changes. Log with --vv key.

The DNS errors are not present in the log you shared so you did clear that up.

That's fixed the DNS errors which is good.

I guess the throttling problem is still there?

There are no errors in the log now, so if it is slow, it is either network problems or throttling at yandex.

1 Like