OneDrvie Business Mount Problem

[chun_lee]

What is the problem you are having with rclone?

I mounted Onedrive as a disk in my server, when I download big files, it works fine, but when I download many small files such as 100 1M-2M filses, it may comes 503 errors, I set vfs cache mode = writes, and read chunk = 5M, --tpslimit = 2, and use my own application apis, errors still happen.Is there anybody else have this problem?

What is your rclone version (output from rclone version)

latest version

Which OS you are using and how many bits (eg Windows 7, 64 bit)

centos 7.6

Which cloud storage system are you using? (eg Google Drive)

OneDrive Bussiness

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount onedrive:path path --tpslimit 2 --log-file=/root/rclone_mont.log --vfs-read-chunk-size 10M --vfs-read-chunk-size-limit 100M --vfs-cache-max-size 2G --allow-other --allow-non-empty --vfs-cache-mode writes --no-checksum --dir-perms 0777 --file-perms 0777 --vfs-cache-max-size 2G --umask 000 &

The rclone config contents with secrets removed.

I can connect it perfectly, so this is not the point

A log from the command with the -vv flag

2021/03/01 15:35:40 ERROR : One drive root 'onedrive': Statfs failed: about failed: serviceNotAvailable: Service unavailable
2021/03/01 17:19:56 ERROR : One drive root 'onedrive': Statfs failed: about failed: serviceNotAvailable: Service unavailable

[Animosity022]

What version are you running?
What's your rclone.conf look like without the secrets / passwords?

Can you share the debug log of the command above?

[chun_lee]

Q1:
A:
rclone v1.53.1

• os/arch: linux/amd64
• go version: go1.15

Q2：
A:
[api1_api]
type = onedrive
client_id = ...
client_secret = ...
token = {"..."}
drive_id = ...
drive_type = business

Q3:
A:

2021/03/02 15:32:26 DEBUG : /: Attr:
2021/03/02 15:32:26 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxrwx, err=
2021/03/02 15:32:26 DEBUG : pacer: low level retry 6/10 (error serviceNotAvailable: Service unavailable)
2021/03/02 15:32:28 DEBUG : pacer: low level retry 7/10 (error serviceNotAvailable: Service unavailable)
2021/03/02 15:32:29 DEBUG : : Statfs:
2021/03/02 15:32:29 DEBUG : : >Statfs: stat={Blocks:1342177280 Bfree:746795200 Bavail:746795200 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2021/03/02 15:32:29 DEBUG : /: Attr:
2021/03/02 15:32:29 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxrwx, err=
2021/03/02 15:32:31 DEBUG : pacer: Reducing sleep to 1.5s
2021/03/02 15:32:31 DEBUG : 乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4: ChunkedReader.Read at 679936 length 8192 chunkOffset 675840 chunkSize 134217728
2021/03/02 15:32:31 DEBUG : &{乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4 (r)}: >Read: read=4096, err=
2021/03/02 15:32:31 DEBUG : &{乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4 (r)}: Read: len=4096, offset=806912
2021/03/02 15:32:31 DEBUG : 乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4: ChunkedReader.Read at 688128 length 16384 chunkOffset 675840 chunkSize 134217728
2021/03/02 15:32:31 DEBUG : 乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4: ReadFileHandle.seek from 679936 to 806912 (fs.RangeSeeker)
2021/03/02 15:32:31 DEBUG : 乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4: ChunkedReader.RangeSeek from 704512 to 806912 length -1
2021/03/02 15:32:31 DEBUG : 乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4: ChunkedReader.Read at -1 length 4096 chunkOffset 806912 chunkSize 134217728
2021/03/02 15:32:31 DEBUG : 乾隆王朝.Qianlong.Dynasty.2003.1080p.WEB-DL.H264.AAC-FLTTH/乾隆王朝.Qianlong.Dynasty.2003.E02.1080p.WEB-DL.H264.AAC-FLTTH.mp4: ChunkedReader.openRange at 806912 length 134217728
2021/03/02 15:32:32 DEBUG : : Statfs:
2021/03/02 15:32:32 DEBUG : : >Statfs: stat={Blocks:1342177280 Bfree:746795200 Bavail:746795200 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2021/03/02 15:32:32 DEBUG : /: Attr:
2021/03/02 15:32:32 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxrwx, err=
2021/03/02 15:32:32 DEBUG : pacer: low level retry 1/10 (error serviceNotAvailable: Service unavailable)
2021/03/02 15:32:32 DEBUG : pacer: Rate limited, increasing sleep to 2s
2021/03/02 15:32:34 DEBUG : pacer: low level retry 2/10 (error serviceNotAvailable: Service unavailable)
2021/03/02 15:32:35 DEBUG : : Statfs:
2021/03/02 15:32:35 DEBUG : : >Statfs: stat={Blocks:1342177280 Bfree:746795200 Bavail:746795200 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2021/03/02 15:32:35 DEBUG : /: Attr:
2021/03/02 15:32:35 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxrwx, err=
2021/03/02 15:32:37 DEBUG : pacer: low level retry 3/10 (error serviceNotAvailable: Service unavailable)
2021/03/02 15:32:38 DEBUG : : Statfs:
2021/03/02 15:32:38 DEBUG : : >Statfs: stat={Blocks:1342177280 Bfree:746795200 Bavail:746795200 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=
2021/03/02 15:32:38 DEBUG : /: Attr:
2021/03/02 15:32:38 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxrwx, err=
2021/03/02 15:32:38 DEBUG : pacer: low level retry 4/10 (error serviceNotAvailable: Service unavailable)
Thank you a lot!

[Animosity022]

You are a few versions behind. Can you update and retest(pretty sure that won't fix it, but helps to rule things out).

Can you share the full debug log when you retest?

[chun_lee]

Thank you for answering, after updating to the latest, problem still there, here's a full log, you can search "retry" to see where it stops, it begins fine but after downloading 1.52GB, it stops, and download the same chunk over and over again.rclone_mont.log (3.5 MB)

[jcastill (Jose)]

Checked your log and on a quick glance it looks like a file is being read as usual, then OneDrive is pacing you so everything stalls.

In my own experience Microsoft is horrible at pacing unless you are using their own client (no luck if using Linux) as even mover.io gets paced under the same rules.

Try using the following flags, in my use it helps a lot in having the applications not get paced as much:

---tpslimit 2 --user-agent "ISV|rclone.org|rclone/v1.54.0"

Basically the first will limit how many transactions per second happen, this helps Microsoft avoid thinking you are trying to abuse the service, the second helps Microsoft know the application that is being used, and in theory avoid pacing as well.

Hope it helps

[chun_lee]

cool！It works!
And I think the point is --user-agent "ISV|rclone.org|rclone/v1.54.0", because I tried to limit the tpslimit before.

[jcastill (Jose)]

Glad it worked! definitely the user-agent flag helps a lot. In my use, I've seen tpslimit also helps a bit, might be placebo, but specially when reading a lot of small files it helps keeping Microsoft happy.

[chun_lee]

Thank you very much.
I find even there is no 503 problem, MS also limits you download speed after 1GB file transfered, Is this normal? I think 1GB maybe too small.

[chun_lee]

Here's a new log, you can see at the end of this log, it begins stalls again.
The log is over 10MB, I put it in my server:
http://la.imlc.tk:8880/down/p92q3ZSQJPU2

[ncw (Nick Craig-Wood)]

@jcastill

definitely the user-agent flag helps a lot.

Do you think rclone should be doing this automatically?

I don't think it is even mentioned in the docs