Trying to mount Icedrive on Mac M1 using rclone for ARM64 but it terminates after a few minutes

Help and Support
1

What is the problem you are having with rclone?

I am able to mount a Backblaze B2 bucket as a drive, and it works without issues; however, when trying to mount Icedrive, it mounts, but then after opening the drive, a few minutes later, rclone crashes.

I can see in the debug logs there are some status code 429 (too many requests).

I also see something about failing to write to cache for one of the files. The file in question, a video file, is about 303MB in size and is already uploaded to Icedrice.

I tried searching for 429's and WebDav/rclone, but nothing comes up. I also looked at cache settings, but am not yet sure what the problem is.

What is your rclone version (output from rclone version)

rclone v1.56.2
- os/version: darwin 11.5.2 (64 bit)
- os/kernel: 20.6.0 (arm64)
- os/type: darwin
- os/arch: arm64
- go/version: go1.16.8
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Icedrive, which uses WebDav.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount icedrive:iDrive /Users/james/iDrive --vfs-cache-mode full -vvv

The rclone config contents with secrets removed.

[remote]
type = b2
account = REDACTED
key = REDACTED

[icedrive]
type = webdav
url = https://webdav.icedrive.io/
vendor = other
user = REDACTED
pass = REDACTED

A log from the command with the -vv flag

$ date
Mon Oct  4 16:50:09 IST 2021

$ /usr/local/bin/rclone-arm64 mount icedrive:iDrive /Users/james/iDrive --vfs-cache-mode full -vvv
...
...
...
2021/10/04 16:56:26 DEBUG : pacer: low level retry 1/10 (error 429 Too Many Requests)
2021/10/04 16:56:28 DEBUG : pacer: low level retry 2/10 (error 429 Too Many Requests)
2021/10/04 16:56:30 DEBUG : pacer: low level retry 3/10 (error 429 Too Many Requests)
2021/10/04 16:56:32 DEBUG : pacer: low level retry 4/10 (error 429 Too Many Requests)
2021/10/04 16:56:34 DEBUG : pacer: low level retry 5/10 (error 429 Too Many Requests)
2021/10/04 16:56:36 DEBUG : pacer: low level retry 6/10 (error 429 Too Many Requests)
2021/10/04 16:56:38 DEBUG : pacer: low level retry 7/10 (error 429 Too Many Requests)
2021/10/04 16:56:40 DEBUG : pacer: low level retry 8/10 (error 429 Too Many Requests)
2021/10/04 16:56:42 DEBUG : pacer: low level retry 9/10 (error 429 Too Many Requests)
2021/10/04 16:56:44 DEBUG : pacer: low level retry 10/10 (error 429 Too Many Requests)
2021/10/04 16:56:44 INFO  : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: downloader: error count now 9: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:56:44 ERROR : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: failed to download: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:56:44 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.RangeSeek from -1 to 302202880 length -1
2021/10/04 16:56:44 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.Read at -1 length 4096 chunkOffset 302202880 chunkSize 134217728
2021/10/04 16:56:44 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.openRange at 302202880 length 134217728
2021/10/04 16:56:46 DEBUG : pacer: low level retry 1/10 (error 429 Too Many Requests)
2021/10/04 16:56:48 DEBUG : pacer: low level retry 2/10 (error 429 Too Many Requests)
2021/10/04 16:56:50 DEBUG : pacer: low level retry 3/10 (error 429 Too Many Requests)
2021/10/04 16:56:52 DEBUG : pacer: low level retry 4/10 (error 429 Too Many Requests)
2021/10/04 16:56:54 DEBUG : pacer: low level retry 5/10 (error 429 Too Many Requests)
2021/10/04 16:56:56 DEBUG : pacer: low level retry 6/10 (error 429 Too Many Requests)
2021/10/04 16:56:58 DEBUG : pacer: low level retry 7/10 (error 429 Too Many Requests)
2021/10/04 16:57:00 DEBUG : pacer: low level retry 8/10 (error 429 Too Many Requests)
2021/10/04 16:57:02 DEBUG : pacer: low level retry 9/10 (error 429 Too Many Requests)
2021/10/04 16:57:04 DEBUG : pacer: low level retry 10/10 (error 429 Too Many Requests)
2021/10/04 16:57:04 INFO  : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: downloader: error count now 10: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:04 ERROR : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: failed to download: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:04 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.RangeSeek from -1 to 302202880 length -1
2021/10/04 16:57:04 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.Read at -1 length 4096 chunkOffset 302202880 chunkSize 134217728
2021/10/04 16:57:04 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.openRange at 302202880 length 134217728
2021/10/04 16:57:06 DEBUG : pacer: low level retry 1/10 (error 429 Too Many Requests)
2021/10/04 16:57:08 DEBUG : pacer: low level retry 2/10 (error 429 Too Many Requests)
2021/10/04 16:57:10 DEBUG : pacer: low level retry 3/10 (error 429 Too Many Requests)
2021/10/04 16:57:12 DEBUG : pacer: low level retry 4/10 (error 429 Too Many Requests)
2021/10/04 16:57:14 DEBUG : pacer: low level retry 5/10 (error 429 Too Many Requests)
2021/10/04 16:57:15 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Screenshot 2021-09-28 at 3.42.34 PM.png not removed, freed 0 bytes
2021/10/04 16:57:15 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item debian-11.0.0-amd64-netinst.iso not removed, freed 0 bytes
2021/10/04 16:57:15 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item ubuntu-20.04.2-live-server-arm64.iso not removed, freed 0 bytes
2021/10/04 16:57:15 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov not removed, freed 0 bytes
2021/10/04 16:57:15 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item PreConference Course Voice Over.mp4 not removed, freed 0 bytes
2021/10/04 16:57:15 INFO  : vfs cache: cleaned: objects 5 (was 5) in use 1, to upload 0, uploading 0, total size 14.216Mi (was 14.216Mi)
2021/10/04 16:57:16 DEBUG : pacer: low level retry 6/10 (error 429 Too Many Requests)
2021/10/04 16:57:18 DEBUG : pacer: low level retry 7/10 (error 429 Too Many Requests)
2021/10/04 16:57:20 DEBUG : pacer: low level retry 8/10 (error 429 Too Many Requests)
2021/10/04 16:57:22 DEBUG : pacer: low level retry 9/10 (error 429 Too Many Requests)
2021/10/04 16:57:24 DEBUG : pacer: low level retry 10/10 (error 429 Too Many Requests)
2021/10/04 16:57:24 INFO  : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: downloader: error count now 11: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:24 ERROR : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: failed to download: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:24 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.RangeSeek from -1 to 302202880 length -1
2021/10/04 16:57:24 ERROR : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: too many errors 11/10: last error: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:24 ERROR : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: failed to kick waiters: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:24 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.Read at -1 length 4096 chunkOffset 302202880 chunkSize 134217728
2021/10/04 16:57:24 ERROR : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: failed to _ensure cache vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:24 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: vfs cache: failed to _ensure cache vfs reader: failed to write to cache file: 429 Too Many Requests is not out of space
2021/10/04 16:57:24 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: ChunkedReader.openRange at 302202880 length 134217728
2021/10/04 16:57:24 DEBUG : Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov(0x140004fc440): >_readAt: n=0, err=vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:24 ERROR : IO error: vfs reader: failed to write to cache file: 429 Too Many Requests
2021/10/04 16:57:24 DEBUG : /Cloud-Run-Selenium-noVNC-480p-Screen Recording 2021-10-01 at 10.14.47 AM.mov: >Read: n=-5
2021/10/04 16:57:24 DEBUG : webdav root 'iDrive': Destroy: 
2021/10/04 16:57:24 DEBUG : webdav root 'iDrive': >Destroy: 
2021/10/04 16:57:24 DEBUG : Not calling host.Unmount as mount already Destroyed
2021/10/04 16:57:24 DEBUG : Unmounted successfully
2021/10/04 16:57:24 DEBUG : rclone: Version "v1.56.0" finishing with parameters ["/usr/local/bin/rclone-arm64" "mount" "icedrive:iDrive" "/Users/james/iDrive" "--vfs-cache-mode" "full" "-vvv"]
2021/10/04 16:57:24 DEBUG : vfs cache: cleaner exiting

$ date
Mon Oct  4 16:57:28 IST 2021
2

Looks like your backend provider is telling you to slow down.

You can try limiting the transactions per second.

Maybe start at 1 and test adding 1 each time to see what works or does not work:

      --tpslimit float                       Limit HTTP transactions per second to this.
3

Thanks for the reply. At 1 transaction per second, --tpslimit 1, I'm still getting the same behavior. I assume going higher would just overrun the system even more.

What stands out to me is the line about failing to write to the cache. Is the cache located on my local system or is this referring to some public cache or intermediary cache?

Thank you,
James

4

Yes, you may need to go lower, perhaps much lower. The log you posted shows that rclone itself already had backed off to 1 request every 2 seconds, that is --tpslimit=0.5

It could also be that the webdav backend doesn’t handle the Retry-After property of the 429’s properly. It has a minimum value of 1 second, and I often see several seconds or minutes in OneDrive - sometimes even hours.

We can check this by getting some extra HTTP response information from the execution of this command:

rclone lsd icedrive:iDrive --dump headers -vv

If that doesn’t return any 429’s then try something a bit more demanding like:

rclone copy icedrive:iDrive/somefile.mov ./testfolder --ignore-times --dump headers -vv

No need to post the entire log file, I just need the app. 20 lines above “error 429” to see the entire HTTP response header (please redact secrets and tokens, if any)

5

Your interpretation of the log isn't entirely correct.

The log tells that rclone was unable to store/write the files in the cache
.. because it was unable to download them
.. because the Icedrive servers reponded with HTTP error code 429 (Too many requests)
...ten times in a row
...when rclone requested the files
...with minimum request speed (1 request every 2 seconds)

and yes the cache is located on your local system.

6

Hi Ole,

Thanks again for the follow up. I got Icedrive to mount by using the following command:

rclone mount icedrive:iDrive /Users/james/iDrive --vfs-cache-mode full -vvv --tpslimit .03125

The tpslimit of .03125, if I'm doing the math right, should be about 1 request every 32 seconds. I may at some point see how far I can push it, but for now it was able to sync the files I stored there.

I'm on the lifetime 5TB plan, and Icedrive doesn't yet have a mountable version for Mac OS, only Windows, so this solution should help get anyone by who wants to mount Icedrive as a drive on Mac OS until Icedrive comes up with their own solution similar to pCloud.

Thank you,
James

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.