Some steam apps wont be installed on rclone

Suspected Bug
1

What is the problem you are having with rclone?

I would like to get straight to the point. I know that I always can buy bigger disks, but I would like to address issue I have right now, because this is just one case how I would like to use rclone.
I have low free space on my SSD (30GB), and relatively low space on HDD (500GB). And I have NAS in local network were I have few TB free (3 for now).

I made set of rclone "caches".

  1. NAS: Main storage, where everything goes into.
  2. NAS: Rclone webdav share that, pointed to that location, and vfs cache full, with cache location at NAS SSD
  3. RIG: Rclone to the webdav, with vfs caching full at local HDD
  4. RIG: Rclone to local rclone, with vfs caching full at local SSD

Point is - I would like to have games I play regularly to be at fastest locations possible. It seems to work perfectly, when game was already installed and copied there.

But every time steam wants to update a game, or when I remove a game and try to install it back, at some point steam says 'Write error'.

I think this is related to low compatibility with WinFSP or something. Instead of this chain of rclones, I can have just one backend mount, and I will still get same results. I've tried different settings.

I can share my configs, but I actually tried tens of combinations so far, and I am willing to try any you would share with me. I doesn't matter which I use, I still will get write error at some point. I thought with vfs cache full I should get complete compatibility with "win fuse".

With same setup at my linux NAS, and sharing the location through cifs, steam works fine (just slow), so I assume this is issue related strictly to a windows version of rclone.

What is your rclone version (output from rclone version)

v1.55.x (including v1.55.1 and beyond [betas])
I tried few latests WinFSP, with clean uninstall, reboot, install, reboot.

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Its a windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

local.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --vfs-cache-mode full --no-modtime ....

The rclone config contents with secrets removed.

A log from the command with the -vv flag

There is no usefull data, there just success writes with different offsets in logs. With nothing noticeable even under -vvv flag steam just gives up. I can upload a log, but it will take several megabytes, cause it successfully downloads a game, and fails at install stage (when it's being moved from downloading to common folder).

Could you please check any setup with rclone mount local, and confirm that this is just for me, and not a bug. Please try with games that weight more than 10GB.

2

Without a full log, there's nothing to help out with.
Your rclone.conf is missing too.

3

Alright. I've actually might found the problem with at least one game. I am unable to copy this folder to any rclone mount (in Windows).

I tried manually, I can create New Folder, rename it to ad, and write some files to them. But when I'm trying to write this set of files, it fails. Sometimes with errors in log, sometimes not. But I never can read content of the folder.

unknown
unknown1920×1080 862 KB

I would prefer if someone will share config that works, because I've tried tens of them. I will make clean test this evening.

4

A screenshot doesn't help unfortunately as we need the log file.

What command are you running to make the log file?

The use case of running steam games on a mounted cloud storage seems strange to me so not sure who else would be using it. I don't know what would work or not work as we can't analyze the log file to see what's going on and what flags you are using so I can't even try to recreate it properly.

5

This was an issue with several game store clients when i played around with it.

6

The use case of running steam games on a mounted cloud storage seems strange to me so not sure who else would be using it.

I never said that I want to use it with clouds. And it's not just for games. I stumbled on this, but I would like to have stable fuse. I know this is not what rclone was designed for, but I think it might solve all the compatibility issues with Windows. It also might be an issue with WinFSP.

I want it to be a cache layer between my NAS (in LAN) and my laptops SSD. Sadly rclone is single project with WinFSP. In a future I would like to have proper local caching with my CephFS. And before I reinvent a wheel, I would like give rclone a shot.

I don't know what would work or not work as we can't analyze the log file to see what's going on and what flags you are using so I can't even try to recreate it properly.

Sorry for the delayed replay, I'm investigating logs and source code on daily basis. The reason I wrote so soon - I needed an affirmation that someone else experienced similar issues.

This was an issue with several game store clients when i played around with it.

This helps a lot. So far - it seems that this is an issue between rclone serve webdav and rclone mount webdav. I can create ad folder, and I can read its contents through chrome browser. But I can't get it through rclone ls or rclone mount.

2021/07/04 19:32:13 ERROR : Battlerite/bundles/ad: error listing: couldn't list files: XML syntax error on line 1: illegal character code U+0001
2021/07/04 19:32:18 ERROR : webdav root 'common': Statfs failed: couldn't list files: XML syntax error on line 1: illegal character code U+0001

I will share more details once I have isolated case on what's fails exactly.

7

News updates. It seems that with correct configuration most of the game works. I've read whole conversation /t/testing-for-new-vfs-cache-mode-features/16905/328, and after that I decided to set there values.

  1. Laptop win10, remote: webdav, vfs cache - local ssd. Network 1Gbit.
    D:\rclone mount -vv --local-no-check-updated --transfers 128 --buffer-size 4K --vfs-read-ahead 4092K --vfs-read-chunk-size 4096K --vfs-read-chunk-size-limit 64M --attr-timeout 120h --dir-cache-time 480h --poll-interval 5h --no-modtime -o UserName="Guest Room Gaming" -o GroupName="Users" --file-perms 0777 --dir-perms 0777 --vfs-cache-max-age 9000h --vfs-cache-max-size 25G --vfs-write-back 10s --vfs-cache-mode full --vfs-cache-poll-interval 1m --cache-dir C:\haze_cache --config "C:\Users\Guest Room Gaming.config\rclone\rclone.conf" webdav:common D:\haze\SteamLibrary\steamapps\common

Not every game able to download with speeds more than 50Mbit, will play with vfs-read-chunk-size limit more. Would like to keep it relatively small, to have low time-to-first-byte.

  1. NAS Arch, remote: local hdd, vfs cache: local ssd. Network 1Gbit.

rclone serve webdav
-v
--addr :8080
--user guest
--pass guest
--dir-cache-time 48h
--buffer-size 1M
--cache-dir /mnt/sources/ceph/merged/cache/webdav
--vfs-cache-mode full
--vfs-cache-max-age 240h
--vfs-cache-max-size 500G
--vfs-cache-poll-interval 1h
--vfs-write-back 30m
--no-modtime
--vfs-read-chunk-size 16M
--vfs-read-chunk-size-limit 1024M
--transfers 1
--umask 000
--no-update-modtime
/mnt/sources/drive/V2/steam

Current problems: (during all of them HDD, CPU, NET on laptop is busy less than 20%)

  1. Rclone mount with different param values, including default hangs from time to time: under read or write pressures. It hangs for few seconds, while webdav that I'm looking through google chrome works, fast, and I can download files/open dirs.
  2. Rclone mount REALLY slow at creating tons of small files. New game install begins, never-yet-existed path was created. Then steam creates all the subfolders, and touches all files. And then fills files with content (copies). It seems than rclone makes some extra lookup and waste time.
  3. There is no LRU on file parts, as was pointed /t/testing-for-new-vfs-cache-mode-features/16905/328
  4. There is no LFU/ARC for file items at all.
  5. I think that by default --vfs-read-ahead and --buffer-size should alight to a --vfs-read-chunk-size, because it could create 2 requests for upstream for single block of data. I can see performance benefit from (buffer-size + vfs-read-ahead = vfs-read-chunk-size * X[my X = 1])
  6. Rclone seems to loose file and dir attr on reboot. The design is fine, but:
    a) --vfs-used-is-size isn't usable when you have over 1M files, on GDrive - it will take HOURS to mount.
    b) opening directory or files after start makes a noticable delay (~200ms).
  7. The way cache-mode=full and vfsMeta stored and being updated causes slow drive to be a bottleneck very quick. I think that full fledged database such as CouchDB should improve performance.
  8. Now I will focus on why some games failed to install and why I can't create 'ad' folder on webdav mount.

Disclaimer: Although some points raise valid concerns, I understand that most of them aren't for everybody. And that rclone isn't designed to be used that way. I'm willing to merge few features, but not everything.)

8

hi,

if i understand your rclone setup, you are using vfs-cache-mode full twice, not sure how that affects performace

  • rclone serve webdav
  • rclone mount

also, curious, if you are running a rclone mount to play real-time games on the w10, then why use the NAS as a middle man?

item 2 - might be due to --transfers 1

9

You're right, that's what's happening.
I use gaming to stress-test setup, because modern games are really demanding on IOPS and throughput.

if i understand your rclone setup, you are using vfs-cache-mode full twice, not sure how that affects performace

I use vfs-cache-mode full on different machines. It affects performance, for sure. But with correct buffer sizes this isn't that bad.
End goal - I want to merge my CephFS, OpenZFS, MinIO into one merged endpoint. MergeFS isn't enough, because I want to have cache layers. It's just happens that rclone is single tool that have it, and supports Windows.

I use transfer = 1 at webdav serve, because one of sub-folders actually pointed at GDrive mount with --transfers 128, and writing on local disk is much much faster than sending data there.

Rclone mount taking ridiculous amount of time to bootstrap(while rclone serve doesn't have this issue), when it have tens of hundreds of files, so I'm trying to keep it's vfs small. (There I have mode=writes, writeback 1s, and vfs max age 10s)

item 2 - might be due to --transfers 1

Directory listing shouldn't hang even when just some files were written, it's clearly a bug. These operations should't block each other.

also, curious, if you are running a rclone mount to play real-time games on the w10, then why use the NAS as a middle man?

I have dataset of 600TB, some data on tapes, and I have only 3TB available in SSD (with bit-rot protection). For sure I could buy 2TB NVMe for games and solve this particular issue.

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.