Cannot access some files stored on internet archive

Roman_Paholik · January 19, 2023, 7:12pm

What is the problem you are having with rclone?

Some file seem not to be accessible in some program. Using Retroarch with o2em libretro core which uses bios file o2rom.bin. I have mounted retroarch bios folder from internet archive. When I have that file on local folder without rclone, there is no problem reading that file.

the command I am using:

params="--no-modtime --attr-timeout 10h --dir-cache-time 10h --poll-interval 10h --vfs-cache-mode full --allow-non-empty --daemon"
rclone mount "archive:retroarch-bios" ~/.config/retroarch/system $params

It is strange that it works on my x86 machine now (Atari VCS).

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

os/version: ubuntu 22.10 (64 bit)
os/kernel: 6.0.0-cadmium (aarch64)
os/type: linux
os/arch: arm64
go/version: go1.19.4
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Internet Archive

The rclone config contents with secrets removed.

[archive]
type = internetarchive
access_key_id = deleted
secret_access_key = deleted

[myrient]
type = ftp
host = ftp.myrient.erista.me
tls = false
explicit_tls = true
pass = myUcMnWBKX9R-Gya--f8j0K26zYNvaWCqyqL
user = anonymous

[thumbnails]
type = http
url = https://thumbnails.libretro.com

Retroarch log

[INFO] [Environ]: SYSTEM_DIRECTORY: "/home/wizzard/.config/retroarch/system/".
[libretro WARN] [O2EM]: o2rom.bin not found, cannot load BIOS

ncw · January 20, 2023, 10:54am

Can you add a log from rclone with -vv when Retroarch tries to access the file please?

Roman_Paholik · January 20, 2023, 4:11pm

Thank you very much for the reply!

I mounted that share with -vv flag and this is what was logged:

2023/01/20 17:04:22 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "mount" "archive:retroarch-bios" "/home/wizzard/.config/retroarch/system" "-vv" "--log-file" "/tmp/rclone.log"]
2023/01/20 17:04:22 DEBUG : Creating backend with remote "archive:retroarch-bios"
2023/01/20 17:04:22 DEBUG : Using config file from "/home/wizzard/.config/rclone/rclone.conf"
2023/01/20 17:04:25 INFO  : Internet Archive item retroarch-bios: poll-interval is not supported by this remote
2023/01/20 17:04:25 NOTICE: Internet Archive item retroarch-bios: --vfs-cache-mode writes or full is recommended for this remote as it can't stream
2023/01/20 17:04:25 DEBUG : Internet Archive item retroarch-bios: Mounting on "/home/wizzard/.config/retroarch/system"
2023/01/20 17:04:25 DEBUG : : Root: 
2023/01/20 17:04:25 DEBUG : : >Root: node=/, err=<nil>
2023/01/20 17:10:05 DEBUG : /: Attr: 
2023/01/20 17:10:05 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2023/01/20 17:10:08 DEBUG : /: Lookup: name="o2rom.bin"
2023/01/20 17:10:10 DEBUG : /: >Lookup: node=o2rom.bin, err=<nil>
2023/01/20 17:10:10 DEBUG : o2rom.bin: Attr: 
2023/01/20 17:10:10 DEBUG : o2rom.bin: >Attr: a=valid=1s ino=0 size=1024 mode=-rw-rw-r--, err=<nil>

Roman_Paholik · January 23, 2023, 5:21pm

It is possible that given libretro core is opening that bios file some unusual way?

ncw · January 24, 2023, 9:11am

This shows that the file exists, but no attempt to open it or access it.

I guess.

Are you 100% sure it is looking in the right place?

You can try to strace it to see what error it gets back from the OS.

Roman_Paholik · January 24, 2023, 12:49pm

Yes, it am using the same folder with mounted files as I tried with file physically in that folder.
Also, it works with the same mount script on another computer (x86 vs arm, so also that libretro core version is a little different)
Will try that command, thank you.

Roman_Paholik · January 24, 2023, 5:14pm

I logged strace to file and this is what I found about that file:

write(3, "[INFO] [Content]: Updating firmw"..., 150) = 150
statx(AT_FDCWD, "/home/wizzard/.config/retroarch/system/o2rom.bin", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0664, stx_size=1024, ...}) = 0
statx(AT_FDCWD, "/home/wizzard/.config/retroarch/system/c52.bin", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0664, stx_size=0, ...}) = 0
statx(AT_FDCWD, "/home/wizzard/.config/retroarch/system/g7400.bin", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0664, stx_size=0, ...}) = 0
statx(AT_FDCWD, "/home/wizzard/.config/retroarch/system/jopac.bin", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0664, stx_size=0, ...}) = 0
write(3, "[INFO] [Core]: Using content: \"/"..., 87) = 87
write(3, "[INFO] [Core]: Content ran for a"..., 77) = 77
write(3, "[INFO] [Core]: Unloading core..\n", 32) = 32
write(3, "[INFO] [Core]: Unloading core sy"..., 40) = 40

write(3, "[INFO] [Content]: Content succes"..., 87) = 87
write(3, "[INFO] [Content]: Content loadin"..., 83) = 83
write(3, "[INFO] [Environ]: SET_PIXEL_FORM"..., 44) = 44
write(3, "[INFO] [Environ]: SET_INPUT_DESC"..., 41) = 41
write(3, "[INFO] [Environ]: SYSTEM_DIRECTO"..., 78) = 78
stat64("/home/wizzard/.config/retroarch/system/o2rom.bin", 0xffe86ca0) = 0
write(3, "[libretro WARN] [O2EM]: o2rom.bi"..., 62) = 62
write(3, "[INFO] [Content]: Removing tempo"..., 85) = 85
unlink("/tmp/4 en 1 Ligne (France).bin") = 0
write(3, "[ERROR] [Content]: Failed to loa"..., 42) = 42
write(3, "[INFO] [Core]: Content ran for a"..., 77) = 77
write(3, "[INFO] [Core]: Unloading core..\n", 32) = 32
write(3, "[INFO] [Core]: Unloading core sy"..., 40) = 40

ncw · January 25, 2023, 9:57am

Apparently the file was found OK here (the = 0) but

It gave the not found error which is weird.

This could be a permissions problem? Are you running the mount as the same user running the emulator?

Roman_Paholik · January 25, 2023, 11:06am

Yes, it is really odd, I mount that without root with just my user, also run all the programs as my user. I checked mounted file perms and I have the permissions to read and write it.

ncw · January 25, 2023, 11:08am

I would start tracing the source code of libretro now to see exactly what it is doing to give that warning message...

Roman_Paholik · January 25, 2023, 11:14am

Yes, I even tried to check that libretro core libretro-o2em/libretro.c at master · libretro/libretro-o2em · GitHub sources to find out how it reads the bios files, but I am not so good at C to find out what is wrong. Plus, I dont know why it does not work on my arm machine and works on x86 machine. I even commented some lines in the sources which I considered problematic and built that libretro core, but it did not help

ncw · January 25, 2023, 11:36am

The error comes from here

github.com

libretro/libretro-o2em/blob/3303cc15e4323280084471f694f6d34c78199725/libretro.c#L967-L972


      
          fill_pathname_join(bios_file_path, system_directory_c, bios_file_name, sizeof(bios_file_path));
          if (!path_is_valid(bios_file_path))
          {
             log_cb(RETRO_LOG_WARN, "[O2EM]: %s not found, cannot load BIOS\n", bios_file_name);
             return false;
          }

Which in turn calls path_is_valid

github.com

libretro/libretro-o2em/blob/3303cc15e4323280084471f694f6d34c78199725/libretro-common/file/file_path_io.c#L89-L92


      
          bool path_is_valid(const char *path)
          {
             return (path_stat_cb(path, NULL) & RETRO_VFS_STAT_IS_VALID) != 0;
          }

Which in turn calls path_stat_cb. This is indirected with lots of implementations for different OSes.

I think this is the code which runs

github.com

libretro/libretro-o2em/blob/3303cc15e4323280084471f694f6d34c78199725/libretro-common/vfs/vfs_implementation.c#L1019-L1033


      
             /* Every other platform */
             struct stat buf;
          
          
   if (!path || !*path)
                return 0;
             if (stat(path, &buf) < 0)
                return 0;
          
          
   if (size)
                *size             = (int32_t)buf.st_size;
          
          
   is_dir               = S_ISDIR(buf.st_mode);
             is_character_special = S_ISCHR(buf.st_mode);
          #endif
             return RETRO_VFS_STAT_IS_VALID | (is_dir ? RETRO_VFS_STAT_IS_DIRECTORY : 0) | (is_character_special ? RETRO_VFS_STAT_IS_CHARACTER_SPECIAL : 0);

which looks OK (provided it is the code which is actually running!).

As far as I can see rclone is working - the stat returns sensible results, but libretro rejects it for some reason.

I'm sorry I don't have time to dig any more into the C code.

Does it work on your ARM machine without using rclone mount?

Roman_Paholik · January 25, 2023, 12:01pm

Yes, it works on my arm machine without using rclone mount.

That is indeed interesting that it uses different code for different platforms, I may play with that and try to make a fix for that.

Or I will just not be using this emulator like this. I created an issue in that libretro project. Thank you very much for the help.

system · February 24, 2023, 12:01pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.