Random folders are not visible by rclone for some time, but it's actually there

disconnect5852 · October 15, 2023, 10:20pm

What is the problem you are having with rclone?

I'm using idrive E2 with rclone, and sometimes i sync a big (1,8TB) folder to my local disk, and lately on sync logs i see that full folders are deleted with all files, which i didn't. Only i have the access, but now then this happened again, i've opened the web idrive e2 dashboard and navigated to the deleted folder, and it isn't deleted, its there (not in "show versions", but normally there). I'm running rclone on my windows gamer pc, on my linux notebook, and on my linux home sever, all have the same issue, that folder is not visible from rclone, same for rclone serve webdav, or rclone sync, or viewing with rclone ncdu.
I can even download the "deleted" files from web without problem.

I didn't touched that folder, and settings since months, nor the command lines.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.4.0-163-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.21.1
go/linking: static
go/tags: none

rclone v1.64.0

os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
os/kernel: 10.0.22621.2283 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.21.1
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

idrive E2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone ncdu idrivee2:/store/videos/
rclone sync idrivee2:/store/videos/GM9/gdt_data/ E:\videos\GM9\gdt_data\ --modify-window=10s --exclude-from e:\videos\GM9\rcloexclude.txt --buffer-size=512M --transfers 8 --max-transfer 725G --log-level INFO
rclone serve webdav idrivee2:/store--addr :8889 --dir-cache-time 3h --vfs-read-chunk-size=16Mi --vfs-fast-fingerprint --vfs-cache-mode=minimal --track-renames --user xxxxx--pass xxxxxx

The rclone config contents with secrets removed.

if i select the remote to edit it's like if would create a totally new remote on all machines, all with own local configs, but i can start new instances of rclone and see everything, except that folder
I've created a new access key, and set up remote with it, but works the same, the folder is missing, including the config can't be edited, but only replaced.

asdffdsa · October 16, 2023, 12:15am

please understand that we cannot see into your machine, so need to provide real evidence of what you claim.
so far, you have not posted any details.
you were asked for a debug log?
you were asked for a redacted config?

need to post that.

let's keep this simple, test on a single machine.
a debug log of rclone sync showing a file being deleted and screenshot of idrive showing that the single file exists.

what does that mean?

disconnect5852 · October 16, 2023, 10:44am

it means as i wrote. If i want to edit the idrive e2 remote, i get the screen like if i would createa totally new remote, asking for provider:

It seems, that folder is not visible in rclone, but if on an rclone mount i try to move the folder(downloaded on idrive website) to the mount, the folder appears in mount, and the file manager asks if i want to overwrite the files. Then if i unmount, close rclone mount, and re mount, the folder isn't there again, until i try to upload.

this is when i try to move the downloaded folder to mount: (now --vfs-cache-mode=off for this)

2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Opendir:
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/ (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Opendir: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Releasedir: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Releasedir: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=-2
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=-2
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Mkdir: mode=0770
2023/10/16 13:03:01 DEBUG : share/GM9/gdt_data/p/miko/p: Added virtual directory entry vAddDir: "hwtz"
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Mkdir: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Opendir:
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/hwtz/ (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Opendir: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Releasedir: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Releasedir: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Opendir:
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/ (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Opendir: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Releasedir: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Releasedir: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Opendir:
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/ (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Opendir: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /: Statfs:
2023/10/16 13:03:01 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: Releasedir: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p: >Releasedir: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: OpenEx: flags=0x0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Open: flags=O_RDONLY
2023/10/16 13:03:01 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Open: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >OpenEx: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Flush: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Flush: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Release: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Release: errc=0
2023/10/16 13:03:01 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /: Opendir:
2023/10/16 13:03:01 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /: >Opendir: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /: Releasedir: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /: >Releasedir: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: OpenEx: flags=0x0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Open: flags=O_RDONLY
2023/10/16 13:03:01 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Open: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >OpenEx: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Flush: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Flush: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Release: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Release: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Opendir:
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/hwtz/ (r), err=<nil>
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Opendir: errc=0, fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF1outside.avi: >Getattr: errc=0
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Releasedir: fh=0x1
2023/10/16 13:03:01 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Releasedir: errc=0

the file manager asks if i want to ovewrite, and same time the folder appears in mount. Then i skip overwrite, not moves on the next file in the folder:

2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: OpenEx: flags=0x0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:05:48 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Open: flags=O_RDONLY
2023/10/16 13:05:48 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Open: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi (r), err=<nil>
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi (r), err=<nil>
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >OpenEx: errc=0, fh=0x1
2023/10/16 13:05:48 DEBUG : /: Statfs:
2023/10/16 13:05:48 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Flush: fh=0x1
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Flush: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Release: fh=0x1
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Release: errc=0
2023/10/16 13:05:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /: Opendir:
2023/10/16 13:05:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:05:48 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2023/10/16 13:05:48 DEBUG : /: >Opendir: errc=0, fh=0x1
2023/10/16 13:05:48 DEBUG : /: Releasedir: fh=0x1
2023/10/16 13:05:48 DEBUG : /: >Releasedir: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: OpenEx: flags=0x0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:05:48 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Open: flags=O_RDONLY
2023/10/16 13:05:48 DEBUG : share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Open: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi (r), err=<nil>
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi (r), err=<nil>
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >OpenEx: errc=0, fh=0x1
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Flush: fh=0x1
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Flush: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Release: fh=0x1
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Release: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Opendir:
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >OpenFile: fd=share/GM9/gdt_data/p/miko/p/hwtz/ (r), err=<nil>
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Opendir: errc=0, fh=0x1
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz/TF2outside.avi: >Getattr: errc=0
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: Releasedir: fh=0x1
2023/10/16 13:05:48 DEBUG : /share/GM9/gdt_data/p/miko/p/hwtz: >Releasedir: errc=0

then again and again the same for each file.

Now after tryin to copy the folder to mount, it appeared in mount, after cancelling overwrite, now i renamed the folder in mount to "Hwtz asdf", closed rclone mount, and re mounted again. Now the folder appears correctly without the need of try to upload, but the old folder isn't, however it's empty already. But in ncdu both appear just like on web.
After this i closed rclone mount again restored the initial command line settings (--vfs-cache-mode=minimal --vfs-fast-fingerprint), and started again, now all folders appear correctly, same as on web.

As i remember, when similar happened 2 weeks ago, it was also a folder with only lower case characters, and whitespaces in it (of course not the first char), but for it i didn't do anything, just few days later the folder re-appeared for rclone.

kapitainsky · October 16, 2023, 11:36am

You still have not provided any details...

Also you talk about mount but not provided your mount command.

disconnect5852 · October 16, 2023, 11:57am

it looks the same from every rclone commands

rclone mount idrivee2:/store x: --buffer-size=2048M --max-transfer 725G --vfs-read-chunk-size=64Mi --vfs-fast-fingerprint --dir-cache-time 5m --vfs-cache-mode=minimal -v

Now when i looked the mount, another full lower case folder is not appearing, which follows the previous non-appearing folder in name sort, so the folder hame is "hwxa". So first "hwtz" was not appearing, but after if i rename to Hwtz it appears, but now the hwxa isn't appears. I bet if i rename it to other name with upper case start character, it will appear.

kapitainsky · October 16, 2023, 12:01pm

BTW

It causes 5 min delay before any changes on remote appear on your mount.

kapitainsky · October 16, 2023, 12:11pm

If you keep telling stories without providing any evidence I am afraid nobody can help.

disconnect5852 · October 17, 2023, 9:05pm

lol, if you want a screenshot from idrive e2 website about what i described, then i'll create it with gimp (since after the name changes the error is not present)... or not, it's not worth the time.

But about the config problem i took the screenshot, and i can take anytime... So idrive E2 remotes cant be edited, just recreated from scratch.

asdffdsa · October 17, 2023, 9:58pm

if you want to edit an existing remote, then edit it,

run rclone config
choose edit

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config

sorry, still not fully understanding what you need help with. we cannot see into your machine.
you are making vague statements, using too many rclone commands and this makes it impossible to help you.

please, now, need to create a minimal set of simple steps to re-produce your issue including rclone debug logs.
and as i mentioned, keep it simple, test on a single machine and try to avoid using rclone mount

disconnect5852 · October 18, 2023, 7:14am

Thats what i wrote and shown on screenshot, that i cant, it shows me to choose provider, and everything, to re-create the remote, not to edit.
I will make a video about it

kapitainsky · October 18, 2023, 7:43am

Do not make any video. Just provide data.

your description like above is not clear

Provide:

Content of your config file.
Any command you run with output
relevant logs

e.g.

rclone ls/lsd - to show remote content

rclone mount command

ls/dir /mount/point - to show what is visible (or not) in your mountpoint

Try to restrict output to problematic file/folder only - so it is easier to analyse.

command you use to rename file/folder

state of remote and mount after rename.

Do not provide any screenshots (they are often difficult to read and impossible to copy to test) but copy/paste actual content.

In general make people who try to help life easier:)

disconnect5852 · October 18, 2023, 8:08am

I did provide.

rclone config, select edit remote, select my idrive e2 remote, and then it look like if i selected new remote. For other remotes (gdrive, opendrive, mega), the editing is editing as it should. Only idrive e2 is the exception, where edit means new.

kapitainsky · October 18, 2023, 8:11am

where????

run

rclone config redacted and post output here. It was all explained in the Help and Support template you did not follow.

asdffdsa · October 18, 2023, 1:49pm

your screen image looks ok to me.
if you want to edit/change the provider, then type in a number or name of the provider.
if you do not want to edit/change the provider, then press enter to move onto the next item.

kapitainsky · October 18, 2023, 2:07pm

Not really bringing anything new to this thread but from usability perspective I think it would be nicer if information "Press Enter for the default (Storj)." was displayed not at the beginning of this very long list but at the end - where user prompt is. It would be less prone to be confusing.

At the moment:

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own string value.
Press Enter for the default (Storj).
 1 / Amazon Web Services (AWS) S3
   \ (AWS)
 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
   \ (Alibaba)
...
29 / Any other S3 compatible provider
   \ (Other)
provider>

I think better way:

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own string value.
 1 / Amazon Web Services (AWS) S3
   \ (AWS)
 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
   \ (Alibaba)
...
29 / Any other S3 compatible provider
   \ (Other)

Press Enter for the default (Storj).
provider>

I might try to change it as it should be easy cosmetic code edit.

disconnect5852 · November 6, 2023, 10:48pm

sorry i can't share the real folder and filenames, but happened again, again with random folder in a huge folder (~900 folders), but nothing changed anything around it.

It became missing in sync, mounts, both on windows, and linux, serve webdav, but visible on idrivee2 website.
BUT its also visible in list if i do a simple:

rclone ls remote:bucket/this/that/another/again/hugefolder/

but nothing appears if i do a

rclone ls remote:bucket/this/that/another/again/hugefolder/ --include "theexactcase firstpartnameoffolder*"

nor without "", or with ''.

And again, it happens a full lowercase folder name with space in it, starting with character from the end of the alphabet (now t, last time it was z).

I can see the contents of the folder anywhere if i mount the exact folder like this:

rclone mount "remote:bucket/this/that/another/again/hugefolder/theexactcase firstpartnameoffolderandlastpart/" x:

also can ls the contents in same way.
After mounting with -vv in every combination above, and open folder there are no errors, or warnings.

kapitainsky · November 7, 2023, 6:46am

This is how filtering works.

You have to use

"theexactcase firstpartnameoffolder*/*"

or

"theexactcase firstpartnameoffolder**"

more details in docs.

As for the other part of the problem until you share all details as per Help and Support template it is impossible to guess what is going on.

disconnect5852 · November 7, 2023, 8:47pm

there were many command lines, version information supplied too, with long description what happens, only not in a single post, but multiple.

Anyway, i've been thinking something that might solve the current instance of the problem, and it worked! I've just done a rclone backend cleanup-hidden remote:bucket/this/that/another/again/hugefolder/ and after that, the nearly invisible subfolder appeared everywhere, so it's maybe connected to versioning.
Now i'll disable the versioning on the bucket, and watch if this problem comes up again, or not.

kapitainsky · November 7, 2023, 9:06pm

if you posted all details than somebody maybe could figure it out earlier.

No. For example you were asked many times to share your rclone config. You never did. Instead you kept posting "descriptions" probably only you understood fully.

There are many people on this forum willing to help but they need some cooperation:)

Maybe next time.

PS. Thanks for sharing your solution. Maybe it will help others.

disconnect5852 · November 7, 2023, 9:13pm

A config with the problem? Here. But what can you do with this?

[idrivee2]
type = s3
provider = IDrive
access_key_id = xxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxxxxxxx
acl = private
endpoint = i6m4.fra.idrivee2-26.com