Rclone mount files gone (but not really gone)

[dr.mcgillicuddy]

Before I open an issue on this, I'd like to collect some thoughts on what I might be doing wrong.

this is a crypt mount of a google drive (team drive).

rclone mount remote: ~/rclone doesn't show my files anymore.

Instead, it mounts a volume with about 100 files called eg "@-position-left-55" or "!-position-right-1" or some variation of that. The files are all exactly 50 bytes.

None of the proper directories are listed, just these files.

rclone lsd remote: shows all the proper directories, however
rclone lsf remote: shows all the "position" files again

rclone serve webdav remote: shows even more "position" files (about 400) like the ones I mentioned above and all of my directories and files.

It doesn't appear my data is gone, at least. But I'm stumped.
Just upgraded to 1.51.0.

$ rclone version
rclone v1.51.0
os/arch: darwin/amd64
go version: go1.13.7

[Animosity022]

If you use the fields in the question template, that helps solve an issue quicker.

Can you please include a debug log?

Thanks!

[thestigma]

@dr.mcgillicuddy

Instead, it mounts a volume with about 100 files called eg "@-position-left-55" or "!-position-right-1" or some variation of that. The files are all exactly 50 bytes.

I believe these weird files are created if you ever run the developer test command for checking what characters are invalid on the backend ( forget what the exact command is..). It doesn't clean up after itself. I was scratching my head when I saw this too (a good while after I ran the test out of curiosity). You can just delete that folder manually.

I'm not sure why those test files would impact your mount, but if you haven't made other changes too then chances are that if you just clean it up things will get back to normal.

[dr.mcgillicuddy]

I'm not sure why those test files would impact your mount, but if you haven't made other changes too then chances are that if you just clean it up things will get back to normal.

Unfortunately, that's not an option, as my mount shows only these files, and none of my actual data.

I have also tried re-creating my rclone.conf, refreshing all tokens, etc. No difference.

Here's a paste of my debug info:

https://pastebin.com/nH8VpAPG

There's not much there of use, I don't think.

[thestigma]

Can you not simply correct this via the web-interface of whatever backend you are using (ie Google Drive webpage for example) ? That way you'd be looking at the raw files stored which should circumvent any issues that occur in the mount-layer.

I'm not really sure what the ~/rclone part of your command does. That may be related to this.

[dr.mcgillicuddy]

@thestigma

I'm not really sure what the ~/rclone part of your command does. That may be related to this.

~/Desktop/rclone is the mount location

But you raised an interesting point, which did work. I went to the web interface, and I copied all of the subdirectories to a new team drive, then re-created my rclone.conf for that new team drive. It worked. I can see all my files now.

But this is still a pretty big wtf for me. There are 394 "position" files of unknown origin on my old drive. Moreover, why would rclone only show those "position" files, and not my actual data, even though my actual data still existed? And moreover, moreover, why would I see different contents of the remote with rclone mount and rclone serve webdav?

Is rclone mount limited to 400 objects, by any chance?

[thestigma]

@dr.mcgillicuddy

Is rclone mount limited to 400 objects, by any chance?

No, definitely not.
And I'm not sure why this would happen - but that test is meant for development so it might cause weird behaviors in how the mount tries to read the contents. They are just files however - and as far as i can tell the test just uploads some files and sees what happens to them. Simply removing them should revert everything. Moving everything to a whole other Teamdrive should be unnecessary overkill.

I have seen those files before so they are not of unknown origin - they are from rclone. Some spesific illegal characters test I believe in some rather obscure command I came across at some point. Can't quite remember what the command was though as it wasn't terribly useful to me - so I kind of dumped that info from my brain Ask NCW if you are very curious about the spesific origins. I can only assume you ran that command at some point by mistake or maybe a typo.

[dr.mcgillicuddy]

@thestigma

that test is meant for development

I assume you mean the test for invalid characters, but I don't know what that test is or how to invoke it.
@ncw any thoughts?

[Animosity022]

Can you step through the issue and the steps/expected output?

What rclone ls on a specific remote:path show?
What is being shown on the same directory on the mount?

Can you capture that on the debug level and share the output?

The debug log you shared shows many files there.

2020/02/02 19:06:12 DEBUG : 4-position-left-34: Attr:
2020/02/02 19:06:12 DEBUG : 4-position-left-34: >Attr: a=valid=1s ino=0 size=50 mode=-rw-r--r--, err=<nil>
2020/02/02 19:06:12 DEBUG : 5-position-left-35: Attr:
2020/02/02 19:06:12 DEBUG : 5-position-left-35: >Attr: a=valid=1s ino=0 size=50 mode=-rw-r--r--, err=<nil>
2020/02/02 19:06:12 DEBUG : 6-position-left-36: Attr:
2020/02/02 19:06:12 DEBUG : 6-position-left-36: >Attr: a=valid=1s ino=0 size=50 mode=-rw-r--r--, err=<nil>
2020/02/02 19:06:12 DEBUG : 7-position-left-37: Attr:
2020/02/02 19:06:12 DEBUG : 7-position-left-37: >Attr: a=valid=1s ino=0 size=50 mode=-rw-r--r--, err=<nil>
2020/02/02 19:06:12 DEBUG : 8-position-left-38: Attr:
2020/02/02 19:06:12 DEBUG : 8-position-left-38: >Attr: a=valid=1s ino=0 size=50 mode=-rw-r--r--, err=<nil>
2020/02/02 19:06:12 DEBUG : :-position-left-3A: Attr:

[dr.mcgillicuddy]

Can you step through the issue and the steps/expected output?

1. Run rclone mount
2. Open newly mounted disk
3. Expected: See all my files and directories; Actual: See 93 files named something like "4-position-left-34" and see zero directories

What rclone ls on a specific remote:path show?

rclone ls remote: shows all of my uploaded files and subdirectories and 395 "position" files and some of these files the name of which is just like 50 "a"s. Example:

Excluding my actual files:

What is being shown on the same directory on the mount?

Only the position files. No directories are shown. None of my data is shown. Only 93 (note the different count) files. "Héroique" is not mine either haha

Can you capture that on the debug level and share the output?

That is what is in the paste (you've included a snippet).

The debug log you shared shows many files there.

Yes, it does. haha Just not files that I put there. Note "size=50". They appear to just be 50 (pseudo)random bytes.

[ncw (Nick Craig-Wood)]

@dr.mcgillicuddy

But this is still a pretty big wtf for me. There are 394 "position" files of unknown origin on my old drive. Moreover, why would rclone only show those "position" files, and not my actual data, even though my actual data still existed? And moreover, moreover, why would I see different contents of the remote with rclone mount and rclone serve webdav ?

Those files were definitely created by rclone info which is used in development. I should probably hide it a bit better! It isn't destructive though but it can create files which cause listings to go wrong which might be what happened here.

[dr.mcgillicuddy]

So basically it just created so many files that some things were no longer being displayed? Moving all files larger than 50 bytes to a new bucket was easy and solved the problem.

I'm just confused as to why mount showed 98 files when there were closer to 400.

[ncw (Nick Craig-Wood)]

@dr.mcgillicuddy

So basically it just created so many files that some things were no longer being displayed? Moving all files larger than 50 bytes to a new bucket was easy and solved the problem.

It created files with crazy names - what I expect happened is one of these names couldn't be accepted by FUSE so it stopped the listing there. I expect all the files were really there though.

[dr.mcgillicuddy]

All the files were in fact there. There was no data loss. I would never expect rclone to do that. I trust it.

I was just trying to figure out why it behaved the way it did since it seemed like every way I interacted with the storage (web interface, command line, mount, webdav) showed completely different results.

I'll go run rclone info a few more times and see what happens next (just kidding)

[ncw (Nick Craig-Wood)]

@dr.mcgillicuddy

I'll go run rclone info a few more times and see what happens next (just kidding)

If you want to run it, then run it on a subdirectory, eg rclone info drive:test-info - that is what I do.

I suspect it was probably the very long filenames rclone info creates that stopped the FUSE listing working. Drive can have 8k long file names which is well above what FUSE can manage. In fact I put a patch into 1.51 to work-around this.