Lsjson - lsjson when walking circular symlinks exits for that path instead of reporting the error and moving on

Suspected Bug
1

What is the problem you are having with rclone?

I have a tree structure on my local system like this:
download

folder b and c are circular symlinks pointing to each other. When I run lsjson with -R and -L flag, rclone encounters an error when parsing these symlinks. Now theoretically, lsjson should log these error on console and keep walking the system (./a/d and ./a/d/3 should get walked). But what happens is rclone exits for this directory (./a and child dirs). It does complete the walk for other dirs.

What is your rclone version (output from rclone version)

rclone v1.53.1

  • os/arch: linux/amd64
  • go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux Ubuntu 20.04 64 bit. Windows 10 x64. (replicable on both)

Which cloud storage system are you using? (eg Google Drive)

Local System

The command you were trying to run (eg rclone copy /tmp remote:tmp)

./rclone lsjson ~/Desktop/testdata/ -RL

The rclone config contents with secrets removed.

No Config is required

A log from the command with the -vv flag

2020/11/06 11:58:14 DEBUG : rclone: Version "v1.53.1" starting with parameters ["./rclone" "lsjson" "/home/manish/Desktop/testdata/" "-RLvv"]
2020/11/06 11:58:14 DEBUG : Creating backend with remote "/home/manish/Desktop/testdata/"
2020/11/06 11:58:14 NOTICE: Config file "/home/manish/.config/rclone/rclone.conf" not found - using defaults
[
{"Path":"1","Name":"1","Size":9,"MimeType":"application/octet-stream","ModTime":"2020-11-05T14:28:54.001243583+05:30","IsDir":false},
{"Path":"a","Name":"a","Size":-1,"MimeType":"inode/directory","ModTime":"2020-11-05T14:30:09.998220527+05:30","IsDir":true},
{"Path":"e","Name":"e","Size":-1,"MimeType":"inode/directory","ModTime":"2020-11-05T22:30:11.928621662+05:30","IsDir":true},
{"Path":"e/4","Name":"4","Size":10,"MimeType":"application/octet-stream","ModTime":"2020-11-05T22:30:07.051978492+05:30","IsDir":false},
{"Path":"e/f","Name":"f","Size":-1,"MimeType":"inode/directory","ModTime":"2020-11-05T22:30:18.741237589+05:30","IsDir":true}2020/11/06 11:58:14 ERROR : a: error listing: stat /home/manish/Desktop/testdata/a/c: too many levels of symbolic links
,
{"Path":"e/f/5","Name":"5","Size":12,"MimeType":"application/octet-stream","ModTime":"2020-11-05T22:30:18.549223589+05:30","IsDir":false}2020/11/06 11:58:14 DEBUG : 2 go routines active
2020/11/06 11:58:14 Failed to lsjson with 2 errors: last error was: error in ListJSON: stat /home/manish/Desktop/testdata/a/c: too many levels of symbolic links
2

I think this is the same issue as here

I haven't had time to work on it - do you want to?

3

@ncw I think its a separate issue. The link https://github.com/rclone/rclone/issues/4402 refers to how we can detect circular symlinks in rclone.
While this issue is related to List function in local backend. The issue is that List function returns if any circular symlink is observed. Instead, we should log the error and continue with the list function.

4

Ah, I see what you mean I think.

The List() interface on the backends either returns a listing or an error, it doesn't allow a return of partial items yet.

It would be possible to change this of course but it would require changing that interface or at least changing how it is used.

So that is quite a big change but not impossible!

5

So that is quite a big change but not impossible!

It is. For now, we can just add code to handle the circular symlinks. Like in this code block https://github.com/rclone/rclone/blob/master/backend/local/local.go#L459, we just log the error and then continue with the walk/scan.

We can add a similar handler to check if the error returned is because of circular symlinks or not. If it is then continue.

6

Yes that is a more sensible idea!

We are looking to detect ELOOP which will probably require a bit of cross platform code to detect it - so an isLoop function in a conditionally compiled blob of code (see the lchtimes*.go files for an example).

Do you want to have a go at this?

7

@ncw I opened an MR but some checks ae failing. Can you help me look at it?

8

I have replied on the Pull request :slight_smile:

9

@ncw Thanks. I made the changes and now all checks are passing. Can you review the changes?

1 Like
10

I've merged that now - thank you very much :slight_smile:

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.