Error with build v1.61.1 tree command: panic: runtime error: invalid memory address or nil pointer dereference

Help and Support
1

Trying to run the tree command as follows (newly downloaded v1.61.1 for Windows 64):
rclone --config -v tree --dirs-only remote:bucket

Getting error:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0x1915af6]

goroutine 1 [running]:
github.com/a8m/tree.(*Node).Visit(0xc001036180, 0x32aef80)
github.com/a8m/tree@v0.0.0-20210414114729-ce3525c5c2ef/node.go:145 +0x476
...
...

Please help.
The AWS bucket has a lot of files (several thousands that I know of).

2

Any chance you'd like to use that template and share all the info?

3

Problem: Getting the error below with the tree command:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0x1915af6]

goroutine 1 [running]:
github.com/a8m/tree.(*Node).Visit(0xc001036180, 0x32aef80)
        github.com/a8m/tree@v0.0.0-20210414114729-ce3525c5c2ef/node.go:145 +0x476
github.com/rclone/rclone/cmd/tree.Tree({0x24109c0?, 0xc0008362c0?}, {0x23eb840?, 0xc00014a008?}, 0x32aef80)
        github.com/rclone/rclone/cmd/tree/tree.go:139 +0x157
github.com/rclone/rclone/cmd/tree.glob..func1.1()
        github.com/rclone/rclone/cmd/tree/tree.go:123 +0x33
github.com/rclone/rclone/cmd.Run(0x0, 0x0, 0xc00011e010?, 0xc0004fbcf0)
        github.com/rclone/rclone/cmd/cmd.go:255 +0x13b
github.com/rclone/rclone/cmd/tree.glob..func1(0x3298260?, {0xc00041c230?, 0x1, 0x6?})
        github.com/rclone/rclone/cmd/tree/tree.go:122 +0x2b5
github.com/spf13/cobra.(*Command).execute(0x3298260, {0xc0000997a0, 0x5, 0x6})
        github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0x32890e0)
        github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.6.1/command.go:968
github.com/rclone/rclone/cmd.Main()
        github.com/rclone/rclone/cmd/cmd.go:559 +0x76
main.main()
        github.com/rclone/rclone/rclone.go:14 +0x17

Command used:
rclone --config configfile -v tree --dirs-only remote:bucket

Version:
rclone v1.61.1

  • os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
  • os/kernel: 10.0.19044.2486 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: cmount

Content of configfile:
[AWS]
type = s3
provider = Other
env_auth = false
access_key_id = ***
secret_access_key = ***
region = eu-west-1
endpoint = https://s3.eu-west-1.amazonaws.com/
location_constraint =
acl = bucket-owner-full-control

4

hi,
make sure there is enough free memory,
i just did a rclone tree on a bucket in wasabi, s3 clone, with over 1,000,000 files.
peaked at 3.2GiB memory

5

Did you also have --dirs-only?

6

That is a little weird. If I look at the place rclone crashed

		nnode := &Node{
			path:   filepath.Join(node.path, name),
			depth:  node.depth + 1,
			vpaths: node.vpaths,
		}
		d, f := nnode.Visit(opts)   // <---- rclone crashed here with invalid memory address or nil pointer dereference
		if nnode.IsDir() {
			// "prune" option, hide empty directories
			if opts.Prune && f == 0 {

nnode can't be nil there as it was defined in the previous line.

So a bit puzzling.

@abasu is the error repeatable?

7

v1.61.1 using --dirs-only

imho, the OP should post the real command and the debug output.

edit: i first ran v1.60.0 without --dirs-only, maxed out at 3.2GiB memory.

now i am running v1.61.1 with --dirs-only, so far maxed out using just 0.4GIB.
the command is still running.

8

It is repeatable every time.
I used --dirs-only option too.

9

Yes I did. Sorry forgot to copy the real commnd.

10

I am consistently getting the same error even with --level 1 option.
Could it be that some bad file of some sort causing this issue?
Or could it be some corruption on AWS bucket itself?

11

please add -vv to the command and post the full debug output.

12

I think you may be seeing line 144 instead of 145:

	nnode := &Node{
		path:   filepath.Join(node.path, name),
		depth:  node.depth + 1,
		vpaths: node.vpaths,
	}
	d, f := nnode.Visit(opts) 
	if nnode.IsDir() { // <---- rclone crashed here with invalid memory address or nil pointer dereference
		// "prune" option, hide empty directories
		if opts.Prune && f == 0 {

This may be possible if the stat fails earlier in the function which will cause FileInfo to not be defined for the IsDir function:

	// stat
	fi, err := opts.Fs.Stat(node.path)
	if err != nil {
		node.err = err
		return
	}
	node.FileInfo = fi
1 Like
13

Thank you for the --verbose optoin suggestion. Could this be the issue that some it is failing to stat the file shown (folder name with a colon)? Below is last few lines of the screen dump.

2023/02/03 14:40:55 DEBUG : Stat: filePath="\C:\Users\kzqm438\Desktop\test_run\DF_SRC_BOX\Docs\Aditya_test_file.pdf"
2023/02/03 14:40:55 DEBUG : >Stat: fi=, err=couldn't find "C:/Users/kzqm438/Desktop/test_run/DF_SRC_BOX/Docs/Aditya_test_file.pdf" in directory cache
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0x1915af6]

14

i am not sure what is going on, that is for others.
just trying to collect basic info.

can you re-run the command with -vv and post the top 20 lines?

which folder do you mean?

15

Top few lines with bucket name redacted

rclone-v1.61.1-windows-amd64\rclone : 2023/02/03 14:40:47 DEBUG : rclone: Version "v1.61.1" starting with parameters ["C:\\Arindam\\rclone\\rclone-v1.61.1-windows-amd64\\rclone.exe" "--config" "AZ_aws.conf" "-v" "tree" "--dirs-only" 
"bucket-name-xxx/" "--verbose"]
At line:1 char:1
+ rclone-v1.61.1-windows-amd64\rclone --config AZ_aws.conf -v tree --di ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2023/02/03 14:4.../" "--verbose"]:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2023/02/03 14:40:47 DEBUG : Creating backend with remote "AZ_AWS:bucket-name-xxx/"
2023/02/03 14:40:47 DEBUG : Using config file from "C:\\Arindam\\rclone\\AZ_aws.conf"
2023/02/03 14:40:47 DEBUG : fs cache: renaming cache item "AZ_AWS:bucket-name-xxx/" to be canonical "AZ_AWS:bucket-name-xxx"
2023/02/03 14:40:55 DEBUG : Stat: filePath="/"
2023/02/03 14:40:55 DEBUG : >Stat: fi=, err=
2023/02/03 14:40:55 DEBUG : ReadDir: dir=/
2023/02/03 14:40:55 DEBUG : >ReadDir: names=[1GB test (1).txt 2 2.rames 2018SE11903#1#RECORDATI_AZPRODSE00_RECORDATI_59477_ARGPRDRECORDATI_20220706094931.xml 
2018SE11903#1#RECORDATI_AZPRODSE00_RECORDATI_59477_ARGPRDRECORDATI_20220706094931_E2B.xls 2018SE11903#1#RECORDATI_AZPRODSE00_RECORDATI_59477_ARGPRDRECORDATI_20220706094931_content_dict.json 
2018SE11903#1#RECORDATI_AZPRODSE00_RECORDATI_59477_ARGPRDRECORDATI_20220706094931_output_dict.json 2018SE24375#18#AZ_SE_22859128.xml 2018SE24375#18#AZ_SE_22859128_E2B.xls 2018SE24375#18#AZ_SE_22859128_content_dict.json 
2018SE24375#18#AZ_SE_22859128_output_dict.json 2018SF32622#6#E-ASTRAZENECA-20220729-201843320-20220707260.xml 2018SF32622#6#E-ASTRAZENECA-20220729-201843320-20220707260_E2B.xls 
2018SF32622#6#E-ASTRAZENECA-20220729-201843320-20220707260_content_dict.json 2018SF33202#5#E-ASTRAZENECA-20220729-201843602-20220707274.xml 2018SF33202#5#E-ASTRAZENECA-20220729-201843602-20220707274_E2B.xls 
2018SF33202#5#E-ASTRAZENECA-20220729-201843602-20220707274_content_dict.json 2018SF36229#4.01#E-ASTRAZENECA-20220624-201844285-20220606299.xml 2018SF37162#5#E-ASTRAZENECA-20220603-201844375-20220601172.xml 
2018SF37162#5#E-ASTRAZENECA-20220603-201844375-20220601172_E2B.xls 2018SF37162#5#E-ASTRAZENECA-20220603-201844375-20220601172_content_dict.json 2018SF49318#2.02#E-ASTRAZENECA-20220523-201847911-20220503453.xml 2019SE56115_D0818C00001.pdf 
2019SE56115_D0818C00001_E2B.xls 2019SE56115_D0818C00001_content_dict.json 2019SE56115_D0818C00001_log.pdf 2019SF27200_D933SC00001_920_initial.pdf 2019SF27200_D933SC00001_920_initial_E2B.xls 
2019SF27200_D933SC00001_920_initial_content_dict.json 2019SF27200_D933SC00001_920_initial_log.pdf 2019SF31829_SD_D933QC00001.pdf 2019SF31829_SD_D933QC00001_E2B.xls 2019SF31829_SD_D933QC00001_content_dict.json 
2019SF31829_SD_D933QC00001_extract_dict.json 2019SF31829_SD_D933QC00001_log.pdf 2019SF74292 d_D081SC00001.pdf 2019SF74292 d_D081SC00001_E2B.xls 2019SF74292 d_D081SC00001_content_dict.json 2019SF74292 d_D081SC00001_extract_dict.json 
2019SF74292 d_D081SC00001_log.pdf 2020SE07217 d D933IC00001 .pdf 2020SE07217 d D933IC00001 _E2B.xls 2020SE07217 d D933IC00001 _content_dict.json 2020SE07217 d D933IC00001 _log.pdf 2020SE62604__SD_D933RC00001_initial.pdf 
2020SE62604__SD_D933RC00001_initial_E2B.xls 2020SE62604__SD_D933RC00001_initial_content_dict.json 2020SE62604__SD_D933RC00001_initial_log.pdf 2020SE83916_D3250C00036.pdf 2020SE83916_D3250C00036_E2B.xls 
2020SE83916_D3250C00036_content_dict.json 2020SE83916_D3250C00036_extract_dict.json 2020SE83916_D3250C00036_log.pdf 2020SF27963__SD_ipsaz_820_final_D361BC00001.pdf 2020SF27963__SD_ipsaz_820_final_D361BC00001_E2B.xls 
2020SF27963__SD_ipsaz_820_final_D361BC00001_content_dict.json 2020SF27963__SD_ipsaz_820_final_D361BC00001_log.pdf 2021A647034 - INITIAL - D967MC00001 .pdf 2021A647034 - INITIAL - D967MC00001 _E2B.xls 2021A647034 - INITIAL - 
D967MC00001 _content_dict.json 2021A647034 - INITIAL - D967MC00001 _output_dict.json 2021A650195-4.00_D081RC00001 (1).pdf 2021A650195-4.00_D081RC00001 (1)_extract_dict.json 2021A650195-4.00_D081RC00001 (1)_log.pdf 2021A756938 - INITIAL 
- D419JC00001.pdf 2021A756938 - INITIAL - D419JC00001_E2B.xls 2021A756938 - INITIAL - D419JC00001_content_dict.json 2021A756938 - INITIAL - D419JC00001_extract_dict.json 2021A756938 - INITIAL - D419JC00001_log.pdf 
2021A784720_1_D822BC00001.pdf 2021A784720_1_D822BC00001_E2B.xls 2021A784720_1_D822BC00001_content_dict.json 2021A784720_1_D822BC00001_log.pdf 2021A806694_D9675C00001_FU.xml 2021A806694_D9675C00001_FU_E2B.xls
16

that should be -verbose or -vv

17

It has the verbose in the end...

rclone --config AZ_aws.conf -v tree --dirs-only  AZ_AWS:bucket-name-xxx/ --verbose
18

yes, that was a typo in my last post.

19

Yes you are right, I think I must have been looking at the wrong version of the code.

Yes that would fit the symptoms exactly and looking at the place it crashed the error handling is messed up there - it should check node.err == nil before checking node.IsDir

Yes, looks like that bug was introduced in the last commit to the repo (in April 2021)

I've made a PR to fix this here: Fix nil pointer exception in Visit on stat error and Go modernisation by ncw · Pull Request #21 · a8m/tree · GitHub

When this is merged (I'm a maintainer on this repo so quickly hopefully!) I'll make an rclone update for @abasu to try.

20

Upstream is fixed :slight_smile:

@abasu can you please give this a go - it should fix the crash

v1.62.0-beta.6724.0f9ed7a4e.fix-tree-crash on branch fix-tree-crash (uploaded in 15-30 mins)