Error with build v1.61.1 tree command: panic: runtime error: invalid memory address or nil pointer dereference

Help and Support
21

Still the same error....

rclone-v1.62.0-beta.6724.0f9ed7a4e.fix-tree-crash-windows-386\rclone --config AZ_aws.conf -v tree --dirs-only AZ_AWS:az-eu-decision-fabric2-dev-output/
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x1c pc=0x17f5549]

goroutine 1 [running]:
github.com/a8m/tree.NameSort({0x2195fd0, 0xdcf5dc8}, {0x0, 0x0})
        github.com/a8m/tree@v0.0.0-20230204152422-53dd42c535e0/sort.go:32 +0x39
github.com/a8m/tree.ByFunc.Less(...)
        github.com/a8m/tree@v0.0.0-20230204152422-53dd42c535e0/sort.go:14
sort.order2(...)
        sort/zsortinterface.go:299
sort.median({0x2191a24, 0xe27acd0}, 0x3, 0x6, 0x9, 0xeb3bbc4)
        sort/zsortinterface.go:308 +0x39
sort.choosePivot({0x2191a24, 0xe27acd0}, 0x0, 0xd)
        sort/zsortinterface.go:284 +0x152
sort.pdqsort({0x2191a24, 0xe27acd0}, 0x0, 0xd, 0x4)
        sort/zsortinterface.go:89 +0xd3
sort.Sort({0x2191a24, 0xe27acd0})
        sort/sort.go:48 +0x8f
github.com/a8m/tree.(*Node).sort(0xe0e9080, 0x2fadf40)
        github.com/a8m/tree@v0.0.0-20230204152422-53dd42c535e0/node.go:214 +0x17c
github.com/a8m/tree.(*Node).Visit(0xe0e9080, 0x2fadf40)
        github.com/a8m/tree@v0.0.0-20230204152422-53dd42c535e0/node.go:174 +0x6ca
github.com/rclone/rclone/cmd/tree.Tree({0x219be9c, 0xd805500}, {0x2188800, 0xd80a030}, 0x2fadf40)
        github.com/rclone/rclone/cmd/tree/tree.go:139 +0x128
github.com/rclone/rclone/cmd/tree.glob..func1.1()
        github.com/rclone/rclone/cmd/tree/tree.go:123 +0x42
github.com/rclone/rclone/cmd.Run(0x0, 0x0, 0x2fa46e0, 0xeb3be84)
        github.com/rclone/rclone/cmd/cmd.go:257 +0xf8
github.com/rclone/rclone/cmd/tree.glob..func1(0x2fa46e0, {0xd8740f0, 0x1, 0x5})
        github.com/rclone/rclone/cmd/tree/tree.go:122 +0x2dc
github.com/spf13/cobra.(*Command).execute(0x2fa46e0, {0xd8740c0, 0x5, 0x6})
        github.com/spf13/cobra@v1.6.1/command.go:916 +0x7e6
github.com/spf13/cobra.(*Command).ExecuteC(0x2f9c8e0)
        github.com/spf13/cobra@v1.6.1/command.go:1044 +0x412
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.6.1/command.go:968
github.com/rclone/rclone/cmd.Main()
        github.com/rclone/rclone/cmd/cmd.go:561 +0x9a
main.main()
        github.com/rclone/rclone/rclone.go:14 +0x17

I believe the issue is with specific files already uploaded in AWS that has the full path in the file name with special characters (this came out with the --version option) - here are these file names (there are hidden non-printable chars in between the slashes). Somehow rclone is uanble to handle them

C:\Users\kzqm438\Desktop\test_run\DF_SRC_BOX\Docs\Aditya_test_file.pdf
C:\Users\kzqm438\Desktop\test_run\DF_SRC_BOX\Docs\Sample_rave_S3_bucket_tesst.pdf
C:\Users\kzqm438\Desktop\test_run\DF_SRC_BOX\Priority\Initial_12-Aug-2021_D967MC00001_01_re.pdf
C:\Users\kzqm438\Desktop\test_run\DF_SRC_BOX\Priority\Jasper_DF_2188836.xml

Before it was crashing at the first of these files. Now it is passing that point, but failing here... the bottom of the verbose log below...

2023/02/05 11:16:08 DEBUG : Stat: filePath="\\xml_dtd_parser_daemon_out.log"
2023/02/05 11:16:08 DEBUG : >Stat: fi=xml_dtd_parser_daemon_out.log, err=
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x1c pc=0x17f5529]

goroutine 1 [running]:
github.com/a8m/tree.NameSort({0x0, 0x0}, {0x2195fd0, 0xe3e0218})
	github.com/a8m/tree@v0.0.0-20230204152422-53dd42c535e0/sort.go:32 +0x19
github.com/a8m/tree.ByFunc.Less(...)
	github.com/a8m/tree@v0.0.0-20230204152422-53dd42c535e0/sort.go:14
22

Yes its crashing at a different place in the tree lib now!

I'd like to be able to replicate this.

Could you do

rclone lsf -vv --dump bodies S3:bucket/dir --log-file rclone.log

Where dir is a directory with one of these bad files. Then if you could attach the log here or email it to me nick@craig-wood.com with a link to this page that would be very useful.

23

Email sent as requested

1 Like
24

Thanks for the listing.

I think the root cause is

2023/02/05 11:16:08 DEBUG : Stat: filePath="\\xml_dtd_parser_daemon_out.log"

So I think you've got an item with a \\ in it. This is confusing tree when it is running on Windows as it is taking those \ as path separators.

Rclone needs to be able to tell the tree library to use unix separators / but there isn't any way of doing that at the moment.

You should now see the file/directory which fails with filename [reason for failure] - it would be interesting to see what the reason is.

Please try this which should fix the crash in the tree command

v1.62.0-beta.6724.e9dc7d2cf.fix-tree-crash on branch fix-tree-crash (uploaded in 15-30 mins)

25

I only gave the bottom of the log. All files in the root folder of the bucket has the \ - I have provided some more from the bottom of the log....

2023/02/05 11:16:08 DEBUG : Stat: filePath="\\new 5.xml"
2023/02/05 11:16:08 DEBUG : >Stat: fi=new 5.xml, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\new 5_E2B.xls"
2023/02/05 11:16:08 DEBUG : >Stat: fi=new 5_E2B.xls, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\new 5_content_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=new 5_content_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\new_5.xml"
2023/02/05 11:16:08 DEBUG : >Stat: fi=new_5.xml, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\new_5_E2B.xls"
2023/02/05 11:16:08 DEBUG : >Stat: fi=new_5_E2B.xls, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\new_5_content_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=new_5_content_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\nohup.out"
2023/02/05 11:16:08 DEBUG : >Stat: fi=nohup.out, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\ontology_out.log"
2023/02/05 11:16:08 DEBUG : >Stat: fi=ontology_out.log, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\reg_testing.xml"
2023/02/05 11:16:08 DEBUG : >Stat: fi=reg_testing.xml, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\reg_testing_E2B.xls"
2023/02/05 11:16:08 DEBUG : >Stat: fi=reg_testing_E2B.xls, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\reg_testing_content_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=reg_testing_content_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\reg_testing_log.pdf"
2023/02/05 11:16:08 DEBUG : >Stat: fi=reg_testing_log.pdf, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\reg_testing_output_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=reg_testing_output_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\report-BR31-FRHB0009-01.html"
2023/02/05 11:16:08 DEBUG : >Stat: fi=report-BR31-FRHB0009-01.html, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\report-BR31-FRHB0009-01_E2B.xls"
2023/02/05 11:16:08 DEBUG : >Stat: fi=report-BR31-FRHB0009-01_E2B.xls, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\report-BR31-FRHB0009-01_content_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=report-BR31-FRHB0009-01_content_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\report-BR31-FRHB0009-01_output_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=report-BR31-FRHB0009-01_output_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\study_D5180C00021_01_Round 5010_3.pdf"
2023/02/05 11:16:08 DEBUG : >Stat: fi=study_D5180C00021_01_Round 5010_3.pdf, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\study_D5180C00021_01_Round 5010_3_E2B.xls"
2023/02/05 11:16:08 DEBUG : >Stat: fi=study_D5180C00021_01_Round 5010_3_E2B.xls, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\study_D5180C00021_01_Round 5010_3_content_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=study_D5180C00021_01_Round 5010_3_content_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\study_D5180C00021_01_Round 5010_3_log.pdf"
2023/02/05 11:16:08 DEBUG : >Stat: fi=study_D5180C00021_01_Round 5010_3_log.pdf, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test XML (1).xml"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test XML (1).xml, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test XML (1)_E2B.xls"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test XML (1)_E2B.xls, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test XML (1)_content_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test XML (1)_content_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test XML (1)_output_dict.json"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test XML (1)_output_dict.json, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test2"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test2, err=
2023/02/05 11:16:08 DEBUG : ReadDir: dir=\test2
2023/02/05 11:16:08 DEBUG : >ReadDir: names=[832P12_q_image.tgz 91P2_q_image.tgz 91P7_q_image.tgz], err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test2\\832P12_q_image.tgz"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test2/832P12_q_image.tgz, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test2\\91P2_q_image.tgz"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test2/91P2_q_image.tgz, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\test2\\91P7_q_image.tgz"
2023/02/05 11:16:08 DEBUG : >Stat: fi=test2/91P7_q_image.tgz, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\testfile"
2023/02/05 11:16:08 DEBUG : >Stat: fi=testfile, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\testfile2"
2023/02/05 11:16:08 DEBUG : >Stat: fi=testfile2, err=
2023/02/05 11:16:08 DEBUG : Stat: filePath="\\xml_dtd_parser_daemon_out.log"
2023/02/05 11:16:08 DEBUG : >Stat: fi=xml_dtd_parser_daemon_out.log, err=
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x1c pc=0x17f5529]
26

Was that using the new code I posted? From here: https://beta.rclone.org/branch/fix-tree-crash/v1.62.0-beta.6724.e9dc7d2cf.fix-tree-crash/

I need to see the lines following this.

27

Using the latest beta code you prepared..... here's the bottom of the tree command...

2023/02/06 12:32:22 DEBUG : >Stat: fi=Bavani_Initial_D4194R00013_2021A107416_2021A038791_01_content_dict.json, err=
2023/02/06 12:32:22 DEBUG : Stat: filePath="\\Bavani_Initial_D4194R00013_2021A107416_2021A038791_01_log.pdf"
2023/02/06 12:32:22 DEBUG : >Stat: fi=Bavani_Initial_D4194R00013_2021A107416_2021A038791_01_log.pdf, err=
2023/02/06 12:32:22 DEBUG : Stat: filePath="\\Bavani_Initial_D4194R00013_2021A107416_2021A038791_01_output_dict.json"
2023/02/06 12:32:22 DEBUG : >Stat: fi=Bavani_Initial_D4194R00013_2021A107416_2021A038791_01_output_dict.json, err=
2023/02/06 12:32:22 DEBUG : Stat: filePath="\\C:\\Users\\kzqm438\\Desktop\\test_run\\DF_SRC_BOX\\Docs\\Aditya_test_file.pdf"
2023/02/06 12:32:22 DEBUG : >Stat: fi=, err=couldn't find "C:/Users/kzqm438/Desktop/test_run/DF_SRC_BOX/Docs/Aditya_test_file.pdf" in directory cache
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0x1915af6]

goroutine 1 [running]:
github.com/a8m/tree.(*Node).Visit(0xc000d16d20, 0x32aef80)
        github.com/a8m/tree@v0.0.0-20210414114729-ce3525c5c2ef/node.go:145 +0x476
github.com/rclone/rclone/cmd/tree.Tree({0x24109c0?, 0xc0003ccdc0?}, {0x23eb840?, 0xc00014a008?}, 0x32aef80)
        github.com/rclone/rclone/cmd/tree/tree.go:139 +0x157
github.com/rclone/rclone/cmd/tree.glob..func1.1()
        github.com/rclone/rclone/cmd/tree/tree.go:123 +0x33
github.com/rclone/rclone/cmd.Run(0x0, 0x0, 0xc00011e010?, 0xc0014b1cf0)
        github.com/rclone/rclone/cmd/cmd.go:255 +0x13b
github.com/rclone/rclone/cmd/tree.glob..func1(0x3298260?, {0xc000514900?, 0x1, 0x6?})
        github.com/rclone/rclone/cmd/tree/tree.go:122 +0x2b5
github.com/spf13/cobra.(*Command).execute(0x3298260, {0xc0005148a0, 0x6, 0x6})
        github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0x32890e0)
        github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.6.1/command.go:968
github.com/rclone/rclone/cmd.Main()
        github.com/rclone/rclone/cmd/cmd.go:559 +0x76
main.main()
        github.com/rclone/rclone/rclone.go:14 +0x17
28

That is definitely the old library code. I might have messed up the build or you might be running the wrong version.

Can you check what rclone version prints? It should print

$ rclone version
rclone v1.62.0-beta.6724.e9dc7d2cf.fix-tree-crash
[snip]

Thanks

29

rclone v1.61.1

  • os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
  • os/kernel: 10.0.19044.2486 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: cmount
30

hi,

ncw created a beta build for you to test.
can you download this and use it for testing?

https://beta.rclone.org/branch/fix-tree-crash/v1.62.0-beta.6724.e9dc7d2cf.fix-tree-crash/rclone-v1.62.0-beta.6724.e9dc7d2cf.fix-tree-crash-windows-amd64.zip

and this should be the output of rclone version

rclone version
rclone v1.62.0-beta.6724.e9dc7d2cf.fix-tree-crash
- os/version: Microsoft Windows 11 Pro 21H2 (64 bit)
- os/kernel: 10.0.22000.1455 Build 22000.1455.1455 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20
- go/linking: static
- go/tags: cmount
1 Like
31

With beta-62 version, the tree command is below....
The error files are being called out.

←[1;34m/←[0m
├── ←[1;34mFailed_files←[0m
├── ←[1;34mFileMonitoring←[0m
├── ←[1;34mONTAP←[0m
│   └── ←[1;34mimage←[0m
├── ←[1;34mPriority←[0m
├── ←[1;34mProcessedBYDF←[0m
├── ←[1;34mUnProcessedBYDFR2←[0m
├── ←[1;34mUnProcessedBYDFR3←[0m
├── ←[1;34mtest2←[0m
├── Jasper_DF_2188836.xml [couldn't find "C:/Users/kzqm438/Desktop/test_run/DF_SRC_BOX/Priority/Jasper_DF_2188836.xml" in directory cache]
├── Initial_12-Aug-2021_D967MC00001_01_re.pdf [couldn't find "C:/Users/kzqm438/Desktop/test_run/DF_SRC_BOX/Priority/Initial_12-Aug-2021_D967MC00001_01_re.pdf" in directory cache]
├── Sample_rave_S3_bucket_tesst.pdf [couldn't find "C:/Users/kzqm438/Desktop/test_run/DF_SRC_BOX/Docs/Sample_rave_S3_bucket_tesst.pdf" in directory cache]
└── Aditya_test_file.pdf [couldn't find "C:/Users/kzqm438/Desktop/test_run/DF_SRC_BOX/Docs/Aditya_test_file.pdf" in directory cache]
32

Thank you for testing and - glad it is no longer crashing.

This is the underlying error

I see what the problem is in rclone now - can you give this a try please

v1.62.0-beta.6730.8da51732b.fix-tree-windows on branch fix-tree-windows (uploaded in 15-30 mins)

Note that adding --color NEVER will fix the funny [1;34m in the output.

@albertony it looks like tree colouring isn't working in Windows cmd windows? It doesn't work for me in my Windows 10 VM

33

Looks like the funky characters only come with --dirs-only option. Happens with the latest build as well, not with --colors NEVER

image
image1055×183 4.69 KB

1 Like
34

Sorry, I did not notice this question.

This discussion seems much like the one in https://github.com/rclone/rclone/issues/6668. When testing that case it worked everywhere on my Windows computer, but the reporter had to set the HKEY_CURRENT_USER\Console\VirtualTerminalLevel registry value to enable it. When testing now I see the same as you; colouring no longer works in cmd, but if I set that registry value it works again! Since my previous testing I've been upgraded from Windows 10 21H2 to 22H2, and the reporter of the mentioned issue seems was on 22H2
too, so I don't know if it is something related to that. Though, that seems a bit strange. Maybe we can use Windows API function SetConsoleMode to enable it from rclone source, I will look into that.

1 Like
35

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.