How the fsck does one build a Go program, such as rclone, without PIE, without getting a trillion linker errors

I just want to build this stupid program on my local system. With PIE disabled for slightly easier debugging. That's literally all I'm trying to do. And it's taken multiple hours, and nothing fucking works.

I'm much more of a C++ person than a Go person. In part because my experiences using the Go tools/toolchain/whatever to build third-party software has been so frustrating, annoying, opaque, problematic, and so forth, that it makes me want to literally throw my chair across the room. It pisses me off with how much it tries to do "automatically" behind the scenes, without ever telling you what it's actually doing, or even making it possible to find out.

Anyway. This is on Arch Linux, incidentally. Which, as I understand it, has PIE enabled by default for Go and various other things.

The PKGBUILD file for building the rclone distro package has, by default, export GOFLAGS="-buildmode=pie -trimpath" in it. Okay, that wasn't too difficult to figure out; I guess I just change the buildmode to exe, which the docs say will do the same thing just without PIE, and that should do it right?

Nope; hundreds of megabytes of GOPATH GOCACHE garbage later, I get linker errors.

So I make sure that every possible permutation of -no-pie, -fno-pie, -fno-PIE, -Wl,--no-pie, etc. is present in CFLAGS and LDFLAGS and CGO_CFLAGS and CGO_LDFLAGS. Nope, still linker errors.

I try using ld.gold instead of standard bfd ld. Nope, that doesn't work either; internal linker error once again. Relocations it can't work with, because some piece of something, somewhere, was apparently compiled assuming PIE, even though everything that I've built myself, as far as I can tell, should have been built for not-PIE.

I try using more GOFLAGS, like -v -x and -work, so that it'll actually show me the commands it's running and also not immediately delete all the temporary files after it's done failing a build. But ultimately that's a waste of time, because the temporary working directory for linking is unaffected by -work apparently! Go figure! So when I get

GOROOT_FINAL='go' /usr/lib/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=1Iiyp_uE2ApFLP1q8yRz/P-HolFMOg21QJ-pufDky/3xdl1dW8Dqji9VGMDRxx/1Iiyp_uE2ApFLP1q8yRz -s -X github.com/rclone/rclone/fs.Version=v1.57.0 -extld=gcc /tmp/GOCACHE/b4/b4fa5f44bfb0a6a4fe2574a8460349a96e6a0289727c0e2a79ed74f6ea7e08fe-d
# github.com/rclone/rclone
/usr/lib/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-2802890774/000024.o: warning: relocation against `stderr@@GLIBC_2.2.5' in read-only section `.text'
/usr/bin/ld: /tmp/go-link-2802890774/000017.o: relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a PDE object; recompile with -fPIE
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status

my first instinct is to go over to /tmp/go-link-2802890774/ and check out the (completely opaquely named) object files in that directory. Except that I actually can't, because that directory is deleted unconditionally at the end of the link step regardless of -work or anything else.

Here's the slightly different, but still ultimately useless, output I get when using -fuse-ld=gold:

GOROOT_FINAL='go' /usr/lib/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=Wf2wZattAuZgKyyFVsWH/P-HolFMOg21QJ-pufDky/3xdl1dW8Dqji9VGMDRxx/Wf2wZattAuZgKyyFVsWH -s -X github.com/rclone/rclone/fs.Version=v1.57.0 -extld=gcc /tmp/GOCACHE/b4/b4fa5f44bfb0a6a4fe2574a8460349a96e6a0289727c0e2a79ed74f6ea7e08fe-d
# github.com/rclone/rclone
/usr/lib/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld.gold: internal error in check_non_pic, at /tmp/binutils/src/binutils-gdb/gold/x86_64.cc:3522
collect2: error: ld returned 1 exit status

I even ended up using execsnoop to figure out what linker commands were actually being invoked, because /usr/lib/go/pkg/tool/linux_amd64/link simply doesn't give a shit and goes "lol I'm not telling, you just get to guess".

But even that isn't particularly enlightening, because any useful source file, object file, or binary file names have all been transformed into opaque numerical identifiers devoid of any context. So it's basically impossible for me to figure out what internal or external or file/module/object/whatever is responsible for the problems.

(Also note that /usr/lib/go/pkg/tool/linux_amd64/link was helpful enough to cut off whatever other prior lines of ld.gold output preceded the internal error message. It's not even possible for ld.gold to print that message by itself, without having previously printed a warning and set a bool that then allows it to subsequently actually hit the ICE assert the next time. So that's super cool.)

I thought the appeal of Go involved, in part, (a) a super-cool build system where you type go build or whatever and it "just works"; and (b) that everything is statically linked into the executable for maximum code bloat (which, I was led to believe, should presumably mean that I'd be building all the code that goes into the executable and so therefore it shouldn't even be possible to have this sort of PIE/not-PIE problem? but I guess not? are there static library binary blobs involved that were pre-built with -static-pie or something? how would I even find this out since I can't tell what's even being linked together?)

So. Rant aside... Anyone have any tips, or should I just give up? Thanks.

Hi jgottula,

Interesting battle, though I fail to understand your reason to enter it:

I use VS Code and find it super easy to debug Go/rclone.The most difficult part was to make the first launch configuration. My development environment is setup according to the rclone contributing guide.

What IDE/debugger are you using?
How does PIE make it more difficult?
What part of rclone are you trying to debug?

Are you doing remote development (SSH,WSL2,Docker,…)?

That is generally the Go experience! The build system is super slick, has caching, etc and just works. The price you pay for this though is that making it do something non-standard is quite hard.

When I tried this on Ubuntu 21.10 with a Go compiler (go1.17.2) I compiled myself from source I get this which looks like the default here is to build without pie, but it builds with pie also:

~/go/src/github.com/rclone/rclone$ go build -buildmode=pie
~/go/src/github.com/rclone/rclone$ file rclone
rclone: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3a6b99bc49e12ab4963e5e490c55b817fa2477c5, for GNU/Linux 3.2.0, not stripped
~/go/src/github.com/rclone/rclone$ go build -buildmode=exe
~/go/src/github.com/rclone/rclone$ file rclone
rclone: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6f25b3575ba8dac3fd4ad546e9430f296cac7985, for GNU/Linux 3.2.0, not stripped
~/go/src/github.com/rclone/rclone$ go build
~/go/src/github.com/rclone/rclone$ file rclone
rclone: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8316b8e9009eaa5d2ae2a879dd2067d84616ab80, for GNU/Linux 3.2.0, not stripped

So I suspect that Arch have patched the Go compiler somehow... Can you try the vanilla compiler as downloaded from Downloads - The Go Programming Language ?

Hi @jgottula, did we solve your issue?

In my testing on Arch distro in WSL, using pacman installed go 1.17.6, the default is also -buildmode=exe:

$ go build
$ file rclone
rclone: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=dc86b10a61e113b3f3f72964e8054f0521e22ad7, for GNU/Linux 4.4.0, not stripped
$ go build -buildmode=exe
$ file rclone
rclone: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=00f545d319a4b3419ed2ad9b1091e451576e05df, for GNU/Linux 4.4.0, not stripped
$ go build -buildmode=pie
$ file rclone
rclone: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=866aca2b3f88e925ecff63e861d9cb711555c5a2, for GNU/Linux 4.4.0, not stripped
1 Like

Hi @jgottula, did we solve your issue?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.