What to do when Go will not run, nor install from source ?

Submitted by Frederic Marand on
Google Go = Golang

Some days ago, at the AWS Summit 2014, DamZ renewed my long-sleeping interest for the Google Go language with wonderous stories about its use in infrastructure of the new Commerce Guys Platform they were launching that same days, so I've been doing my homework getting up to date on Go programming: that's what holidays are for, aren't they ?

The symptoms

So this morning it was time to consider deploying to production... but alas, programs which worked normally on my Ubuntu development workstation failed to run on the target server, running Debian Squeeze, with rather mystifying results:

  • Laptop: uname -a : Linux ubuntu 3.2.0-64-generic #97-Ubuntu SMP Wed Jun 4 22:04:21 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  • Target: uname -a : Linux v1085 3.2.23-vs2.3.2.12-beng #1 SMP Tue Jul 24 10:15:16 BST 2012 x86_64 GNU/Linux
  • binary copied from one machine to the other with its config files, database schema and privileges set up on both machines
  • All OK on the laptop.
$ ls -l myapp
-rwxrwxr-x 1 root root 3522952 Jun  6 11:32 myapp
$ ./myapp
bash: ./myapp: no such file or directory
$ strace -f -e open ./myapp
exec: no such file or directory
$ file myapp
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped
$ ldd myapp
ldd:
	not a dynamic executable
$

So, for "file", this is an dynamically linked executable, but not for ldd ? And, even more surprising, a trivial helloworld.go compiled the same way worked on both machines. What could be going wrong ?

The diagnostic

How about that ldd myapp on the laptop ?

 ldd usemysql 
	linux-vdso.so.1 =>  (0x00007fff64fff000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff681e8e000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff681b03000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff6820b7000)

Now we're going somewhere: the same file is diagnosed differently by the same command on the two machines. How about checking that binary itself ?

readelf -l usemysql

Elf file type is EXEC (Executable file)
Entry point 0x4258a0
There are 9 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001f8 0x00000000000001f8  R E    1000
  INTERP         0x0000000000000be4 0x0000000000400be4 0x0000000000400be4
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x0000000000111c40 0x0000000000111c40  R E    1000
  LOAD           0x0000000000112000 0x0000000000512000 0x0000000000512000
                 0x0000000000231c50 0x0000000000231c50  R      1000
  LOAD           0x0000000000344000 0x0000000000744000 0x0000000000744000
                 0x0000000000017ac8 0x0000000000035db8  RW     1000
  DYNAMIC        0x00000000003440c0 0x00000000007440c0 0x00000000007440c0
                 0x0000000000000130 0x0000000000000130  RW     8
  TLS            0x000000000035bac8 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000010  R      8
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     8
  LOOS+5041580   0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000         8

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .text .plt .interp 
   03     .rodata .typelink .gosymtab .gopclntab .dynsym .rela .gnu.version .gnu.version_r .hash .dynstr .rela.plt 
   04     .got .got.plt .dynamic .noptrdata .data .bss .noptrbss 
   05     .dynamic 
   06     .tbss 
   07     
   08    

So it turns out that this program needs /lib64/ld-linux-x86-64.so.2 which exists on one machine but not on the other. Symlinking ld-linux.so.2 to the missing path does change things (corrupted library error), but does not fix the problem.

OK, then how about installing Go 1.2.2 on that server and compiling there to see the difference ?

The golang-go package for Debian Squeeze only contains Go 1.0, so it is not a solution either, so let's follow the instructions on http://golang.org/doc/install... and get the same error from the official go binary for that distribution. Well, not /that/ surprising, at this point: let's build from source ! It's well documented too, on http://golang.org/doc/install/source, so let's proceeed.

# cd /opt
# tar xf go1.2.2.linux-amd64.tar.gz
# cd go/src
# ./app.bash
[...snip...]
/usr/include/features.h:324:26: fatal error: bits/predefs.h: No such file or directory
 #include <bits/predefs.h>

So the official distribution does not compile either. Right, Squeeze is probably too old, and that server needed a Wheezy upgrade anyway... let's upgrade.

[...skip Squeeze to Wheezy upgrade...]
# cd go/src
# ./app.bash
[...snip...]
/usr/include/features.h:324:26: fatal error: bits/predefs.h: No such file or directory
 #include <bits/predefs.h>

Well, not that much better... and predefs.h is actually present in /usr/include/i386-linux-gnu/bits/predefs.h. Was already there on Squeeze, actually. So it looks like for some reason, the go build appears to have an undocumented dependency on non-default multilib architecture configurations.

The solution

So let's just install that multilib support, even though it was not needed on Ubuntu 12.04 :

# apt-get install gcc-multilib
# cd /opt/go/src
# ./all.bash
[...gobs of output, include net test panics...]

At last, the Go install process succeeds ! Let's skip the failing net tests, which cannot handle the IP specifics of the VM anyway (edit src/pkg/net/ipraw_test.go and src/pkg/net/http/fs_test.go to swap case on the failing tests), and we have a winner:

# go version
go version go1.2.2 linux/amd64
# cd /tmp
# ./myapp
[...produces expected output...]
#

Long story short: in such deployment situations, digging into the entrails of binaries can be helpful. The file, ldd / readelf combo can work wonders.

And special thanks to to taruti on #go-nuts, who really showed me the path.