I am running the same Dockerfile on two basically identical Linux machines. It works on one side, it fails on the other, with the message:

 standard_init_linux.go:211: exec user process caused "exec format error"

I reduced the Dockerfile to a minimum:

FROM xxxx/debian:jessie_2017-03-23_armhf
CMD echo "Hello World"

The system information of machine 1:

docker version

Client: Version: 18.09.7 API version: 1.39 Go version: go1.10.4 Git commit: 2d0083d Built: Fri Aug 16 14:19:38 2019 OS/Arch: linux/amd64 Experimental: false
Server: Engine: Version: 18.09.7 API version: 1.39 (minimum version 1.12) Go version: go1.10.4 Git commit: 2d0083d Built: Thu Aug 15 15:12:41 2019 OS/Arch: linux/amd64 Experimental: false

docker system info

Containers: 204 Running: 0 Paused: 0 Stopped: 204
Images: 69
Server Version: 18.09.7
Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 466 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version: N/A
init version: v0.18.0 (expected: fec36.....)
Security Options: apparmor seccomp Profile: default
Kernel Version: 4.4.0-154-generic
Operating System: Linux Mint 18.1
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.31GiB
Name: some_namx
ID: ABCU:F34S:12BA:1Y67:N692:HERA:SGTZ:1278:34O8:XZ45:2N6K:COHJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry:
Labels:
Experimental: false
Insecure Registries: xxx.yyy.zzz.ab:1234 127.0.0.0/8
Live Restore Enabled: false

Further information

dpkg -l | grep docker
ii docker 1.5-1 amd64 System tray for KDE3/GNOME2 docklet applications
rc docker-engine 17.05.0~ce-0~ubuntu-xenial amd64 Docker: the open-source application container engine
ii docker.io 18.09.7-0ubuntu1~16.04.5 amd64 Linux container runtime
dpkg --print-architecture amd64
dpkg --print-foreign-architectures i386
lscpu
Architecture: x86_64 CPU
op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) persocket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
Stepping: 9
CPU MHz: 1329.234
CPU max MHz: 4500.0000
CPU min MHz: 800.0000
BogoMIPS: 8399.90
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
lsb_release -a
No LSB modules are available.
Distributor ID: LinuxMint
Description: Linux Mint 18.1 Serena
Release: 18.1
Codename: serena
docker run golang go version go version go1.13.3 linux/amd64
docker run mplatform/mquery golang Image: golang * Manifest List: Yes
* Supported platforms: - linux/amd64 - linux/arm/v7 - linux/arm64 - linux/386 - linux/ppc64le - linux/s390x - windows/amd64:10.0.14393.3274 - windows/amd64:10.0.17134.1069 - windows/amd64:10.0.17763.805

The system information of machine 2:

docker version

Client: Version: 18.09.7 API version: 1.39 Go version: go1.10.4 Git commit: 2d0083d Built: Fri Aug 16 14:19:38 2019 OS/Arch: linux/amd64 Experimental: false
Server: Engine: Version: 18.09.7 API version: 1.39 (minimum version 1.12) Go version: go1.10.4 Git commit: 2d0083d Built: Thu Aug 15 15:12:41 2019 OS/Arch: linux/amd64 Experimental: false

docker system info

Containers: 262 Running: 0 Paused: 0 Stopped: 262
Images: 160
Server Version: 18.09.7
Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 734 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version: N/A
init version: v0.18.0 (expected: fec36....)
Security Options: apparmor seccomp Profile: default
Kernel Version: 4.4.0-165-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.859GiB
Name: a_name
ID: 1246:XV56:89G5:O8AM:2W9A:LEGO:1HJS:WG1K:SKAN:EPOX:ONCA:CA24
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry:
Labels:
Experimental: false
Insecure Registries: 127.0.0.0/8
Live Restore Enabled: false

Further information:

dpkg | grep docker
ii docker 1.5-1 amd64 System tray for KDE3/GNOME2 docklet applications
ii docker.io 18.09.7-0ubuntu1~16.04.5 amd64 Linux container runtime
dpkg --print-architecture
amd64
dpkg --print-foreign-architectures
i386
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 1
On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping: 1
CPU MHz: 2096.015
BogoMIPS: 4192.03
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht pbe syscall nx pdpe1gb lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq dtes64 ds_cpl ssse3 sdbg fma cx16 xtpr pcid dca sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase bmi1 hle avx2 bmi2 erms invpcid rtm xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local arat
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
docker run golang go version
go version go1.13.3 linux/amd64
docker run mplatform/mquery golang
Image: golang * Manifest List: Yes * Supported platforms: - linux/amd64 - linux/arm/v7 - linux/arm64 - linux/386 - linux/ppc64le - linux/s390x - windows/amd64:10.0.14393.3274 - windows/amd64:10.0.17134.1069 - windows/amd64:10.0.17763.805

The first one (on a Jenkins server) is working correctly, while the second one fails with the given error. What could be the issue? I am out of ideas! Thanks!

1 Answer

From the tag on your image, it looks like you are trying to run ARM based binaries on a x86 based host. This typically won't work without something to interpret the binary. On Linux there's Qemu plus the binfmt_misc kernel module that provides this support. You also need the various static binfmt binaries from qemu to enable this per architecture.

On Debian the static binaries are available in the qemu-user-static package with one catch. At last check in stable, Debian doesn't load these binaries in a way that supports a chroot type of environment like you have in Debian, it will look for the qemu static binaries inside the container rather than using the files on the host. See this issue and consider installing the binaries in a different way or use an unstable version.

When you have everything configured correctly, you should see the following:

$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags: OCF
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

Two important things to look for, first that the binfmt_misc and qemu-arm directory and file exist. And second, that the flags section includes a F to indicate the binaries is fixed and won't be affected by the container root filesystem changing.