Running vcsim native on Mac Silicon with M1 chip in docker

So, I bought this brand new laptop to do some development work. For a couple of reasons, the MacBook Pro with an M1 chip (either Pro or Max - you cannot go wrong here) suited best for me. It has long battery life, it's very powerful, silent, and has high memory for loads of simultaneous containers (and hopefully in the future x86/x64 emulation for VMs so I can run things like virtual vCenters, ESXi etc). Now, I've been always been a PC user with Windows, so it's a dive into the deep, but so far I like it. In fact, this article is the first written on this device - you've got to start sometime right?

x64 emulation with Rosetta

In case you didn't know, this new MacBook has a totally different CPU architecture which isn't compatible with x86/x64 Intel/AMD architecture. To run 'older' software, Apple provides real-time translation emulation through something called Rosetta which works good, and since the CPU is quite powerful, most of the time you don't notice it. But as a nerd, I want everything to be tuned and perfect, also for running VCSIM in a docker container.

About VCSIM

VCSIM is a vCenter simulator, a mock framework, written in GO and maintained by VMware through GOVMOMI. There are already quite some good articles already to be found online. Check them out to learn more. I like it a lot since you can run the binary or pull the official docker container from dockerhub. However, there is no official release for the M1 Silicon chip, for both binary or container, so I needed to create one of my own.

The 'notification'

When trying to run the official container created by VMware (and a couple of others who created the same container for the x64 architecture), you get the following message when running the container:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Which is fine, since it is working, but running through Rosetta:

boukegroenescheij@Boukes-MBP arm % docker run -p 8989:8989 vmware/vcsim:latest /vcsim version
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Build Version: 0.27.2
Build Commit: f04d77d6
Build Date: 2021-11-23T21:22:18Z 

Also, if you look at Docker desktop, it gives you a warning, the container is running not natively: 

A workaround is to add someting like '--platform linux/amd64' to your docker run command:

docker run -p 8989:8989 --platform linux/amd64 vmware/vcsim:latest /vcsim version

It doesn't result in a warning message, but that doesn't change the fact the container is still amd64 based so running emulated.

Build from source

My first thought was to build the binary from source, compiled on the M1 chip, resulting in a native executable, and put that into a container. So I downloaded the source and compiled it to a binary, which went into the default bin folder of my go path. Running it didn't give an issue, but it DOES work:

boukegroenescheij@Boukes-MBP bin % ./vcsim version
Build Version:
Build Commit:
Build Date:
boukegroenescheij@Boukes-MBP bin % ./vcsim
export GOVC_URL=https://user:pass@127.0.0.1:8989/sdk GOVC_SIM_PID=96608

Also, when checking the file, I'm getting the correct identifier:

boukegroenescheij@Boukes-MBP bin % file vcsim
vcsim: Mach-O 64-bit executable arm64

So, the next step is to create a docker container, using this binary:

docker build --platform linux/arm64 -t bouke/vcsim-test -f ../pkg/mod/github.com/vmware/This email address is being protected from spambots. You need JavaScript enabled to view it..2/Dockerfile.vcsim .

However, the original docker file points to an older go baseImage: golang:1.16.0-buster amd64. And I want the latest arm image.

So in file 'Dockerfile.vcsim' change the 3rd line from:

FROM golang@sha256:f254180c5defa2653955e963fb0626e3d4fbbb162f7cff6490e94607d1d867ff AS build

To:

FROM golang:1.17.5-buster AS build

When building the container you can define which platform it uses for the baseImage.

And run it with:

docker run -p 8989:8989 bouke/vcsim-test:latest /vcsim version

But that doesn't work:

boukegroenescheij@Boukes-MBP bin % docker build --platform linux/arm64 -t bouke/vcsim-test -f ../pkg/mod/github.com/vmware/This email address is being protected from spambots. You need JavaScript enabled to view it..2/Dockerfile.vcsim .
[+] Building 0.6s (14/14) FINISHED
=> [internal] load build definition from Dockerfile.vcsim 0.0s
=> => transferring dockerfile: 43B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/golang:1.17.5-buster 0.5s
=> [build 1/4] FROM docker.io/library/golang:1.17.5-buster@sha256:007c4b51431057912f538a180a486e06c972e5462dce1fffbbd3e1ba229949d6 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 29B 0.0s
=> CACHED [build 2/4] WORKDIR /go/src/app 0.0s
=> CACHED [build 3/4] RUN adduser --disabled-password --gecos "" --home "/nonexistent" --shell "/sbin/nologin" --no-create-home --uid "10001" "app 0.0s
=> CACHED [build 4/4] RUN mkdir /temporary-tmp-directory && chmod 777 /temporary-tmp-directory 0.0s
=> CACHED [stage-1 1/5] COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ 0.0s
=> CACHED [stage-1 2/5] COPY --from=build / etc/passwd / etc/passwd 0.0s
=> CACHED [stage-1 3/5] COPY --from=build /etc/group /etc/group 0.0s
=> CACHED [stage-1 4/5] COPY --chown=appuser --from=build /temporary-tmp-directory /tmp 0.0s
=> CACHED [stage-1 5/5] COPY vcsim /vcsim 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:06e8f38f8eef742d2474b62524cbfc532c202236b6f82433760d2f7415574d16 0.0s
=> => naming to docker.io/bouke/vcsim-test 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

boukegroenescheij@Boukes-MBP bin % docker run -p 8989:8989 bouke/vcsim-test:latest /vcsim version
standard_init_linux.go:228: exec user process caused: exec format error

When searching for a solution, you'll find this has to do because the binary isn't compliant. It sounds strange, but this is actually correct!

Container vs binary 

So the container runs native on the M1 chip and the binary runs native on the M1 chip - however, that doesn't mean the binary runs natively in the container. And this is exactly the issue here. We need a linux/arm64 binary to run inside the container. For us, we're in luck, since multiple binaries are available, including a linux arm64 version.

Build the correct container with the correct binary

Download the latest linux arm64 binary and extract to a folder, then check the file type:

boukegroenescheij@Boukes-MBP arm % file vcsim
vcsim: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=SuLsEHbM2lgBYi5NGMPM/8zch4MzgNtKKWKUA7Rxb/O6IVIdmx5jADCEpnaol6/YBBXW1FKuGNuEkol6nWa, not stripped

This one is the right one to use, now we can build and run the container:

boukegroenescheij@Boukes-MBP arm % docker build -t bouke/vcsim -f /Users/boukegroenescheij/go/pkg/mod/github.com/vmware/This email address is being protected from spambots. You need JavaScript enabled to view it..2/Dockerfile.vcsim .
[+] Building 2.0s (15/15) FINISHED
=> [internal] load build definition from Dockerfile.vcsim 0.0s
=> => transferring dockerfile: 1.32kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/golang:1.17.5-buster 1.1s
=> [auth] library/golang:pull token for registry-1.docker.io 0.0s
=> [build 1/4] FROM docker.io/library/golang:1.17.5-buster@sha256:007c4b51431057912f538a180a486e06c972e5462dce1fffbbd3e1ba229949d6 0.0s
=> [internal] load build context 0.8s
=> => transferring context: 30.41MB 0.8s
=> CACHED [build 2/4] WORKDIR /go/src/app 0.0s
=> CACHED [build 3/4] RUN adduser --disabled-password --gecos "" --home "/nonexistent" --shell "/sbin/nologin" --no-create-home --uid "10001" "appuser" 0.0s
=> CACHED [build 4/4] RUN mkdir /temporary-tmp-directory && chmod 777 /temporary-tmp-directory 0.0s
=> CACHED [stage-1 1/5] COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ 0.0s
=> CACHED [stage-1 2/5] COPY --from=build / etc/passwd / etc/passwd 0.0s
=> CACHED [stage-1 3/5] COPY --from=build /etc/group /etc/group 0.0s
=> CACHED [stage-1 4/5] COPY --chown=appuser --from=build /temporary-tmp-directory /tmp 0.0s
=> CACHED [stage-1 5/5] COPY vcsim /vcsim 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:70468ba740d304176dbf5ffe5ce7904dc3a7f71c485758a258665a3116643f6a 0.0s
=> => naming to docker.io/bouke/vcsim 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
boukegroenescheij@Boukes-MBP arm % docker run -p 8989:8989 bouke/vcsim:latest /vcsim version
Build Version: 0.27.2
Build Commit: f04d77d6
Build Date: 2021-11-23T21:22:18Z

So, no errors, no warnings!

And also Docker Desktop isn't showing errors anymore:

Next steps

I've created the container ready for you to download from my docker hub page:

https://hub.docker.com/r/bouke/vcsim

So pull it:

docker pull bouke/vcsim

And run it:

docker run -p 8989:8989 bouke/vcsim:latest /vcsim version

Should work! Now create something 'large':

docker run -d -p 8989:8989 --name vcsim-large bouke/vcsim:latest /vcsim -dc 4 -folder 10 -ds 8 -pod 2 -nsx 2 -pool 2 -app 1 -vm 8 

Conclusion

So, creating your own container for a specific platform can be useful so I learned a lot here. I hope this article also helped you too. I plan to work a bit more with this setup and if I find something interesting, I'll write a new post about it. Cheers!