What I learned making a Docker container for building C++ on Windows

2019 July 9

I would normally begin the title of this kind of post with "How to ...", but that would imply that I'm a knowledgable Windows developer who knows the best way to build a Docker container for the platform. Instead, I'm a Linux developer who just stumbled my way through pages of documentation, issue trackers, and blog posts until I was able to scramble together something that worked. I want to share the lessons I learned to save the next guy some trouble.

Docker on Windows 10

Until recently, the only way to run Docker containers on Windows 10 was with Hyper-V, Microsoft's own hypervisor. This is like running each container as a virtual machine, and it negatively affects performance, memory footprint, and launch time.

Even worse, Hyper-V does not coexist with other hypervisors, e.g. VirtualBox or VMWare. I develop mostly on a Linux virtual machine (VM) running on VMWare Workstation, which means every time I want to run a Hyper-V-based Windows container with Docker, I would need to suspend my VM, enable Hyper-V, and reboot my machine. To return to Linux, I would need to disable Hyper-V, reboot my machine, and resume my VM. Talk about a non-starter.

Thankfully, Windows 10 version 1809 (the October 2018 Update) and Docker Engine version 18.09.1 let us run Windows containers using process isolation, which is similar to how containers work on Linux. These are lighter weight and don't need a hypervisor, though they require the Windows kernel of the host to match that of the container. Lately, Microsoft seems to release two kernel versions every year, one in the spring named with a number ending in 03 and one in the fall with a number ending in 09. We can check our kernel version by running winver (by just typing ⊞ Windowswinver↵ Enter).

That said, Docker Desktop for Windows refuses to even start the daemon unless Hyper-V is enabled, even if I will never use Hyper-V isolation. Fortunately, there is a way around it, by installing Docker Engine - Enterprise. After it was installed, I configured Docker to always run containers with process isolation (Hyper-V is the default). I created the Docker configuration file—at %ProgramData%\Docker\config\daemon.json^[1] by default, commonly equal to C:\ProgramData\Docker\config\daemon.json—and added the execution option:

{
  "exec-opts": ["isolation=process"]
}

Visual Studio Build Tools

For a Windows build environment, I need the Microsoft Visual C++ (MSVC) compiler toolchain. I quickly found some great documentation from Microsoft for installing Visual Studio Build Tools (VSBT) in a Windows container:

A basic tutorial.
An advanced version that adds channel selection and an arcane method for collecting logs in a ZIP archive (Microsoft seems allergic to printing diagnostics to stderr).
A blog post walking all the way through a compilation.
A GitHub repository with the Dockerfile used in the blog post.

VSBT is an installer for the tools required to build C++ projects from the command-line, without installing the Visual Studio IDE. This saves us the cost of installing a GUI application that we cannot launch in a container anyway, resulting in a faster docker build and a smaller image.

VSBT is a newer installer alongside more traditional installers (which Microsoft likes to call "bootstrappers") for Visual Studio Community (VSC), Professional, and Enterprise. (VSC is the free one.) These installers are small, under 1.44 MB, just large enough know where on the internet to fetch the rest of the desired components. Each installer knows about a different set of components, which are grouped into "workloads". Installing a workload installs all of its required components, with the option to further install its recommended and optional components. The installers have a command-line interface for setwise assembling the collection of components to install, starting with nothing or everything, and then adding or removing whole workloads or individual components as needed.

VSBT has the lightweight (~350 MB) Microsoft.VisualStudio.Component.VC.CoreBuildTools component (compare to the ~1290 MB Microsoft.VisualStudio.Component.VC.CoreIde component from VSC), but it doesn't have everything I want in a build environment. VSC has Python (Component.CPython3.x64) and Git (Microsoft.VisualStudio.Component.Git). I tried copying the pattern in Microsoft's Dockerfile for VSC:

# Install MSVC C++ compiler, CMake, and MSBuild.
RUN C:\TEMP\install.cmd C:\TEMP\vs_buildtools.exe `
    --quiet --wait --norestart --nocache `
    --installPath C:\VisualStudio `
    --channelUri C:\TEMP\VisualStudio.chman `
    --installChannelUri C:\TEMP\VisualStudio.chman `
    --add Microsoft.VisualStudio.Workload.VCTools;includeRecommended `
    --add Microsoft.Component.MSBuild `
 || IF "%ERRORLEVEL%"=="3010" EXIT 0
# Install Python and Git.
RUN C:\TEMP\install.cmd C:\TEMP\vs_community.exe `
    --quiet --wait --norestart --nocache `
    --installPath C:\VisualStudio `
    --channelUri C:\TEMP\VisualStudio.chman `
    --installChannelUri C:\TEMP\VisualStudio.chman `
    --add Microsoft.VisualStudio.Component.Git `
    --add Component.CPython3.x64 `
    --add Component.CPython3.x86 `
 || IF "%ERRORLEVEL%"=="3010" EXIT 0

I ran into trouble, however. The container built successfully, but when I ran it, the MSVC build tools were on the PATH and Python and Git were not.

Debugging

At this point, I built and ran a container with just the installers downloaded so that I could interactively test them:

FROM mcr.microsoft.com/dotnet/framework/sdk:4.8
ADD https://aka.ms/vs/16/release/channel C:\TEMP\VisualStudio.chman
ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe
ADD https://aka.ms/vs/16/release/vs_community.exe C:\TEMP\vs_community.exe

> docker build . --tag sandbox
> docker run --rm -it sandbox

I tried manually executing a command from the Dockerfile:

> C:\TEMP\vs_buildtools.exe `
    --quiet --wait --norestart --nocache `
    --installPath C:\VisualStudio `
    --channelUri C:\TEMP\VisualStudio.chman `
    --installChannelUri C:\TEMP\VisualStudio.chman `
    --add Microsoft.VisualStudio.Workload.VCTools

This didn't work: the installer returned immediately without doing anything, without even printing a diagnostic explaining why it failed. If you're an experienced Windows command-line user, you might see why, but this problem proved incredibly difficult to debug. I couldn't find anything written about it on the internet!

I tried taking out the --quiet argument thinking it might be suppressing diagnostics, but that changed nothing. I tried running the program with just the --help option, but it printed nothing. (It actually doesn't have a --help option, but it won't print any diagnostics for unrecognized options!) I tried all these commands in both cmd and powershell. Nothing.

I looked up the SHELL Dockerfile command, and then the /S and /C options for cmd:

C:\> cmd /?
Starts a new instance of the Windows command interpreter

CMD [/A | /U] [/Q] [/D] [/E:ON | /E:OFF] [/F:ON | /F:OFF] [/V:ON | /V:OFF]
    [[/S] [/C | /K] string]

/C      Carries out the command specified by string and then terminates
/K      Carries out the command specified by string but remains
/S      Modifies the treatment of string after /C or /K (see below)

If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:

    1.  If all of the following conditions are met, then quote characters
        on the command line are preserved:

        - no /S switch
        - exactly two quote characters
        - no special characters between the two quote characters,
          where special is one of: &<>()@^|
        - there are one or more whitespace characters between the
          two quote characters
        - the string between the two quote characters is the name
          of an executable file.

    2.  Otherwise, old behavior is to see if the first character is
        a quote character and if so, strip the leading character and
        remove the last quote character on the command line, preserving
        any text after the last quote character.

It seems like a long way to explain that cmd /S /C is the equivalent of sh -c. I don't see how that could make a difference, but I tried it anyway:

> cmd /S /C C:\TEMP\vs_buildtools.exe --help
> cmd /S /C C:\TEMP\vs_buildtools.exe `
    --wait --norestart `
    --installPath C:\VisualStudio `
    --add Microsoft.VisualStudio.Workload.VCTools

Still nothing!

I reached out for help on GitHub, Stack Overflow, and Twitter, tagging Heath Stewart, who is apparently The Guy who knows everything about installing Visual Studio. Heath mentioned that the installer is a "Windows app", which meant nothing to me at first, but then tickled some ancient memories about WinMain, "the user-provided entry point for a graphical Windows-based application". Then, the dots started connecting.

After playing around with a few more commands, I learned that vs_buildtools.exe --quiet --wait and cmd /S /C vs_buildtools.exe are not enough on their own; they must be used together:

> cmd /S /C C:\TEMP\vs_buildtools.exe `
    --quiet --wait --norestart `
    --installPath C:\VisualStudio `
    --add Microsoft.VisualStudio.Workload.VCTools

If the application tries to launch a graphical user interface in a container, it will silently fail, but even if it doesn't launch a GUI, it seems to return immediately unless prefaced with cmd /S /C. I still haven't found an explanation why, but it is the behavior I've observed.

I could finally run the installers successfully, but I am still disappointed that they never print any diagnostics. Logs can be captured by wrapping the commands with a special script, but it requires me to copy a ZIP file out of the container with docker cp, unzip it into many files across many directories, and collate them myself.

I have read multiple "explanations" and still have no idea how return codes are supposed to work. cmd has %ErrorLevel% and PowerShell has $? for the "last operation" and $LastExitCode for the "last Win32 executable execution". What is the difference between an "operation" and an execution? Can I assume that every executable is a "Win32 executable"? What if it isn't? In practice, I have rarely been able to get the code I'm looking for.

Microsoft, please!

Python and Git

Now that I could test the installers, I discovered that Python and Git were not being installed by my command. It seems that two different installers (e.g. VSBT and VSC) cannot install to the same --installPath, even though they share many different components.

Each installer writes to its install path a shell initialization script, VsDevCmd.bat, that modifies the PATH to include the tools it installed. If I have to use two different install paths, which initialization script should I use? Is it safe to call both? How?

Installing Git manually, I discovered that both Windows and MinGW builds of Git are installed in 3 different places each, and that these places include both (deep) within the --installPath that I chose and in C:\Program Files, the traditional installation prefix:

C:\> dir git.exe /s
Directory of C:\Program Files\Git\bin
        39,192 git.exe
Directory of C:\Program Files\Git\cmd
        39,192 git.exe
Directory of C:\Program Files\Git\mingw64\bin
     2,364,184 git.exe
Directory of C:\Program Files\Git\mingw64\libexec\git-core
     2,364,184 git.exe
Directory of C:\VisualStudio\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\cmd
        40,216 git.exe
Directory of C:\VisualStudio\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\bin
     3,015,448 git.exe

Frustratingly, these installation paths are mentioned nowhere in the documentation I've read, and the installers say nothing, and there is no uniform convention. It seems I just have to search for them.

Installing Python manually, I discovered that it is not placed on the PATH by VsDevCmd.bat, even if it is installed with a separate --installPath. My only option is to search for its installation path (%ProgramFiles(x86)%\Microsoft Visual Studio\Shared\Python37_64)^[2] and add it to my PATH myself.

Things are starting to get nasty.

Windows package managers

I would prefer to install everything with one official installer released by Microsoft, but they have a long way to go to accommodate command-line users. I had heard of Chocolatey as a package manager for Windows, much like Homebrew for OSX or APT for Debian, but my co-worker (a former Microsoft employee) recommended Scoop.

Compared to Visual Studio, Scoop makes it easier and faster to install Git and Python, installs one copy of each in one place, and automatically adds them to my PATH.

Tying it all together

I've got a working Dockerfile now. Let's step through each line to understand exactly what is going on.

# escape=`

This changes the Dockerfile escape character to backtick (`). The default escape character is backslash (\). By changing the escape character, we don't have to escape the path separator in Windows paths, and by changing it to backtick, our RUN commands look like PowerShell commands (where backtick is the escape character)^[3].

FROM mcr.microsoft.com/dotnet/framework/sdk:4.8

This is one of the featured tags of the dotnet/framework/sdk image, and the tutorial warns to "base your image on microsoft/dotnet-framework:4.8 or later".

SHELL ["cmd", "/S", "/C"]

Docker makes cmd /S /C the default SHELL on Windows, but dotnet/framework/sdk changes it to PowerShell, which has different syntax:

+ ... Tools;includeRecommended --add Microsoft.Component.MSBuild || IF %ERR ...
+                                                                ~~
The token '||' is not a valid statement separator in this version.

ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\Temp\vs_buildtools.exe
ADD https://aka.ms/vs/16/release/channel C:\Temp\VisualStudio.chman
RUN C:\Temp\vs_buildtools.exe `
    --quiet --wait --norestart --nocache `
    --installPath C:\BuildTools `
    --channelUri C:\Temp\VisualStudio.chman `
    --installChannelUri C:\Temp\VisualStudio.chman `
    --add Microsoft.VisualStudio.Workload.VCTools;includeRecommended `
    --add Microsoft.Component.MSBuild `
 || IF "%ERRORLEVEL%"=="3010" EXIT 0

The installation of VSBT is essentially copied from the sample. 3010 is the return code meaning "Operation completed successfully, but install requires reboot before it can be used", but Docker doesn't know that and will stop the build on a non-zero return code.

RUN powershell.exe -ExecutionPolicy RemoteSigned `
  iex (new-object net.webclient).downloadstring('https://get.scoop.sh'); `
  scoop install python git

Scoop recommends PowerShell for its installation, which requires no special flags, unlike cmd, to execute and wait on a command.

ENTRYPOINT C:\BuildTools\Common7\Tools\VsDevCmd.bat &&
CMD ["powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"]

By default, the container launches a PowerShell (which is more capable than cmd) with VSBT and Scoop packages on its PATH.

Caveats

There are some known issues when installing Visual Studio in a Docker container:

When docker building the image, pass --memory 2GB to be safe.
The default storage limit for images has increased to 127 GB from 20 GB (and a full install of Visual Studio Build Tools can take 58 GB), but if you run into problems, you can edit your Docker configuration at %ProgramData%\Docker\config\daemon.json to add "storage-opts": ["size=xxxGB"]. My Dockerfile, as it is now, builds an image that is 12.6 GB.

It is unclear whether I can publish my image to Docker Hub based on this statement from Microsoft Program Manager Marc Goodner:

Remember that the VS Build Tools are licensed as a supplement to your existing Visual Studio license. Any images built with these tools should be for your personal use or for use in your organization in accordance with your existing Visual Studio and Windows licenses. Please don’t share these images on a public Docker hub.

VSC is free for:

individuals (even for commercial use)
open source
academic research
training and education
up to 5 employees of a small business (defined as fewer than 250 employees and less than $1 million in annual revenue)

If I am using VSC and VSBT under these conditions, am I still prohibited from sharing an image for other users under the same conditions? Regardless, it seems I should be able to share a Dockerfile just like Microsoft.

Footnotes

Environment variables are written %VARIABLE% in the Command Prompt, or cmd, and $env:VARIABLE in PowerShell. ↩︎
Yes, it installed the x64 version of Python to the x86 installation prefix. ↩︎
The caret (^) is the escape character in cmd. Because my SHELL in the Dockerfile is cmd, I would prefer to use that escape character, but the only two values that Docker permits are backslash and backtick. ↩︎