Azure DevOps comes with several options to use as build agents in your Azure Pipelines. Microsoft has hosted agents where you don’t have to maintain your own hardware and you can turn any machine you own into a agent by installing the agent script on that machine.
The hosted agents are packed with lots of pre-installed software to support you in your builds. If you run your own private agents you can customize them as you would like. I’m currently at a large enterprise where I’m a consultant in a IT for IT team that hosts a number of private agents for all other development teams to use. Our agents are fully set up through automation and have all common tools used by teams (Based on the Hosted Azure DevOps agent images which are open source). These Agents work for the largest group of development teams but there are always teams who need some special tools. What we do to give teams freedom on their tool selection is having them run their builds inside a docker container. This is a new feature released at the end of September 2018.
When you run builds inside a container all steps in your pipeline are executed inside this container. The work directory of the agent is volume mapped inside the container. The ability of running your pipeline in a custom container gives you all the freedom of creating an image that has all the tools required for you to execute your build. The Docker image has 2 requirements. Bash and Node.js have to be available within the container and then you’re ready to go.
How to create a containerized build pipeline
An important note is that containerized pipelines are currently only available in YAML based pipelines. I don’t know if pipelines created in the portal will eventually also support this but in my opinion YAML based pipelines are the way forward from now on because they have a lot of advantages over traditional pipelines. The official documentation on YAML pipelines can be found here.
Let’s take a simple YAML pipeline as this example. I’ve create a simple Asp.Net Core application and have set up a pipeline for that. This is what my azure-pipelines.yml file looks like.
I tried to make the build as simple as possible. It’s just a basic .Net core build that we want to execute. For this example it doesn’t matter what the exact steps are that we are executing. It could be anything, a .Net build, npm, Go, Java Maven, anything goes. We use one of the hosted agent queues to execute the build.
Next step is to make this regular build execute exactly the same steps in a container. We can do this fairly simple by adding some settings to our pipeline. You’ll need to add a container to your resources defined at the start of your YAML file. This can either be a public container from Docker hub or a container from a private repository. The container resource will receive a name. In my example “dotnet-geert”. We can use this name to set a reference to this container in our pipeline so all build steps will be executed in this container. You can do that by adding a line just below your build pool saying which container should be used. container: dotnet-geert
In this example we run our build in a Hosted Ubuntu Agent. Downsides of running this on a hosted agent is that the hosted agent won’t cache your docker image so it has to download the full image at every run. Because of this I don’t think this approach will be that effective compared to private agents which cache the docker image locally so spinning up a container is only a matter of seconds.
Running it on your local agents work exactly the same. there are however a few requirements. You either need a Linux machine with docker support or a Windows machine which runs Windows Server and has a higher version than 1803.
Performance improvements
First thing we want to do is run our build on a private agent so we can re-use our Docker images and only have to download them once. Another feature of using containers is that you’ll always receive a fresh instance of your environment. This is nice because you can be sure that every build was run exactly the same and wasn’t relying on previous changes another build might have done to your agent. It’s also something to consider when your builds take a long time. Because you’ll receive a fresh environment each build you’ll also have to download all your dependencies each build. Most applications nowadays are using a lot of external dependencies so let’s have a look on how we can fix this.
Docker has a feature called volume mappings that enables you to map certain directories from your host machine and use them in your container. In my example pipeline we’re building a .Net Core application that uses NuGet packages. We can map a folder on our host machine to function as the global NuGet cache and use this within our container. each time the container is downloading nuget packages It’s storing it outside the container and we run the same build again it can use the cached packages. Same thing would work for npm packages or maven packages when you are building applications with other technologies.
We can create the volume mapping by passing an option to our container. This option has to be -v for volume mapping and then passing in the <source folder>:<destination folder>. In the case of NuGet packages we’re also setting a global environment variable that sets the NuGet cache to this folder. After we do this our builds will be super fast and we have all the flexibility of tools that containerized builds give you. Below is a full sample pipeline that uses a volume mapping for the NuGet cache.
I really like this new feature of Azure DevOps because it gives you a lot of flexibility in your builds without having to do customize your own private agents to much.
Happy Coding! (And building!)
Geert van der Cruijsen
Recent Comments