CICD Archives - Mobile First Cloud First

I love the new Github Actions and all the options it gives to automate things in my workflow such as automating CICD.

When i got added to the beta i wanted to try out if i could create a full CICD pipeline that would first build my code and release it afterwards to Azure. I used Azure DevOps quite a lot in the past and what i loved there was the separation between build and release steps where build creates an immutable artifact that will be deployed in the release stage.

gh actions

I wanted to recreate something similar so my approach was the following:

Create a build workflow that compiles my code
Let the build workflow create a release in Github containing artifacts that could be downloaded later on.
create a release workflow that downloads the release artifacts and deploys them to my Azure environment.

Creating the workflow that compiles code isn’t that hard. There are plenty of samples out there and Github helps you with a starter workflow by checking what kind of code is in your repo as well. Then it was time to create a release in my workflow and upload the artifacts to it. First thing i did was search the marketplace and look for tasks that could do this for me. But after not finding anything i thought to myself: Shouldn’t it be possible to do this from the command line? When building these workflows i see myself doing more and more plain command line tasks instead of using some marketplace tasks that are often not that much more than just a wrapper on top of some CLI API.

Github has a hub CLI that can automate a lot of things from the command line such as creating releases so my thoughts were just calling some hub commands and everything would be OK.. Well it wasn’t that simple because the hub cli isn’t installed by default on the build agents. To fix this I created my own marketplace task to install this CLI to the build agent and from then on you can just use all hub commands from within your Actions workflow.

Creating a release using the hub cli is easy. Just use hub release create. There is a range of other methods you could also use in your own workflow but in this post we’ll focus on the hub release part. Hub release create does need some more setting up though because creating releases does need some authentication and we’ll need some more parameters. Let’s look at all the details. But first here is a sample full build workflow.

Install Hub CLI

As i wrote above I’ve created a custom Github Action called setup-hub that installs the hub cli on your build agent so you can use it in your workflow. If you want to use the hub CLI yourself just add these 2 lines to the workflow steps and hub CLI will be installed and added to the path.

 - name: Install hub cli
   uses: geertvdc/setup-hub@master

Authentication

We’ll be using the hub cli from within a script block of the workflow. By default hub CLI will prompt you for an interactive login when doing authenticated calls such as creating a release. In an Actions workflow however there is already a default secret defined called GITHUB_TOKEN which you can use to authenticate. You will have to pass it into the ‘Run’ task by setting it as an environment variable

 env:
   GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

gh actions release1

The github token will execute the command using a generic “github-actions” user. This is fine in most cases but there are some limitations. Actions created by the Github Actions user will never trigger further other actions. So in my case where i wanted to trigger a deploy workflow after a release is created wouldn’t work. To change this you can use a work around that will execute the task using your own personal user. To do this create 2 secrets. 1 called GITHUB_USER containing your Github username and 1 secret called GITHUB_PAT that contains a personal access token for your user. Pass this PAT secret into the GITHUB_TOKEN environment variable and now the hub cli command will be executed as if it was executed by you.

 env:
   GITHUB_USER: ${{ secrets.GITHUB_USER }}
   GITHUB_TOKEN: ${{ secrets.GITHUB_PAT }}

gh action release 2

Release Versioning

Ok so now we can do authenticated calls towards the hub CLI. Now we can create a release but to actually create a release we’ll need a version number of some sort. What i do in my builds is just auto incrementing the previous release by 1. To do this we can use the hub cli to query the last release by calling hub release -L 1. this gives us the last release. Now we’ll add some bash script to increment this version

 version=$(hub release -L 1| awk -F. -v OFS=. 'NF==1{print ++$NF}; NF>1{$NF=sprintf("%0*d", length($NF), ($NF+1)); print}')
 echo $version

This oneliner will take the current version number in the form of v#.#.# and will increase the lastest number by 1.

now we have everything to call hub release in our workflow. using the -a tag you can upload artifacts as you can see in the full example i listed above.

hub release create -m $version -a $HOME/artifacts/frontend-$version.zip $version

That’s all. Now each time a push is made to the master branch a new release will be created and saved in your Github releases. You could hook up another workflow that triggers on a release creation using the following yaml:

 on:
   release: 
     types: [created]

Hub CLI can do far more than just releases. It can also help you in automation around pr’s,or issues. I’m really curious what others will use it for. Please let me and the other readers know in the comments if you found new cool ways of using this.

If you like my setup-hub Github Action please give it a star on Github or give feedback on Twitter

Happy Coding!

Geert van der Cruijsen

Where Continuous delivery for web applications on Azure is becoming quite popular and common I couldn’t find anything about settting this up for your API definitions in Azure API management. Since i had to set this up at my current customer I thought it was a good idea to share this in a blogpost so everyone can enjoy it. In this blogpost i’ll explain how you can set up Continuous delivery of your API definitions in Azure API management including the actual API implementation in Azure Web apps using VSTS (Visual Studio Team Services)

azure api management

First let me explain the architecture we use for our API landscape. As explained we use Azure API management for exposing the APIs to the outside world and we use Azure Web Apps for hosting the API implementation. These Web apps (Both .Net Core and Full framework .Net Web APIs) are hosted in an ASE (App Service Environment) so they are not exposed directly to the internet while we can still use all the cool things Azure Web Apps offer. These API web apps then connect to datastores hosted in Azure or connect to the on premise hosted environments through an express route or VPN.

To be able to set up our Continuous Delivery pipeline we have to arrange the following things.

Build your API implementation so we have a releasable package
Create infrastructure to host the API implementation
Deploy the API implementation package to the newly created infrastructure
Add API definition to Azure API management.
Repeat above steps for each environment. (DTAP)

Building your App

The first step can be different from my example if you’re not building your APIs using .Net technology. In our landscape we have a mix of APIs made with .Net Core and APIs made with .Net Full Framework because they needed libraries that were not available in .Net Core (yet). I’m not going into details on how to build your API using VSTS because i’ll assume you’re already doing this or you know how to do this. If not here is a link to the official documentation.

One thing to keep in mind is that your API web app does have to expose a API definition so Azure API Management can import this. We use Swashbuckle for this to automatically generate a swagger definition. If you’re using .Net Core you’ll have to use Swashbuckle.AspNetCore

Deploying the API implementation & adding it to Azure API management

For automating the deployments we’re going to the use Release Management feature of VSTS. In our first environment we’ll create steps to do all the things we described above.

The steps in our workflow are the following:

Create web application infrastructure by rolling out an ARM template
Set environment specific variables
deploy the API implementation package
Use a task group to add the API definition to Azure API management.

Creating the web app infrastructure & deploying the API Implementation package

the first and third steps are the basic steps of deploying a web application to Azure web apps. This is no different for APIs so i’ll just link to an existing blogpost here that explains these if you don’t know what they do.

Setting environment specific variables

the second task is a custom task created by my colleague Pascal Naber. It can help you overwrite specific variables you want to use in your environments by storing these settings as App Settings on your Azure web app. We use this to set the connection strings to backend systems for example a Redis Cache or a database.

Add API to API Management

So if we release the first 3 steps we would have an API that would on it’s own. But the main reason of this blogpost was that we want to have our API exposed through Azure API management so let’s have a look on how we can do that.

Azure API management has Powershell commands to interact with it and we can use this to add API definitions to Azure API management too. Below is a sample piece of Powershell that can import such an API definition from a Swagger file.

The script is built up out of 3 parts: first we retrieve the API management context by using the New-AzureRmApiManagementContext Commandlet. When we’ve gotten a context we can use this to interact with our API management instance. The second part is retrieving the swagger file from our running Web app through wget which is short for doing a GET web request. We’ll download the swagger file to a temporary disk location because in our case our web apps are running in an ASE and therefore are not accessible through the Internet. if your web apps are connected to the internet you can also directly use the URL in the 3rd command to import the Swagger file into Azure API Management. Import-AzureRmApiManagementApi.

So now we have a script that we can use to import the API let’s add it to the VSTS release pipeline we could just add the powershell script to our source control and call the powershell using the build in powershell task. I’d like to make the developers’ life in our dev teams as easy as possible so i’m tried to abstract all Powershell mumbo jumbo away from them so they can focus on their APIs. To do this i’ve created a “Task Group” in VSTS containing this Powershell task so developers can just pick the “Add API to API Management Task” from the list in VSTS and supply the necessary parameters.

When we add this task group to the release we can run our release again and the API should be added to Azure API Management.

Success!! Our initial continuous delivery process is fixed. At my current client we have 4 different API management instances and we also deploy our APIs 4x. A Development, Test, Acceptance and Production instance. The workflow we created deploys the API to our development environment. We’ve set this up to be continuous so every time a build completes on the master branch we create a new release that will deploy a new API instance to Azure and will update our Development Azure API management instance.

We can now clone this environment 3x so we create a pipeline that will move from dev, test to acceptance and production. I always set the trigger to automatically after the previous environment is completed. if we run our release again we’ll have 4 API instances deployed and in all 4 Azure API management instances they corresponding API will be imported.

Now the only thing you have to add is optionally adding integration tests to the environment you prefer and you are ready to roll!