Mobile First Cloud First

A blog by Geert van der Cruijsen on Apps, Cloud & ALM

Category: Serverless

Connecting to Azure Blob Storage events using Azure Event Grid

I was looking for a solution to sync an Azure blob storage containing images to a third party solution that contained an API to upload images. my initial thoughts were to hook up Azure Functions to react on Azure Blob Storage triggers. One thing that is not possible with blob storage triggers is to act on delete. there is only a trigger on adding or changing files.

Luckily Microsoft announced a new solution called Event Grid a few months back. Event Grid is great for connecting events that come from azure resources (or custom resources) to things like Azure Functions or Logic Apps.

eventgrid

Event Grid also supports events for Blob Storage where you get events for adding, changing or deleting items. So how to get started?

Event Grid is easy to setup and exists out of 2 parts. Topics and Subscriptions. Topics are places where events are sent to by Azure resources or even custom publishers. Subscriptions can be made on topics and will receive the events from a certain topic.

Creating the Storage Account / Event Grid Topic

Azure Blob storage has an Event Grid topic built in so you don’t have to actually create a separate Event Grid Topic. At the time of writing Event Grid is only available in West Central US and US West 2 regions so if you create a Storage account there i’ll automatically also get an Event Grid Topic.

Let’s create the topic using Azure CLI

#create resourcegroup
az group create -n <<ResourceGroupName>> -l westus2
#create storage account 
az storage account create \    
   --location westus2 \ 
   --name <<NameOfStorageAccount>> \  
   --access-tier cool \    
   --kind blobstorage \    
   --resource-group <<ResourceGroupName>> \ 
   --sku Standard_LRS \

Or using the Azure Portal:

Screen Shot 2017-12-06 at 15.12.33

When we open the storage account in the azure portal we’ll see that in the left menu is an option called Event Grid. In this menu you can see a list of all subscriptions to this Event Grid Topic. Current we don’t have one so lets take a look on how we can do that.

Creating the Event Grid Subscriptions

Since a topic can have multiple subscribers let’s add 2 different subscribers. First we’ll create a very simple subscription that allows us to see what the events actually look like and after that we’ll take a look in adding an Azure Function that handles the events.

Simple test subscription using Requestb.in

Requestb.in is a website where you can request a simple URL that collects all http messages sent to that URL. This is a free service and will keep the last 20 messages for a maximum amount of 48 hours. As soon as we created our requestbin we’ll receive a URL that looks like something like this: https://requestb.in/1ckzahm1

we can add this URL as a web hook subscription to the topic we created earlier. this can be done either using Azure CLI or the Azure portal.

az eventgrid resource event-subscription create    
   --endpoint "https://requestb.in/1ckzahm1" \    
   --name requestbinsubscription \  
   --provider-namespace Microsoft.Storage \  
   --resource-type storageAccounts \  
   --resource-group <<ResourceGroupName>> \   
   --resource-name cloudinarysync

Or in the Azure portal:

 Screen Shot 2017-12-06 at 15.15.18

when we upload a file to the storage account now we are able to see the event that was triggered by going to the requestb.in inspect web page. We’ll see the json payload containing the event details so we can use this later in our Azure function.

Screen Shot 2017-12-06 at 15.19.03

Example json for file adding and deleting looks like the following:

File Add/Change event

File Deleted event

So we’ve seen the easiest way to hook up the Storage account events using Event Grid. Let’s add an Azure function that actually does something with the events.

Azure Function

When you create a new Azure Function you’ll have to choose the trigger type. you can choose between several options here like a Http Trigger, Webhook Trigger or Event Grid Trigger.  When looking at the list you might think Event Grid Trigger is the way to go. I tend to disagree for now. The events sent from the Event Grid are just play POST http messages so choosing a webhook trigger or Http trigger works just as good and they are even easier to test locally. The thing that Event Grid Trigger adds is that it maps the json payload shown above to a typed object so you can immediatly use it without parsing the json yourself. Another downside from Http Trigger and Webhook trigger is that you’ll have to arrange security yourself since everyone who knows the url could call the webhook by default. Let’s look at both options.

If you are using the portal use the small link below “Get started on your own” called “Custom Function” to choose the trigger type.

Screen Shot 2017-12-06 at 20.10.54

Screen Shot 2017-12-06 at 20.12.47 Screen Shot 2017-12-06 at 20.12.59 Screen Shot 2017-12-06 at 20.13.11

Event Grid Trigger

When you open your event grid trigger (if you created it via the portal but also for precompiled functions that you uploaded there is a link on the top right called “Add Event Grid Subscription”. Click this to set up the Event Grid Trigger.

Screen Shot 2017-12-06 at 20.14.06

 

 

In the “Create Event Subscription” window select the “Storage Account” topic type and select your storage account. After pressing Create your Azure Function will be triggered after each change in the storage account.

Screen Shot 2017-12-06 at 20.14.31

 

Http / Webhook Trigger

Webhook or Http Triggers work almost the same way. in the Portal there is a link on the top right to get the Function URL. When you click this  you’ll see a popup with the endpoint that you should copy. after this adding the subscription works exactly the same as i described above for the requestb.in except now you’ll enter this URL you just copied.

Screen Shot 2017-12-06 at 20.15.23

Now we’ve set up the plumbing we can start writing our function. i’ll show you the code for a http trigger but an Event grid Trigger function would like almost the same. you can skip the parsing of the json there because you get a typed  object as parameter containing all information.

 

Pitfalls & Tips

So if everything is correct you should have a working Azure Function now. But how to track the events that are coming in? You could set up application insights and track usage yourself but a nice feature of Event Grid is that it has build in logging and metrics. The metrics itself are a bit hidden in the Azure Portal so i’ll explain how to find them.

The subscriptions itself cannot be found in the resource group of the topic. When navigating to the storage account and clicking Event Grid you can get a list of your subscriptions but no metrics.

For metrics of the subscriptions you have to go to your left menu in the Azure portal and click the > arrow for more resources. search for Subscriptions and here the Event Grid Subscriptions will show up here

Screen Shot 2017-12-06 at 20.53.32

 

when you go here you’ll get an overview of all events that were triggered by a Topic and the subscriptions connected to the topic. You can also see when the Event Grid did retries and which events completed successfully or failed. It took me a while to find this so hopefully this helps more people find it.

Screen Shot 2017-12-05 at 17.29.52

 

Another thing to note is that right after creation of the subscription it might take a while for the events to start firing. I don’t know if this is something that is related to the preview state of Event Grid or if it will always be the case.  In the end all events fired but it took a while for the first events to be fired. After it was running for a while the events were triggered really fast. even when i did a large amount of changes the events were fired within seconds.

Hopefully this is useful for you developers who want to react to triggers from Azure Storage.

Happy Coding!

Geert van der Cruijsen

Building, testing and deploying precompiled Azure Functions

Azure functions are great to build small specialized services really fast. When you create an Azure Functions project by using the built-in template from the SDK in Visual Studio you’ll automatically get a function made in a CSX file. This looks like plain old C# but in fact it is actually  is C# Script. When you’re deploying these files to Azure you don’t have to compile them locally or on a build server but you can just upload them to your Azure Storage directly.

In the last update for Azure Functions the option to build precompiled functions was added. Doing this is actually pretty simple. I’ve created a sample project on Github containing a precompiled Azure function, unit tests for the function and an ARM template to deploy the function. Lets go over the steps to create a precompiled Azure function.

Continue reading