Upgrading a .NET standard 2.0 Azure function using Table Storage to a .NET6 function

To reassure my regular readers - no, I am not pivoting from Mixed Reality and HoloLens. Far from that! But you see, any HoloLens application that is more than a simple demo needs some kind of back end. My HoloLens application AMS HoloATC, first published to the Microsoft Store in Q3 2016, uses an Azure function for the complex CPU-intensive calculations, needed to transform the raw aircraft location data coming from ADS-B Exchange into something HoloLens can project. In fact, the very first post I wrote about AMS HoloATC specifically handles data services. More importantly, it caches results of a call to ADS-B Exchange for 20 seconds - as not to annoy the good folks running that site. I donate regularly to them and have a ‘feeder’ myself, but it’s no use overextending your goodwill.

Back then in I create the first function in .NET 4.5. Yikes. In 2018 I upgraded to a .NET standard 2.0 function, but recently Microsoft have started to mail me my old functions won’t be supported indefinitely. Time to get moving. After all, how hard could it be. I only used bit of code and some caching in Table Storage.

Yeah, right.

Current setup

The current solution exists out of three projects: one only containing the function, one containing all other objects, and one containing a few tests to prove it all works

So we have this very little object that’s stored into an Azure Table as JSON text. In my actual code, this contains a list of flights. I this sample, I just add some demo data to make my point.

namespace TableUpgrade.Data.JsonResult
{
    public class FlightSet
    {
        public string SomeProperty { get; set; }

        public string SomeOtherProperty { get; set; }

        public bool IsCachedValue { get; set; }
    }
}

To this end, we employ this Table Entity:

using Microsoft.WindowsAzure.Storage.Table;

namespace TableUpgrade.Data.Storage
{
    public class FlightSetEntity : TableEntity
    {
        public string Airport { get; set; }

        public string FlightSetJson { get; set; }
    }
}

I also created these little helper extension functions making storing, retrieving and deleting Table Entities easier. At least for my usage ;). The fun part of that is - it turned out it made upgrading easier as well.

using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage.Table;

namespace TableUpgrade.Data.Storage
{
    public static class CloudTableExtensions
    {
        public static async Task<T> GetAsync<T>(this CloudTable t, 
            string partitionKey, string rowKey) where T: TableEntity
        {
            var operation = TableOperation.Retrieve<T>(partitionKey, rowKey);
            var result = await t.ExecuteAsync(operation);
            return (T)result.Result;
        }

        public static async Task<T> GetAsync<T>(this CloudTable t, string rowKey) 
            where T : TableEntity
        {
            return await GetAsync<T>(t, "1", rowKey);
        }

        public static async Task<TableResult> StoreAsync<T>(this CloudTable t, 
            T entity, string partitionKey, string rowKey) where T : TableEntity
        {
            entity.PartitionKey = partitionKey;
            entity.RowKey = rowKey;
            return await t.ExecuteAsync(TableOperation.InsertOrReplace(entity));
        }

        public static async Task<TableResult> StoreAsync<T>(this CloudTable t,
            T entity, string rowKey) where T : TableEntity
        {
            return await t.StoreAsync(entity, "1", rowKey);
        }

        public static async Task<TableResult> DeleteAsync<T>(this CloudTable t,
            T entity) where T : TableEntity
        {
            return await t.ExecuteAsync(TableOperation.Delete(entity));
        }
    }
}

You may think of it whatever you like, my main shtick is writing HoloLens apps, not Azure functions. The trick with the fixed partition key here is because for my app, I don’t want to bother with partitions. After all, I store only one call result per airport, and so far, only six airports in the world are supported.

Then we have this ‘brilliant’ piece of code, that basically is a wrapper around the whole caching mechanism, so the Azure function itself does not have to deal with the Entity. It just need to know about it’s payload FlightSet:

using TableUpgrade.Data.JsonResult;
using TableUpgrade.Data.Storage;
using Microsoft.WindowsAzure.Storage.Table;
using Newtonsoft.Json;
using System;
using System.Threading.Tasks;

namespace TableUpgrade.Data.Components
{
    public class FlightCache
    {
        private readonly CloudTable _flightCacheTable;
        private readonly string _airport;

        public FlightCache(CloudTable flightCacheTable, string airport)
        {
            _flightCacheTable = flightCacheTable;
            _airport = airport;
        }

        public async Task<FlightSet> GetCachedFlightSet()
        {
            var cachedData = await
               _flightCacheTable.GetAsync<FlightSetEntity>(_airport);
            if (cachedData != null)
            {
                var duration = (DateTimeOffset.UtcNow - 
                                cachedData.Timestamp.UtcDateTime).Duration();
                if (duration < TimeSpan.FromSeconds(20))
                {
                    cachedData.FlightSetJson = cachedData.FlightSetJson;
                    return JsonConvert.DeserializeObject<FlightSet>(
                        cachedData.FlightSetJson);
                }
            }
            return null;
        }

        public async Task SetCachedFlightSet(FlightSet flightSet)
        {
            var setToCache = new FlightSetEntity
            {
                Airport = _airport,
                FlightSetJson = JsonConvert.SerializeObject(flightSet),
                Timestamp = DateTimeOffset.UtcNow
            };
            await _flightCacheTable.StoreAsync(setToCache, setToCache.Airport);
        }
    }
}

As you can see, the code tries to retrieve data for the current airport - it’s three letter IATA designation serves as row key. If it finds something, it checks if it’s less than 20 seconds old. If so, it returns the cached value, if not - it returns null, signaling to caller ‘good luck, find something fresh yourself’.

And then the actual function, itself:

using System;
using System.Threading.Tasks;
using TableUpgrade.Data.Components;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.WindowsAzure.Storage.Table;
using Newtonsoft.Json;
using TableUpgrade.Data.JsonResult;
using System.Diagnostics;

namespace TableUpgrade
{
    public static class FlightData
    {
        [FunctionName("FlightData")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)]
            HttpRequest req,
            [Table("FlightDataCache", "AzureWebJobsStorage")]
            CloudTable resultsCacheTable)
        {
            var airport = req.Query["airport"];
            if (airport.ToString() == string.Empty)
            {
                airport = "XYZ";
            }
            var flightCache = new FlightCache(resultsCacheTable, airport);

            var flights = await flightCache.GetCachedFlightSet();
            if (flights == null)
            {
                flights = GetSomeRandomData();
                await flightCache.SetCachedFlightSet(flights);
                Debug.WriteLine("New flightset");
            }
            else
            {
                flights.IsCachedValue = true;
                Debug.WriteLine("Cached flightset");
            }

            return new OkObjectResult(JsonConvert.SerializeObject(flights));
        }


        private static FlightSet GetSomeRandomData()
        {
            return new FlightSet { 
                SomeProperty = $"Some property {new Random(25)}", 
                SomeOtherProperty = $"Some property {new Random(35)}" };
        }
    }
}

Basically:

• Try to find the data in the cache
• If no cached data, get fresh data (using GetSomeRandomData) and store in cache
• If cached data found, great
• return whatever data is found (cached or not cached).

All rather straightforward. Except that it’s like 5 years out of date. Like I clearly got told when I ran the function locally.

or when you run the tests:

Starting the upgrade process

To make life a bit easier, we first unload the TableUpgrade project - the one with the function in it. Then we upgrade both the TableUpgrade.Data and TableUpgrade.Test to .NET 6.0, using the project’s property pages

TableUpgrade.Data uses the Microsoft.Azure.WebJobs.Extensions.Storage NuGet package, version 3.08. This is very much outdated. Now you can of course upgrade it. That’s what I did initially. But believe me, later you will run into a badly documented hell-hole of missing Table binding attributes in Microsoft.Azure.WebJobs.Extensions.Storage 5.0.0 - requiring you either to forego table binding and write that code yourself, or go insane ;).

For the sake of simplicity, believe me when I say that in TableUpgrade.Data you should:

• Uninstall the package Microsoft.Azure.WebJobs.Extensions.Storage
• Install the package Azure.Data.Tables
• Install the package Microsoft.Extensions.Primitives

This will send Visual Studio into conniptions and yield you a list of 25 errors.

Fixing CloudTableExtensions

You see - there’s no CloudTable in either package we installed, so we might as well just delete the whole CloudTableExtensions file. And replace it by this. Say hello to your new friend, TableClientExtensions.

using System.Threading.Tasks;
using Azure.Data.Tables;

namespace TableUpgrade.Data.Storage
{
    public static class TableClientExtensions
    {
        public static async Task<T> GetAsync<T>(this TableClient t, 
            string partitionKey, string rowKey)
            where T: class, ITableEntity, new()
        {
            try
            {
                var result = await t.GetEntityAsync<T>(partitionKey, rowKey);
                return result;
            }
            catch
            {
                return null;
            }
        }

        public static async Task<T> GetAsync<T>(this TableClient t, 
            string rowKey)
            where T : class, ITableEntity, new()
        {
            return await GetAsync<T>(t, "1", rowKey);
        }

        public static async Task<bool> StoreAsync<T>(this TableClient t,
            T entity, string partitionKey, string rowKey) 
            where T : class, ITableEntity, new()
        {
            entity.PartitionKey = partitionKey;
            entity.RowKey = rowKey;
            var result = await t.UpsertEntityAsync(entity);
            return !result.IsError;
        }

        public static async Task<bool> StoreAsync<T>(this TableClient t, 
            T entity, string rowKey) where T : class, ITableEntity, new()
        {
            return await t.StoreAsync(entity, "1", rowKey);
        }

        public static async Task<bool> DeleteAsync<T>(this TableClient t, 
            T entity) 
            where T : class, ITableEntity, new()
        {
            var result = await t.DeleteEntityAsync(entity.PartitionKey,entity.RowKey);
            return !result.IsError;
        }
    }
}

Which looks remarkably like it’s predecessor, only it uses a different object to make extensions methods for. It will all become much clearer later on. Let’s mosey on to the next object.

Upgrading FlightSetEntity

This looks dead easy, since there is an TableEntity in using Azure.Data.Tables too. unfortunately, that’s a sealed class.

… for what I hope is a very good reason. Anyway, if you have paid attention to the TableClientExtensions you might have seen referrals to ITableEntity. We can implement that interface, which requires us to implement for extra properties we previously inherited from TableEntity:

using System;
using Azure;
using Azure.Data.Tables;

namespace TableUpgrade.Data.Storage
{
    public class FlightSetEntity : ITableEntity
    {
        public string Airport { get; set; }

        public string FlightSetJson { get; set; }
        public string PartitionKey { get; set; }
        public string RowKey { get; set; }
        public DateTimeOffset? Timestamp { get; set; }
        public ETag ETag { get; set; }
    }
}

Upgrading FlightCache

That is now getting quite simple:

• Change using Microsoft.WindowsAzure.Storage.Table; into using Azure.Data.Tables;
• Change both instances of CloudTable to TableClient

But then there’s this pesky line:

var duration = (DateTimeOffset.UtcNow -
                cachedData.Timestamp.UtcDateTime).Duration();

That complains about UtcDateTime not existing. This is because, apparently, TimeStamp is now nullable. Great. So I changed this to this:

var duration = (DateTimeOffset.UtcNow - 
                cachedData.Timestamp.Value.UtcDateTime).Duration();

Since we explicitly set Timestamp to a value in FlightCache.SetCachedFlightSet I am going to assume the value is always there.

And at this point, my friends, we are down to zero errors in the TableUpgrade.Data project.

Preparing actual function upgrade

Maybe I am being paranoid, but with so much time between the platform my code was using and the new one, I tend to start with a fresh top level project, and copy all the stuff back in.

• First I deleted TableUpgrade from the Visual Studio solution. Mind you, that does not delete anything from disk, it just removes the reference from the solution
• Then I renamed TableUpgrade to TableUpgrade_old
  
• And then I created a fresh new project function project

I made a project reference from TableUpgrade to TableUpgrade.Data, and then I copied my function in from TableUpgrade_old. This gives remarkably little errors.

So we are nearly done? Yes… and no.

Upgrading the TableUpgrade function

Here be dragons. The nasty thing is - CloudTable is known here. But the Table attribute is gone. You get the suggestion to import System.ComponentModel.DataAnnotations.Schema but that attribute is not compatible with CloudTable. You have to understand that CloudTable still comes from Microsoft.WindowsAzure.Storage.Table. So we are going - again - for drastic here.

First remove using Microsoft.WindowsAzure.Storage.Table;. You have now still have three errors, but the last one changed: now Visual Studio complains about not having a reference to CloudTable at all.

You will need to install the following packages into the TableUpgrade project:

• Microsoft.Azure.WebJobs.Extensions.Storage
• Microsoft.Azure.WebJobs.Extensions.Tables

At the moment of this writing, Microsoft.Azure.WebJobs.Extensions.Tables is still in beta - 1.0.0 beta2 to be precise, so you will have check this checkbox:

To even see it.

Now you go back to the TableUpgrade function. You add using Azure.Data.Tables; to your usings, and change CloudTable to TableClient

And once again… that lovely sight of zero errors. Don’t forget to delete the “Function1.cs” file that was created when you re-created the TableUpgrade project. We don’t need it. And the TableUpgrade_old folder can go too.

Proof of the pudding

The TableUpgrade.Tests project contains two simple tests: one to check if the service responds at all (CheckSingleHit) - the other, CheckDoubleHit checks if you get uncached data after 22 seconds, and if you do get cached data if you try again half a second after the first call.

The annoying thing is, you still can’t run unit tests in a running project, so what I usually do with these kind of integration test is open a second instance of Visual Studio and open the project again - and then run the test runner from that second instance.

The test are like this:

public class LocalTest
{
    [Fact]
    public async Task CheckSingleCall()
    {
        var result = await GetCurrentFlightSet();
        Assert.NotNull(result);
    }

    [Fact]
    public async Task CheckDoubleHit()
    {
        await Task.Delay(22000);
        var result = await GetCurrentFlightSet();
        Assert.False(result.IsCachedValue);
        await Task.Delay(500);
        result = await GetCurrentFlightSet();
        Assert.True(result.IsCachedValue);
    }

    private async Task<FlightSet> GetCurrentFlightSet()
    {
        var urlData = "http://localhost:7071/api/FlightData?airport=XYZ";

        var request = new HttpRequestMessage(HttpMethod.Get, urlData);
        using (var httpClient = new HttpClient())
        {
            var response = await httpClient.SendAsync(request);
            response.EnsureSuccessStatusCode();
            var result = await response.Content.ReadAsStringAsync();
            return JsonConvert.DeserializeObject<FlightSet>(result);
        }
    }
}

And if you did everything right - this is the result

Conclusion

Azure was launched in 2008, so it’s now 14 years old. No wonder stuff gets deprecated. Still, I find the whole stuff around Table Storage pretty messy. As far as I have been able to understand: first you had ye olden Table Storage API as I showed it, then came an upgrade, but that broke the Table attribute parameter binding. Parallel you had the ComosDB table access code, and finally the unification came with Azure.Data.Tables. And that still came with without Table attribute binding. Only with a beta package - Microsoft.Azure.WebJobs.Extensions.Tables - we now have API parity with something that worked fine in 2018.

I am not familiar with the why and how, as I am not intimately involved with the Azure development community, but the whole API upgrade path at least appears to be a bit sloppy. Maybe Table Storage is not so important anymore. However, I can guarantee you it’s pretty hard to piece all this information together - it’s all over various forums in bits and pieces, so that’s why I described it. I hope I save at least a few people the scavenging hunt I had to do ;)

Project - of course - can be found on Github. The main branch contains the upgraded code, if you want to see the old code, it’s in branch oldcode.