JSON deserialization with JSON.net - caching results

On January 22 I promised this to be a three-part series. I’ve been kinda busy with upgrading apps, Windows 8 experiments and trivial ;-) stuff like code camps, an MVP summit, preparing my first and second talk about Windows Phone and whatnot and made you wait for the final part for exactly three months – but those who know me, know I stick my promises, so here’s the third and final part of my JSON for Windows Phone series.

In part 1 of this series I described the basics of creating classes from a JSON string and then simply deserializing the string into a (list of) classes. In part 2 I showed how to use JSONConverter subclasses to handle complex stuff the deserializer cannot handle out of the box, like class hierarchies. Part 3, as promised, shows a way to cache results - which makes your application faster, more responsive and more battery/data plan friendly.

Using the demo solution of part 2 as a starting point, I first brought in my wp7nl library on codeplex using NuGet. I am lazy just like any programmer (should be) and I like to defer as much heavy lifting to already existing code as I can ;-).

When dealing with data downloaded from the web everything becomes asynchronous by nature (at least on Windows Phone), and since we don’t have the await and async keyboards aboard our platform yet, I tend to use Observables from Microsoft.Phone.Reactive. To let that work with events, I need an EventArgs child class for my “loading completed” delegate, which I have defined in the following trivial way:

using System;
using System.Collections.Generic;

namespace Wp7nl.Utilities
{
  public class DataLoadCompletedArgs<T> : EventArgs
  {
    public IList<T> Result { get; set; }

    public Exception Error { get; set; }
  }
}

It’s a generic class because I am basically too lazy to write casting statements everywhere. The basic setup of the helper class that does both loading itself is like this:

using System.Linq;
using Microsoft.Phone.Reactive;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Net;
using Newtonsoft.Json;

namespace Wp7nl.Utilities
{
  public class CachedServiceDataLoader<T> where T : class
  {
    private readonly IsolatedStorageHelper<List<T>> storageHelper;

    public CachedServiceDataLoader()
    {
      storageHelper = new IsolatedStorageHelper<List<T>>();
    }

    private void FireDataLoadCompleted(IList<T> result, Exception error)
    {
      if (DataLoadCompleted != null)
      {
        DataLoadCompleted(this,
                          new DataLoadCompletedArgs<T> 
                          {Result = result, Error = error});
      }
    }

    public delegate void DataLoadCompletedHandler(object sender, 
      DataLoadCompletedArgs<T> args);

    public event DataLoadCompletedHandler DataLoadCompleted;
  }
}

So what have we here? A constructor that creates an IsolatedStorageHelper – that’s a class from the latest version from Wp7nl, where it sits in the Wp7nl.Utilities namespace. It’s basically the internal logic of the extension methods I described in my article about tombstoning MVVMLight viewmodels using SilverlightSerializer cut loose from those extension methods – so they can be used for caching all kinds of classes, and not only viewmodels on deactivation or closing of the app. The bottom part of the class is just the event indicating a data load action has been completed, and a convenience method to easily fire that event (I mentioned I was lazy, didn’t I? ;-) ) .

Next up is another little convenience methods for loading stuff from the cache or returning a default (empty) list if the results are not available:

private List<T> LoadFromStorage()
{
  return storageHelper.ExistsInStorage() ? 
     storageHelper.RetrieveFromStorage() : new List<T>();
}

It’s only used once, so I could just a easy have omitted it, but it makes the rest of the code a bit more readable and that’s important too. When you state your intentions in code, that saves on comment.

The method to read stuff from cache is a bit more complex than strictly necessary, but since data coming from the web is coming in asynchronously, why would I want my app to wait on my cache coughing up its results? So I made the cache retrieval asynchronously as well, using some old skool BackgroundWorker hoopla:

public bool StartLoadFromCache()
{
  if (storageHelper.ExistsInStorage())
  {
    var w = new BackgroundWorker();
    w.DoWork += (s, e) =>
      {
        e.Result = LoadFromStorage();
      };

    w.RunWorkerCompleted += (s, e) =>
      FireDataLoadCompleted(e.Result as List<T>, e.Error);

    w.RunWorkerAsync();
    return true;
  }
  return false;
}

I could have used an observable here as well I guess, but I still have not adapted my code snippet. So anyway, I cheat a little by checking first if there’s any cache data at all – if there is not, the method immediately returns false, informing the calling method there’s no cached data and to go get the stuff on the web. Which it can, incidentally, by calling the following method:

public void StartDownloadCacheData(IList<T> currentObjects, Uri serviceUri,
                                   params JsonConverter[] converters)
{
  var w = new SharpGIS.GZipWebClient();
  Observable.FromEvent<DownloadStringCompletedEventArgs>(
    w, "DownloadStringCompleted")
    .Subscribe(r =>
    {
      if (DataLoadCompleted != null)
      {
        if (r.EventArgs.Error == null)
        {
          var deserialized = 
               JsonConvert.DeserializeObject<List<T>>(r.EventArgs.Result,
                                                      converters);
          var result = new List<T>(currentObjects);
          result.AddRange(deserialized.Where(p => !currentObjects.Contains(p)));
          FireDataLoadCompleted(result, r.EventArgs.Error);
          storageHelper.SaveToStorage(result);
        }
        else
        {
          FireDataLoadCompleted(null, r.EventArgs.Error);
        }
      }
    });

  w.DownloadStringAsync(serviceUri);
}

So what this method does is pretty simple:

• It makes a GZipWebClient (that allows data to be loaded zipped, thus saving on amount of bytes actually transmitted)
• It makes an Observable from the event “DownloadStringCompleted” and subscribes an anonymous method
• It fires the DownloadStringAsync method on the uri

Now the anonymous method that is fired when the data arrives

• Checks for errors
• Tries to deserialize the incoming data using the provides converters (if any)
• Adds the existing object list to the result list
• Add the new objects to the list after weeding out duplicates that were already in the list
• Fires the complete event
• Stores the now merged data set in isolated storage.

So the idea is that you can do consecutive downloads, but that resulting data never contains any duplicates. This is why you must provide the list of current objects. This may seem like a little odd, but this is exactly the thing you want to do with working with geographical data – which I do almost all the time for a living. As an inspector you want to download stuff from the areas you want to inspect, not the whole municipality. You move your map to a location, hit “download” and presto, you have the stuff you want to use cached on you device before you go on the road. Repeat until all areas you want to visit today have been processed. Kinda like Nokia Drive does – there is never enough room aboard your phone to store all the possible maps of the world, but when you to Seattle, you download the maps for Washington State, not the whole of North America, and you’re good to go.

But what if you want to start over – clearing the cache in stead of adding it? That’s the last method of this class:

public void StartClearStorage()
{
  var w = new BackgroundWorker();
  w.DoWork += (s, e) => storageHelper.DeletedFromStorage();

  w.RunWorkerCompleted += (s, e) =>
  {
    if (DataLoadCompleted != null)
    {
      FireDataLoadCompleted(new List<T>(), e.Error);
    }
  };

  w.RunWorkerAsync();
}

Also asynchronously, also not strictly necessary, but Microsoft tend to move toward doing everything asynchronously – if you have done any Windows 8 development you know performance requirements are pretty high and strict, so better get used to it already.

I updated the demo solution to show off the workings of this class, that presents itself like showed to the right. If you click on “Load devices”, it will show four devices, as displayed on the image, and the message “Loading from web”. If you hit the “Load devices” button again, it will show the same four devices, but show the message “Loading from cache” and it will show them just a wee bit faster.

If you hit “Load more devices” you will see six devices, – with the Lumia 800 mentioned two times. But wait, what about weeding out the duplicates in StartDownloadCacheData – doesn’t that work? It sure does, but I already mentioned I was lazy, so I did not implement any “Equals” logic on  the Device class – that’s left as exercise for the reader ;-). I assure you it will work properly then.

“Clear list” just clears the list, and if you hit “Load devices” it will load the devices from cache again – very fast.

“Clear cache and list” will actually wipe the cache, so if you hit “Load devices” again it will show “Loading from web”.

I won’t bother you with the XAML – the code in MainPage.xaml.cs is pretty straightforward and shows off all the features of the CachedServiceDataLoader:

using System;
using System.Collections.Generic;
using System.Windows;
using Microsoft.Phone.Controls;
using Microsoft.Phone.Reactive;
using Wp7nl.Utilities;

namespace JsonDemo
{
  public partial class MainPage : PhoneApplicationPage
  {
    private CachedServiceDataLoader<Device> cachedLoader;
    private IList<Device> currentData;
    public MainPage()
    {
      InitializeComponent();
      Loaded += MainPage_Loaded;
    }

    void MainPage_Loaded(object sender, RoutedEventArgs e)
    {
      cachedLoader = new CachedServiceDataLoader<Device>();
      currentData = new List<Device>();
      Observable.FromEvent<DataLoadCompletedArgs<Device>>(
        cachedLoader, "DataLoadCompleted")
       .Subscribe(r =>
                    {
                      currentData = r.EventArgs.Result;
                      PhoneList.ItemsSource = r.EventArgs.Result;
                    });
    }

    private void Load_Click(object sender, RoutedEventArgs e)
    {
      if (!cachedLoader.StartLoadFromCache())
      {
        Message.Text = "Loading from web";
        cachedLoader.StartDownloadCacheData(currentData,
          new Uri("http://www.schaikweb.net/dotnetbyexample/JSONPhones2.txt"),
          new JsonDeviceConverter(), new JsonSpecsConverter());
      }
      else
      {
        Message.Text = "Loading from cache";        
      }
    }

    private void Load2_Click(object sender, RoutedEventArgs e)
    {
      Message.Text = "Loading from web";
      cachedLoader.StartDownloadCacheData(currentData,
        new Uri("http://www.schaikweb.net/dotnetbyexample/JSONPhones3.txt"),
        new JsonDeviceConverter(), new JsonSpecsConverter());
    }      

    private void Clear_Click(object sender, RoutedEventArgs e)
    {
      PhoneList.ItemsSource = null;
      Message.Text = "cleared list (not cache)";
    }

    private void ClearCache_Click(object sender, RoutedEventArgs e)
    {
      cachedLoader.StartClearStorage();
      Message.Text = "cleared list and cache";
    }
  }
}

There isn’t even any of my trademark MVVM code in here. In the MainPage_Loaded – not surprisingly - all the stuff is initialized. The CachedServiceDataLoader is created using a type T, and spits out a IList of T. I use an Observable to keep track of things here as well, but of course you are free to subscribe to events in the ‘old fashioned’ way.

Load_Click shows the usage of StartLoadFromCache and StartDownloadCacheData, the latter one using the JSONconverter child classes I showed in part 2. The rest I assume to be pretty straightforward.

As you can see, working with JSON on Windows Phone is dead easy and my little helper class makes it even more easy. I am still pondering if I should include this in the wp7nl library, as this creates yet two more dependencies (SharpGIS.GZipWebClient and Newtonsoft.Json) to keep in sync. If you have any feedback on this, I’d be happy to hear it. But anyway, this concludes my JSON for Windows Phone series, I hope it will prove to be a useful mini-tutorial for the Windows Phone developer community. And as usual, you can download the full solution from my website.