A HoloLens airplane tracker 4–Reading data and positioning airplanes

Time to rock and roll

All right my friends, it’s time to show the heart of the matter – how to read data from an Azure Mobile App Services data service and make the airplanes appear in the right places.

Adding the JSON data object scripts

As the projects are partially regenerated, and have a total different .NET baseline, I have not found a good way to share classes as binaries between full .NET code and Unity without getting in all kinds of trouble, so I choose the easy way out: I took

• TrackType.cs
• Coordinate.cs
• Flight.cs

from the FlightDataService\DataObjects, moved them the App/Scripts/DataObjects folder in the Unity project. Then of course the service does not compile anymore, but I added the moved file as link. Then at least we have once source of truth as far as these classes are concerned.

Then we need a simple class to hold a list of flights with a timestamp (add that to the DataObjects folder as well):

using System;
using System.Collections.Generic;

namespace FlightDataService.DataObjects
{
  public class FlightSet
  {
    public FlightSet()
    {
    }

    public FlightSet(List<Flight> flights)
    {
      Flights = flights;
      TimeStamp = DateTimeOffset.Now;
    }
    public DateTimeOffset TimeStamp { get; set; }

    public List<Flight> Flights { get; set; }
  }
}

Adding a DataService to read data

I opted to add the DataService as a Singleton that can be accessed in the app. But we have a bit of an issue here -  Unity runs on Mono, e.g. .NET 3.something, and does not have things like packages for reading App services, or support async and Tasks and stuff. But remember – a HoloLens app is a two stage rocket. Unity does not generate an app for the HoloLens, but an UWP project, and that UWP project can use all the goodness from all the latest stuff. Meet the simple way to close the gap - my new friend UNITY_UWP.

using System;
using System.Collections.Generic;
using FlightDataService.DataObjects;
using HoloToolkit.Unity;
#if UNITY_UWP
using Microsoft.WindowsAzure.MobileServices;
using System.Net.Http;
using System.Threading.Tasks;
#endif

public class DataService : Singleton<DataService>
{
  public string DataUrl = "http://yourflightdataservice.azurewebsites.net/";

#if UNITY_UWP
    private MobileServiceClient _client;
#endif

  public DataService()
  {
#if UNITY_UWP
    _client = new MobileServiceClient(new Uri(DataUrl));
#endif
  }

#if UNITY_UWP
  public async Task<List<Flight>> GetFlights()
  {
    var result = await _client.InvokeApiAsync<List<Flight>>(
	                "FlightData", HttpMethod.Get, null);
    return result;
  }
#endif
}

Code between #if UNITY_UWP – #endif blocks is invisible to Unity – but will be accessible in the UWP HoloLens app that is generated by Unity. You will see Unity swallows it – no problems at all. There is only this tiny thing – you cannot add NuGet packages to Unity. So we have to do that in the UWP app, in Visual Studio. But… the UWP app is generated from Unity by File/Build and overwritten. The trick is to know not the whole app is overwritten, but only part of it. What I did was the following:

• Generate the app in a subfolder “App” of the Unity project. This should be <your_root>\HoloATC_Demo\AMS HoloATC Demo\App
• Open the solution in HoloATC_Demo\AMS HoloATC Demo\App (not the one in HoloATC_Demo\AMS HoloATC Demo)
• Go to the project Assembly-Csharp
• Add the NuGet Package Microsoft.Azure.Mobile.Client
• Find the project.json file that belong to this project. You will find it’s not under AMS HoloATC Demo\App at all, but in AMS HoloATC Demo\UWP\Assembly-CSharp\project.json
• This is the only file that has been changed with respect to the generated code. You will have to add this to source control. Even if Git claims it’s an ignored file.
• After that – if you checkout the source in a different location or on a different machine, you will only have to regenerate the solution in the right place (the App folder), and revert this specific file. You will need to do this, or else the UWP app won’t compile.

Adding the AircraftLoader to create and update aircraft

I have created one class that is actually responsible for actually getting the aircraft data from the DataService – and one that is responsible for creating, updating and deleting aircraft. The updating – moving to a new position – is handled by the AircraftController – basically all the AircraftLoader says to the airplane is ‘here is new data for you – handle it’

Remember – it’s always best to start in Unity. So create an AircraftLoader and an AircraftController script in Assets/App/Scripts. Then proceed to double-click on AircraftLoader. This will open a Visual Studio instance. Be aware – this is not the project your will be running. This is the other solution, the one that is in the project root – I call this the ‘Edit’ solution. First, we are going to make sure the “usins” of the class are properly organized – that is, in a way that doesn’t make Unity go belly-up:

using System;
using UnityEngine;
using System.Collections.Generic;
using System.Linq;
using FlightServices.Data;
#if UNITY_UWP
using System.Threading.Tasks;
using Windows.Web.Http;
using Newtonsoft.Json;
#endif

Then we are going to add a whole lot of fields. And yes, some are public. These are things that can be set from the Unity editor. I have talked about that before.

public GameObject Aircraft;

public string TopLevelName = "HologramCollection";

private Dictionary<string, GameObject> _aircrafts;

private readonly TimeSpan _waitTime = TimeSpan.FromSeconds(5);

private DateTimeOffset _lastUpdate = DateTimeOffset.MinValue;

private Queue<FlightSet> _receivedData;

private GameObject _topLevelObject;

• The Aircraft GameObject field will be used to drag our AircraftHolder onto – so this behaviour knows which game object to create
• TopLevelName is the name of the parent object to which all the aircraft are added to. This  “HologramCollection” by default
• _aircrafts is a dictionary of aircraft game objects with their id as key, so we can update/delete existing aircraft game objects based upon the data coming in
• waitTime – minimal time to load new data
• _lastUpdate – the time the last update of aircraft was completed
• _receivedData – a queue with data coming from the service. I use this pattern all the time. I queue up data from the service, then read it in the Update method that Unity calls automatically 60 times a second. Don’t ever try to update game objects from .NET callback methods or events – you might regret it, or just get plain crashes – kind of like happens in UWP XAML apps, where you need the Dispatcher to take care of that
• _topLevelObject – to store the game object with the TopLevelName in it. I have been told the method I use to find an object by name is quite heavy on performance – so better do it once and retain the result, right.

The Start method simply initializes some stuff:

void Start()
{
  _aircrafts = new Dictionary<string, GameObject>();
  _receivedData = new Queue<FlightSet>();
  _topLevelObject = GameObject.Find(TopLevelName);
}

And then there’s this little method – that is used as callback for the DataService’s GetFlight method:

#if UNITY_UWP

  private void ProcessData(Task<List<Flight>> flightData)
  {
    if (flightData.IsCompleted && !flightData.IsFaulted)
    {
      var set = new FlightSet(flightData.Result);
      _receivedData.Enqueue(set);
    }
  }
#endif

In short - when the data received and it is ok, just add it to the queue of received data. And then let Update handle it. As a matter of fact – like this:

private bool _isUpdating;
void Update()
{
  if ((_lastUpdate - DateTimeOffset.Now).Duration() > _waitTime)
  {
    _lastUpdate = DateTimeOffset.Now;
#if UNITY_UWP
    DataService.Instance.GetFlights().ContinueWith(ProcessData);
#endif
  }
  if (!_isUpdating)
  {
    _isUpdating = true;
    if (_receivedData.Any())
    {
      var set = _receivedData.Dequeue();
      var flightIds = set.Flights.Select(p => p.Id).ToList();

      var aircraftToDelete = 
        _aircrafts.Keys.Where(p => !flightIds.Contains(p)).ToList();
      DeleteAircraft(aircraftToDelete);

      var keysToUpdate = _aircrafts.Keys.Where(p => flightIds.Contains(p));
      var aircraftToUpdate = 
        set.Flights.Where(p => keysToUpdate.Contains(p.Id)).ToList();
      UpdateAircraft(aircraftToUpdate);

      var aircraftToAdd = 
        set.Flights.Where(p => !_aircrafts.Keys.Contains(p.Id)).ToList();
      CreateAircraft(aircraftToAdd);
    }

    _isUpdating = false;
  }
}

First it finds out if it’s necessary to download new data, but downloading and adding happens asynchronously. Then, if there’s any data in the queue, it first makes a list of all the id’s in the newly received flights.

• Aircrafts that are in the aircraft game object dictionary (with the flight id as key) but are no longer in the list of flights, can be deleted (they have landed or moved out of Dutch airspace)
• Aircraft that appear in that dictionary and in the list of flights need to be updated – possibly moved to a new position
• Aircraft that do not have an entry in the game object dictionary are new, and need to be created.

It’s not that hard, see ;). The methods for creating, updating and deleting aircrafts are not that hard either.

private void CreateAircraft(IEnumerable<Flight> flights)
{
  foreach (var flight in flights)
  {
    var aircraft = Instantiate(Aircraft);
    aircraft.transform.parent = _topLevelObject.transform;
    aircraft.transform.localScale = new Vector3(0f, 0f, 0f);
    SetNewFlightData(aircraft, flight);
    _aircrafts.Add(flight.Id, aircraft);
  }
}

private void UpdateAircraft(IEnumerable flights)
{
  foreach (var flight in flights)
  {
    if (_aircrafts.ContainsKey(flight.Id))
    {
      var aircraft = _aircrafts[flight.Id];
      SetNewFlightData(aircraft, flight);
    }
  }
}

private void DeleteAircraft(IEnumerable<string> keys)
{
  foreach (var key in keys)
  {
    var aircraft = _aircrafts[key];
    Destroy(aircraft);
    _aircrafts.Remove(key);
  }
}

The CreateAircraft is the most interesting – it instantiates the aircraft game object, makes the HoloGramCollection it’s parent, passes the actual flight data to the object, and adds it to the dictionary of aircraft game objects using the flight id as the key. Oh, and it also sets the scale to 0, making the plane effectively invisible. This is because, without setting a location on instantiation, the aircraft will appear on 0,0,0 before – where in future episodes we will place the center of Schiphol, very near the tower, and that’s not a place where aircraft belong.

UpdateAircraft just sends flight data to the game object, and DeleteAircraft – well, deletes it. There is only this matter of SetNewFlightData:

private void SetNewFlightData(GameObject aircraft, Flight flight)
{
  var controller = aircraft.GetComponent<AircraftController>();
  if(controller != null)
  {
    //controller.SetNewFlightData(flight);
  }
}

But that has the actual method that moves the flight data to the AircraftController commented out, because that does not even exist. What is worse, the Aircraft does not even have to component. So let’s head over back to the Unity Editor.

Wiring up some stuff in Unity

First of all, we drag the DataService and AircraftLoader Script from the App/Scripts folder on top of the HologramCollection’s inspector page. Then we drag AircraftHolder prefab from App/Prefabs on top of the AircraftLoader’s “Aircraft” property. Net result should be this:

Please make sure the Data Url property of the Data Service indeed points to the place where your data service as created in the first post is published.

Then go to the App/Prefabs folder, open the AircraftHolder prefab itself, and drag the (still default) AircraftController on top of the AircraftHolder’s inspector page.

Save the scene, then rebuild the app with File/Build settings etc. Go back to Visual Studio once the building is finished.

Creating the AircraftController

Having programmed OO since the start of this century, I tend to put control where I logically think it belongs, so rather than programming the ‘flight logic’ into a class that loads and translates data too (the AircraftLoader) I opted for putting it into a separate class that would be part of the game object. I tend to conceptualize these combined game objects and components as OO objects - although that is not completely right, it helps me think about it. The start is simple enough

public class AircraftController : MonoBehaviour
{
  private Flight _flightData;
  private float? _speed;
  private float? _heading;
  private TextMesh _text;
  private bool _initComplete;
  private bool _firstMove;

  void Start()
  {
    _text = transform.GetComponentInChildren<TextMesh>();
    _initComplete = true;
  }
}

A few private fields to retain some data.

• _flightData just keeps the last provide flight data available
• _speed and _heading keep the last speed and heading. The stream of data sometimes misses a beat and provides no speed and/or heading – I prefer then to display the latest data, in stead of heaving the aircraft suddenly rotate to the North (heading 0) and back again when the next set of data is correct again
• _text keeps a reference to the text mesh so I don’t have to look it up every update - potentially 60 times a second
• the _initComplete boolean is a trick I use regularly to prevent other routines using variables like _text before the are initialized. Remember, the Update loop is called independently of what you do. It may look a bit overdone now with only one initialization statement, but believe me – there will be more.

Well then, finally the infamous SetNewFlightData method

public void SetNewFlightData(Flight newFlightData)
{
  if(_initComplete)
  {
    var move = _flightData == null ||
               !_flightData.Location.LocationEquals(
                   newFlightData.Location);
    _flightData = newFlightData;

    ExtractSpeedAndHeading();
    if (move)
    {
      SetLocationOrientation();
    }
    else
    {
      SetNewFlightText();
    }
  }
}

I like to write the code at high level as almost self-explanatory. First we determine if the aircraft needs to be moved, then we ingest the data, and extract speed and heading. Then, when the aircraft needs to be moved, change it’s location, if not, just update the label. The ExtractSpeedAndHeading is pretty straightforward. There is only the thing with heading - that sometimes comes in a negative value, and although Unity3D has no problem with that in positioning the aircraft, I think it looks ugly in the label. So I make sure it's always positive.

private void ExtractSpeedAndHeading()
{
  if (_flightData.Heading != null)
  {
    _heading = (float)_flightData.Heading;
  }
  if (_heading < 0)
  {
    _heading += 360;
  }
  if (_flightData.Speed != null)
  {
    _speed = (float)_flightData.Speed;
  }
}

SetFlightText is also rather trivial

private void SetNewFlightText()
{
  var speedText = 
    _speed != null ? string.Format("{0}km/h", _speed) : string.Empty;

    var headingText =
      _heading != null ? string.Format("{0}⁰", _heading) : string.Empty;

    var text = string.Format("{0} {1} {2}m {3} {4}", _flightData.FlightNr,
      _flightData.Aircraft, _flightData.Location.Alt, speedText, 
      headingText).Trim();
  _text.text = text;
}

Just some clever formatting to prevent empty postfixes like km/h and degrees in the label. By the way – stick to ye olde string.Format and don’t be tempted to use C# 6 string interpolation or you will be sorry (unless you put it between “#if UNITY_UWP – #endif). Anyway, on to the next routine, that actually does all the aircraft manipulation:

private void SetLocationOrientation()
{
  SetNewFlightText();

  transform.localPosition = GetFlightLocation();
  if (_flightData.Heading != null)
  {
    transform.localEulerAngles = GetNewRotation();
  }
  if (!_firstMove)
  {
    transform.localScale = new Vector3(0.0015f, 0.0015f, 0.0015f);
    _firstMove = true;
  }
}

It sets the text too, then set’s the location based on the flight’s location, and set rotation based upon the heading of the aircraft and whether it’s going up or down. Notice I use local position, heading, and scale. This means all those things are relative to the position, rotation and scale of the containing GameObject – HologramCollection. This has the advantage that I can move, rotate and scale the containing object and in one go everything that is in it follows suit. So you don’t have to do all calculations for that – Unity takes care of that form me. I don’t use it in this app just yet, but the actual app already has some experimental code to do that (although that code is not yet in the store version).

Also, note the fact the airplane gets it’s size here (when it’s created it’s 0,0,0). I have the feeling the model is actually a 1:1 scale representation of the actual aircraft – the first time I saw it with the HoloLens I could not find it at first, then turned around and had a “Glimly Glider experience” - a giant aircraft silently swooping down on me from what looked only 10-15 meters. I decide to scale it down to 0.0015 of it’s original size so it appears to be about 10-15cm in a HoloLens – a size more beneficial for getting an good overview, not to mention the blood pressure and heart rate of the average user ;).

Next up are these two routines:

private Vector3 GetFlightLocation()
{
  return GetLocalCoordinates(_flightData.Location);
}

private Vector3 GetLocalCoordinates(Coordinate c)
{
  return new Vector3((float)c.X / 15000,
    c.Alt != null ? (float)c.Alt / 2000.0f : 0f,
    (float)c.Y / 15000);
}

I already discussed the how and why of scaling the down coordinates 15000 times in horizontal direction and 2000 times in vertical direction in a recent blog post about converting lat/lon/alt coordinates into the Unity3D X/Y/Z system, so I am not going through that again, because the next part is a lot more interesting. First I will show the last method, GetVerticalAngle

private float GetVerticalAngle()
{
  var tracksize = _flightData.Track.Count;
  if (tracksize > 2)
  {
    var pLast = _flightData.Track[tracksize - 1];
    var pSecondLast = _flightData.Track[tracksize - 3];
    var delta = pLast.Alt - pSecondLast.Alt;
    if (Math.Abs(delta.Value) > 2.5f)
    {
      return delta < 0 ? 10 : -20;
    }
  }
  return 0;
}

I found that the data, although it provides information about whether the aircraft is actually ascending or descending, that data is not always correct. So I decided to calculate that myself, based upon the difference between the current location and an older location. Now since an aircraft usually descends a lot slower than it takes off (which is very fortunate for the passenger’s – or at least my – peace of mind) this method basically returns –20 when the aircraft is going up, and 10 when it’s going down. And then we get some beautiful Unity3D math again – or more accurately, methods that prevent you from having to use all kinds of advanced 3D math:

private Vector3 GetNewRotation()
{
  var heading = _heading ?? 0;
  var rotation = Quaternion.AngleAxis(heading, Vector3.up).eulerAngles +
      Quaternion.AngleAxis(GetVerticalAngle(), Vector3.right).eulerAngles;
  return rotation;
}

Try to picture in your mind how this works:

• For the heading we have to rotate around the axis that is going up (and down, too) from the center of the aircraft – this what they call yaw in aviation
• For the vertical angle – that makes it look whether the aircraft is going up or down – we have to rotate around the axis that goes to the right (and left – so basically over the wings) from the center of the aircraft. In aviation, this is called pitch 

You use summarize Quaternion.AngleAxis(angle, Vector3.<the axis you desire>).eulerAngles over multiple axes to get the combined rotation of an object and assign that in one go. Hence you see in SetLocationOrientation() the statement "transform.localEulerAngles = GetNewRotation();"

The final things

Go to AircraftLoader 's SetNewFlightData and uncomment the line

 //controller.SetNewFlightData(flight);   

Because we have implemented that in the previous section. Then there’s is the method LocationEquals in Coordinate, that we used in AircraftController but that was not implemented yet ;)

public bool LocationEquals(Coordinate other)
{
  if (other == null) return false;
  return (X == other.X && Y == other.Y && Alt != null &&
          other.Alt != null &&
          Alt.Value == other.Alt.Value);
}

And if you run this in a HoloLens or the Emulator, you will see aircraft!

You will notice they won't move through the air but jump from position to position. They also don't show their track, there is no Schiphol Airport map, no ATC tower, no church, no gaze cursor - and you cannot select anything yet - but that is because this post is long enough as it is. The base is here. 3D visualization of a JSON stream. What is left, is basically making things more slick :)

Conclusion

In hindsight I might better have splitted this episode in two blog post still, but I hope you have made it to the end. I feel this blog post is the heart of the matter – reading a data stream and turning it into a 3D model – making ‘dry records’ come to life, almost literally. I also showed you some key concepts about positioning and rotating stuff in 3D.

As usual, the code is here on GitHub.