A HoloLens airplane tracker 7–activating an aircraft by air tapping

Intro

In this episode I am going to describe how the airplanes are selected in the sky by air tapping them. This requires some changes made to both the Azure Mobile App service and the app. The change to the app service is necessary as I want to share the selected airplane between all possible HoloLenses using the app. Why? Because ;). You actually can do this with the app in the store. This is why sometimes airplanes get selected even when you don’t select them – that is because someone else, somewhere in the world, selects an airplane.

Adapting the service

We need to be able to post to the service which aircraft is active, and in the data that is pulled from the server the active airplane needs to be indicated. Remember in the very first episode the “Flight” object had an unused “IsActive” field? It won’t be unused very much longer. First, you add a reference to System.Runtime.Caching

Then, add using statements:

using System.Runtime.Caching;
using System.Net.Http;
using System.Net;

Proceed by adding the following two fields:

private ObjectCache Cache = MemoryCache.Default;

private const string ActiveIdKey = "ActiveId";

We will need to add a method to post the desired active plane’s Id to the server:

[HttpPost]
public HttpResponseMessage Post(string activeId)
{
  Cache[ActiveIdKey] = activeId;
  return Request.CreateResponse(HttpStatusCode.OK);
}

Then, in the Get method, insert right before the “return flights” statement the following code:

var activeKey = Cache[ActiveIdKey] as string;
if (activeKey != null)
{
  var activeFlight = flights.FirstOrDefault(p => p.Id == activeKey);
  if (activeFlight != null)
  {
    activeFlight.IsActive = true;
  }
}

This is a rather naive way of ‘storing’ data on a server but I am just do darn lazy to set up a database if I only want to have one simple id kept active. So I am using the MemoryCache. If you post an id to the Post method, it will simply set the cache key – which will be copied into the data in the next data download, provided it’s actually an id used in the data. Publish your service, and let’s go the Unity project again

Adding a gaze cursor – and messing a bit with it

The HoloToolkit has a number of gaze cursors, but I am not quite happy with those, so I made one for myself – by adding a bit into an existing one. The procedure is as follows:

• From HoloToolkit/Input/scripts, drag Handsmanager on top of the HologramCollection
• From HoloToolkit/Input/scripts, drag Gazemanager on top of the HologramCollection
• From HoloToolkit/Input/Prefabs, drag Cursor on top of the HologramCollection

Then we create a little extra behavior, called HandColorCursor

using UnityEngine;
using HoloToolkit.Unity;

public class HandColorCursor : Singleton<HandColorCursor>
{
  // Use this for initialization
  private GameObject _cursorOnHolograms;

  public Color HandDetectedColor;

  private Color _originalColor = Color.green;

  void Start()
  {
    _cursorOnHolograms = CursorManager.Instance.CursorOnHolograms;
    _originalColor = 
      _cursorOnHolograms.GetComponent<MeshRenderer>().material.color;
  }

  // Update is called once per frame
  void LateUpdate()
  {
    if (_cursorOnHolograms != null && HandsManager.Instance != null)
    {
      _cursorOnHolograms.GetComponent<MeshRenderer>().material.color =
        HandsManager.Instance.HandDetected ? Color.green : _originalColor;
    }
  }
}

Drag this behavior on top of the Cursor prefab. Now if you build and run the app and point the gaze cursor to the airport it should become a kind of blue circle. If you move your hand into view, the cursor should appear and go green.

If you move it back, the cursor should again become blue. Unfortunately, if you point it at anything else, you will still see the fuzzy ‘CursorOffHologram’ image. That it’s because the airport is the only thing that has a collider. To make aircraft selectable, we will need to do some work at AircraftHolder.

Updating the airplane

To be selectable, it needs to have a collider. I have chosen a box collider that is pretty big with respect to the actual airplane model – to make it easier to actually select the airplane (as it can be quite some distance away, or at least seem to be) and a box makes it possible to actually see the cursor sitting pretty stable on the selection area as it hits the collider (as opposed to a mesh collider that will make it wobble around). Go to your AircraftHolder prefab in the App/Prefab folder, and add an 80x80x80 Box Collider

And you will indeed see the cursor appear over the airplane

Getting some selecting done – code first

From HoloToolkit/Input/scripts, drag GestureManage on top of the HologramCollection. GestureManager sends an OnSelect message to any GameObject that happens to be hit by a gaze (and where, incidentally, the cursor is hanging out out) using this code:

private void OnTap()
{
  if (FocusedObject != null)
  {
    FocusedObject.SendMessage("OnSelect");
  }
}

So there needs to be an OnSelect method in the AircraftController to handle this. First we will need to add the following fields to the AircraftController:

private bool _isActive;
private AudioSource _sound;
private Dictionary<MeshRenderer, Color> _originalColors;

To the Start method, we add the following code, just in front of the “_initComplete = true;” statement

_sound = GetComponent<AudioSource>();
_originalColors = new Dictionary<MeshRenderer, Color>();
foreach (var component in GetComponentsInChildren<MeshRenderer>())
{
  _originalColors.Add(component, component.material.color);
}

So this actually initializes some things by getting the audio source and reads the color of all sub components – so we can revert it easily later. Then add this method:

private void SetHighlight()
{
  if (_isActive != _flightData.IsActive)
  {
    foreach (var component in GetComponentsInChildren<MeshRenderer>())
    {
      if (component.material.color == Color.white || 
          component.material.color == Color.red)
      {
        component.material.color = _flightData.IsActive ?
           Color.red : _originalColors[component];
      }
    }
    if (_flightData.IsActive)
    {
      _sound.Play();
    }
    else
    {
      _sound.Stop();
    }
    _isActive = _flightData.IsActive;
  }
}

If the _isActive field of the current airplane differs from the new _flightData.IsActive, then flip white things to red – or red things back to white, depending on the actual value of _flightData.IsActive. And it turns on the sound – the annoying sonar ping you saw in the movie I made in the first episode. And fortunately it can also turn in it off ;)

Next, add a call to this new method at the end of the already existing “SetNewFlightData” method. Then, add this code:

private void OnSelect()
{
  SendMessageUpwards("AircraftSelected", _flightData.Id);
}

This is like SendMessage, but then upwards through the object hierarchy, in stead of down to the children. So this will look for an AircraftSelected method in the parent.

Save the file. Now we go to the DataService, and add the following method. Make sure it is sitting between #if UNITY_UWP – #endif directives.

#if UNITY_UWP
public async Task SelectFlight(string flightId)
{
  var parms = new Dictionary<string, string>
  {
    { "activeId", flightId }
  };
  await _client.InvokeApiAsync<List<Flight>>("FlightData", 
           HttpMethod.Post, parms);
}
#endif

This does the actual calling of the App service. And then we need to go to AircraftLoader, which plays the select sound and passes the call trough to the DataService:

public void AircraftSelected(object id)
{
  _sound.Play();
#if UNITY_UWP
  DataService.Instance.SelectFlight(id.ToString());
#endif
}

Save everything, go back to Unity.

Getting some selecting done – connecting the dots in Unity

There are a few things we need to be doing in Unity, but one thing is very important. One I forgot in my first CubeBouncer demo app and in the app that’s in the store, simply because I was not aware of it. Fortunately someone who shall remain nameless here made me aware of it.

First, in the main Unity window, hit Edit/Project Settings/Audio. Then check “Ms HRTF Spatialization” for “Spatializer plugin”

Then open the AircraftHolder prefab again, and add an AudioSource. Important advice, also from the person mentioned above: hit the “Spatialize” checkbox. Then select the “Loop” checkbox and uncheck the “Play On Awake” checkbox. Finally, find a sound you would like to be played by a selected airplane. I took a sonar ping. I put it in the Assets/App/Audio folder, and dragged it on the Audio Clip property. I also moved the “Spatial Blend” slider all the way to the right but I don’t think that is necessary anymore now I use the Spatializer. Net result:

And sure enough, if you now air tap an airplane it shows like below, and it starts pinging:

Adding confirmation sound to selection

I think it’s good practice to add a confirmation sound of some sorts to any user input that is ‘understood’. This is especially necessary with speech commands, but I like to add it to air taps. Sometimes the effect of what you do is not immediately visible, and an audio confirmation the HoloLens has understood you and is working on making your command getting executed is very nice. I tend to add a kind of “pringgg” sound. I used the “Ready” file that is in Assets/App/Audio.

First, add an AudioSource to the HologramCollection and drag the “Ready” sound on top of the “AudioClip” field. Don’t forget to uncheck the “Play On Awake” checkbox.Then go to the AircraftLoader behaviour. Since this receives the message from every AircraftController and passes it through to the DataService, this is a nice central place to play the confirmation sound. So we add a new field:

private AudioSource _sound;

that needs to be initialized in the Start method:

_sound = GetComponent<AudioSource>();

and finally in the AircraftSelected method, insert this line at the top of the method:

_sound.Play();

And we are done. Now if you airtap on the aircraft your HoloLens will sound a ‘pringgg’ sound. That is, if you tapped correctly.

Conclusion

And that’s that. Selecting an airplane by by air tapping it it, and posting an id to the Azure Mobile App service is pretty easy indeed. A lot of the interaction components are basically dragged in from the HoloToolkit, very little actual programming is required. This is the power of the HoloToolkit – I use as much from what is already there, and make as little as possible myself. As any good programmer I am a lazy programmer ;)

This blog post written far, far away from home, in Sammamish, WA, USA, shortly after the MVP summit that left me very inspired – again. After posting this I will soon retire – and go to SEA-TAC tomorrow for my flight back home in the Netherlands.

Code can be found, as always, here on github.