Responding to focus and showing data when tapping a hologram in Mixed Reality/HoloLens apps

Intro

One of the things you tend to forget as you progress into a field of expertise is how hard the first steps were. I feel that responding when gaze strikes a hologram or showing some data when an hologram is air tapped are fairly basic - but judging by the number of questions I get for "how to do this", this is apparently not so straightforward as it seems. So I decided to make a little sample that you can get from GitHub here.

Project setup

I created a new project in Unity. Then:

• I imported the Mixed Reality Toolkit (this version to be specific)
• I configured Project, Scene and app Capabilities with the Mixed Reality Toolkit menu item
• I download a model of a satellite to have something to interact with

Now this model has some less than ideal size (as in being very big) so I fiddled a bit with the settings to get a more or less the view as displayed to the right

Now this model consist out a lot of sub objects, so this is nice model to interact with.

Creating interaction

What a hologram needs for interaction is pretty simple:

• A collider (so the gaze cursor has something to strike against)
• For an air tap to be intercepted: a behaviour that implements the interface IInputClickHandler
• For registering focus (that is, the gaze cursor strikes it): a behaviour that implements IFocusable.

But this satellite has like 41 parts, and if we have to manually add a collider and 1 or two behaviors to that, it's a bit bothersome. That in this case, we can solve that by using a behaviour that sets that up for us. Mind you, that's not always possible. But for this simple sample we can.

The behaviour looks like this:

using UnityEngine;

public class InteractionBuilder : MonoBehaviour
{
    [SerializeField]
    private GameObject _toolTip;
    
    void Start ()
    {
        foreach (var child in GetComponentsInChildren<MeshFilter>())
        {
            child.gameObject.AddComponent<MeshCollider>();
            var displayer = child.gameObject.AddComponent<DataDisplayer>();
            displayer.ToolTip = _toolTip;
        }
    }
}

If simply walks find every MeshFilter child component, and adds an collider and a DataDisplayer to the game object where the MeshFiilter belongs to. The intention is to drag this on the main CommunicationSatellite object. But not right yet - because for this to work, we first need to create DataDisplayer, which is the behaviour implementing IInputClickHandler and IFocusable

Show tooltip on click - implementing IInputClickHandler

The first version of DataDisplayer looks like this:

using HoloToolkit.Unity.InputModule;
using HoloToolkit.UX.ToolTips;
using UnityEngine;

public class DataDisplayer : MonoBehaviour, IInputClickHandler
{
    public GameObject ToolTip;

    private GameObject _createdToolTip;

    public void OnInputClicked(InputClickedEventData eventData)
    {
        if (_createdToolTip == null)
        {
            _createdToolTip = Instantiate(ToolTip);
            var toolTip = _createdToolTip.GetComponent<ToolTip>();
            toolTip.ShowOutline = false;
            toolTip.ShowBackground = true;
            toolTip.ToolTipText = gameObject.name;
            toolTip.transform.position = transform.position + Vector3.up * 0.2f;
            toolTip.transform.parent = transform.parent;
            toolTip.AttachPointPosition = transform.position;
            toolTip.ContentParentTransform.localScale = new Vector3(0.05f, 0.05f, 0.05f);
            var connector = toolTip.GetComponent<ToolTipConnector>();
            connector.Target = _createdToolTip;
        }
        else
        {
            Destroy(_createdToolTip);
            _createdToolTip = null;
        }
    }
}

Now this may look a bit complicated, but most of it I just stole from the ToolTipSpawner class. Basically it only does this:

• When the hologram-part is clicked (and OnInputClicked is called, which is a mandatory method when you implement IInputClickHandler) it checks if a tooltip already exists.
• If not, it creates one a little above the clicked element
• If a tooltip already exists, it's is deleted again.

This behaviour get it's  tooltip prefab handed from the InteractionBuilder. As I said, InteractionBuider should be dragged on the CommunicationSatellite root hologram, and now we have built our DataDisplayer, we can actually do so.

The Tooltip field needs to be filled by dragging the Tooltip prefab from HoloToolkit/Ux/sefabs on top of it

Now, if you tap on an element of the satellite, you will get tooltip data with the name of the element

Now in itself this is of course pretty much useless, but in stead of displaying the name directly you can also use the name or some other attribute of the Hologram part to reach out to a web service or a local data file, using that attribute as a key, fetch some data connected to that attribute and show that data. Is it a fairly commonly used pattern, but is up to you - and outside the scope of this blog - to have some data file or web service with connect.

Highlighting on focus - implementing IFocusable

It's not always easy to see which part of the satellite is hit by the gaze cursor, as they are both quite lightly colored. How about letting the whole part light up in red? We add the following code to DataDisplayer:

public class DataDisplayer : MonoBehaviour, IInputClickHandler, IFocusable
{
    private Dictionary<MeshRenderer, Color[]> _originalColors = 
            new Dictionary<MeshRenderer, Color[]>();

    void Start()
    {
        SaveOriginalColors();
    }
    
    public void OnFocusEnter()
    {
        SetHighlight(true);
    }

    public void OnFocusExit()
    {
        SetHighlight(false);
    }
}

This looks pretty simple

• A start you save a hologram part's original colors
• If the gaze strikes the hologram, set the highlight colors
• If the gaze leaves, turn highlight off

It sometimes helps to make code self-explanatory and to write it like this.

So saving the original colors works like this - and it conveniently makes an inventory of the components inside each hologram-part and the materials they use:

private void SaveOriginalColors()
{
    if (!_originalColors.Any())
    {
        foreach (var component in GetComponentsInChildren<MeshRenderer>())
        {
            var colorList = new List<Color>();

            foreach (var t in component.materials)
            {
                colorList.Add(t.color);
            }
            _originalColors.Add(component, colorList.ToArray());
        }
    }
} 

Creating the highlight is not very complex now anymore

private void SetHighlight(bool status)
{
    var targetColor = Color.red;
    foreach (var component in _originalColors.Keys) 
    {
        for (var i = 0; i < component.materials.Length; i++)
        {
            component.materials[i].color = status ? 
                targetColor : _originalColors[component][i];
        }
    }
}

Basically we set the color of every material inside the the component to red if status is true - or to it's original color when it is false. And now, if the gaze cursor strikes part of our satellite:

Conclusion

And that's all there is to it. As I wrote, I would like to suggest showing else than the name of the hologram as that is not very interesting. Also, a bit more elaborate way of showing the data than using the the Tooltip might be considered. But the principle - implementing IInputClickHandler and IFocusable and acting on that - stays the same.

The finished demo project can be found here.