HoloLens CubeBouncer application part 2-create the gaze aligned cube grid

Preface

In the previous post, I have showed you how to setup a very basic HoloLens application that only shows one floating cube – exactly on the spot where your head is when you start the app. You actually had to get up and look backwards to where your head was to see it. In this post I will show you how to dynamically create a grid of cubes, aligned with your head, based upon the available space. This space is defined by the distance HoloLens measures between you and where your gaze hits a physical object. Or, failing to find such a object (because you are staring in empty space or an object too far away) it will make a small ‘fixed’ grid.

But first some spatial mapping

In order to find a wall, we first have to instruct HoloLens to actually start looking for it. It’s not turned on by default in an app – it’s a resource intensive process and it would not make sense to turn in on in an app that does not use it. You might have noticed that the gazing cursor, which we added in the previous post, actually never appeared unless you looked directly at the lonely cube. Now we will make it appear when you look at a wall or object. Turning spatial mapping on is actually very easy:

 

First, type “spatial” in the search box. This will search through all your asset folders. Then, locate the SpatialMapping prefab. In Unity3D, a prefab is a reusable set of components that can be used over and over again, without having to build it up from it’s parts and set all the properties every time you want use it. If you did follow my advice and had a look at Rick Barraza’s awesome Creative Coding series you will have had the concept explained.

Hit File/Build/Build settings to re-create the Visual Studio Solution, the deploy the app using Visual Studio. First, you won’t see much difference. But then suddenly the “Spatial Mesh” will appear, and you will the ‘gaze cursor’ following your gaze where ever it hits a wall. (indicated by a red arrow I added later to make it easier to find on the photo).

You can decide to leave it on while developing – some people find it fascinating to see HoloLens actually doing it’s magic trick. I think it’s a great tool to check if the app is actually mapping, but I also have found it to be visually distracting after a while. So after I have ascertained my app is doing it’s spatial mapping, I change it’s ‘Surface Material’ property to ‘Occusion’:

Create a prefab from our cube

In order to be able to dynamically create an object, we need to turn it into a prefab as well. That is actually very easy. Just grab your Cube and drag it into your Assets/Custom/Prefabs folder

And give it a name. I called it “WortellCube”. Then proceed to delete the thing that is now called “WortellCube from the HologramCollection. We won’t need the static cube anymore. And now, my friends, halfway the 2nd episode on this hard core coder’s blog, we are finally nearing the place where actual coding takes place. Just one more step.

Creating a ‘script’

Any code in Unity3D is called a “script”. It supports a lot of languages and one of them is, fortunately, C#. There are all kinds of ways to do this, but I find the easiest one the following:

• Open the Assets/Custom/Scripts folder
• Right-click it, hit “Create”, then “C# script”
• Name it “MainStarter”
• Select “HologramCollection”
• Drag the MainStarter script on top of the Inspector pane, below the “Add Component” button.

Net result should be this

There are at least two other ways that I know of to to create a C# script – this in my work flow.

Hit File/Build settings/Build. Open the resulting project in Visual Studio (or reload the project). Assuming you indeed selected the “Unity C# project” option as I advised in my previous post, the solution will look like this:

And there’s our Script. Now a proper C# class in a Visual Studio Solution.

Adding a game object

All right. So the ‘script’, a C# class, looks initially like this

using UnityEngine;
using System.Collections;

public class MainStarter : MonoBehaviour
{

    // Use this for initialization
    void Start()
    {

    }

    // Update is called once per frame
    void Update()
    {

    }
}

Now imagine, just for a second, this here veteran XAML coder and behavior nut when he saw this. A Unity script is actually called a behaviour. Granted – it’s spelled in The Queen’s English, but it really made my day. Far from ‘home’, still writing things called ‘behaviour’.  Apart from the name, that’s about where the similarities end. 

To get started, we first add a public field to the top of the class:

public class MainStarter : MonoBehaviour
{
    public GameObject Cube;

    // Use this for initialization
    void Start()
    {

    }

Yes. A public field. If you want to do Unity3D, get ready to meet some things that might hurt your coder’s feelings, and do some things you have learned to avoid like the plague.

Hit Save, go back to the Unity Editor, and after a few seconds tops you will see this (assuming you still have the HologramCollection object selected:

The Cube fields shows up. Now that’s the GameObject we can instantiate. So from our Assets/Prefabs folder we drag the WortellCube on top of the field that now says “None”:

And this should be the result. Hit File/Build Settings/Build, reload the project in Visual Studio, and now it’s time for real coding.

Building the initial script

As the comment that’s added by default added shows, there are some methods that are called on events that happen in the Unity object lifecycle. Note that those methods are private, but apparently they can be called anyway. A proper C# way would be to make a base class with overrideable public methods but hey, like I said, things are a bit different here. There is a whole host of private methods that can be called on an equal host of events, but that’s for later. Let’s first fill the Start method:

private bool _distanceMeasured;
private DateTimeOffset _lastInitTime;

// Use this for initialization
void Start()
{
  _distanceMeasured = false;
  _lastInitTime = DateTimeOffset.Now;
}

and then the Update method:

void Update()
{
  if (!_distanceMeasured)
  {
    if (GazeManager.Instance.Hit)
    {
      _distanceMeasured = true;
      CreateGrid(GazeManager.Instance.Position);
    }
    else
    {
      // If we can't find a wall in 10 seconds, create a default grid 
      if((_lastInitTime - DateTimeOffset.Now).Duration() >
           TimeSpan.FromSeconds(10))
      {
        _distanceMeasured = true;
        CreateGrid(CalculatePositionDeadAhead());
      }
    }
  }
}

So this method is called, every ‘frame’. This is typically 60 times per seconds, as far as I understand it. The GazeManager is the object that tracks your gaze. If it hits something within 10 seconds, it sends the position of the hit to the method that creates the grid (CreateGrid). If not, if calculates a location 3.5 meters in front of you, following the direction of your gaze:

private Vector3 CalculatePositionDeadAhead()
{
  var gazeOrigin = Camera.main.transform.position;
  return gazeOrigin + Camera.main.transform.forward * 3.5f;
}

The position Camera.main.transform.position – that is where you are. And there is a transform.forward thingy. Now this was a point where I first met a TransForm, and that quite puzzled me.

Aside: Transform

At this point I don’t quite understand all the finesses of a TransForm, but one thing is key. A TransForm has amongst other, three very important properties. “forward”, “right” and “up”. What helped me understand was thinking of this image: imagine you are an astronaut floating in space next to Earth. Your nose is pointing to Earth. Which way is forward? Well, Earth of course. Which way is up? Whatever direction the top of your head is pointing. And right? Well, to your right of course. Next, you fire the jets of your MMV and it rotates you – so now your back is pointing to Earth and your nose is pointing towards the International Space Station, that was originally behind you. The cardinal question is now – which way is forward? Exactly. Although everything else stayed in place, forward is now IIS.

This is how you can use a TransForm. However an object is rotated in space, if you add 2 times it’s transform.forward to it’s position, it will move 2 meters forward – that is, the direction that is forward from it’s own perspective. If you rotate it in another reaction, it will go in the direction it then points to. If you want it to move backwards, simply subtract it’s transform.forward. Same goes for the rest. Want to move 2 meters up? Add 2 times transform.up. Want to move down? Subtract it. Want to go right? Add transform.right. Go left? Subtract the same. Unity will take care of in what absolute direction the object actually needs to be moved in.

So what the second line of CalculatePositionDeadAhead simply does it add 3.5 meters to your position in front of you, exactly in the direction where you are looking when the grid is created. This is a quite powerful concept. You don’t have to care how stuff is rotated. Transform properties will tell you which way to go.

Calculating the grid

So now that I have explained this, I can show you how to how the grid is calculated:

private void CreateGrid(Vector3 hitPosition)
{
  var gazeOrigin = Camera.main.transform.position;
  var rotation = Camera.main.transform.rotation;

  var maxDistance = Vector3.Distance(gazeOrigin, hitPosition);

  transform.position = hitPosition;
  transform.rotation = rotation;

  int id = 0;

  float size = 0.2f;
  float maxZ = maxDistance - 1f;
  float maxX = 0.35f;
  float maxY = 0.35f;
  float z = 1.5f;
  do
  {
    var x = -maxX;
    do
    {
      var y = -maxY;
      do
      {
        CreateCube(id++,
            gazeOrigin + transform.forward * z +
                         transform.right * x +
                         transform.up * y,
            rotation);
        y += size;
      }
      while (y <= maxY);
      x += size;
    }
    while (x <= maxX);
    z += size;
  }
  while (z <= maxZ);
}

Using the camera’s rotation and the hitPosition I calculate a location about 1 meter from the wall (or the calculated point). The grid will start 1.5 meters in front of you. So the space between 1.5 meters from you and 1 meter from the wall is filled with a 4 x 4 grid, using three loops (one for every dimension). I seem not te be able to new up a TransForm, but every behaviour has a transform property – and since this script is not part of a physically displayed GameObject I can use it for calculations. I do this to take a snapshot of the camera rotation, and prevent slight differences in rotation because the user does not keep his or her head still.The actual TransForm position does not seem to matter much in calculations – you can also take the gazeOrgin in stead of hitPosition. The vector containing forward, up and right is always normalized – values are always 1 or smaller and can therefore be used for this kind of tricks.

Oh, and finally there is this very simple method to actually create the cube:

private void CreateCube(int id, Vector3 location, Quaternion rotation)
{
  var c = Instantiate(Cube, location, rotation) as GameObject;
}

So why is this a method at all, does it have an id property that is not used at all? Because there is more to come, and we will need other parts in a future post.

If you run the project now, after a maximum of 10 seconds, you will see something like this:

That's awesome, right? Except - why are the cubes upside down? In Unity it was displayed correctly!

Once again – think TransForm. If you walk around the cubes, you will see they are actually not upside down. Because we aligned them to the gaze – rotation and all - we are looking at their backside. So what we need to do is – rotate the cube 180 degrees around the axis that goes from top to bottom – in other words, the “up” axis – on it’s own location.

private void CreateCube(int id, Vector3 location, Quaternion rotation)
{
  var c = Instantiate(Cube, location, rotation) as GameObject;
  //Rotate around it's own up axis so up points TO the camera
  c.transform.RotateAround(location, transform.up, 180f);
}

and here we go. One grid of boxes, perfectly aligned to your view angle (notice I have tilted my head to the right now)

Concluding remarks

I have shown you how to add and use Spatial Mapping, create a prefab from your cube, and how you can dynamically instantiate those prefabs. I also showed you how to align the grid with your view angle, using the GazeManager, that I also used to measure distance. Finally, I have showed you how to employ TransForm for calculating relative directions.

As usual, you can find a full working solution of the project so far here.