HoloLens CubeBouncer application part 4-integrating speech commands and moving cubes by code

Preface

In the previous post I showed how you could interact with the cubes using air taps, utilizing the physics engine. In this blog post I am going to show how to move the cubes by code (bypassing the physics engine) – and doing so using speech commands.

But first…

The funny thing is - when retracing you steps and documenting them, you are found out things can be done in a smarter way. It’s like giving a code review to your slightly younger, slightly less knowledgeable self. In post 2, I state you should drag the MainStarter script onto the HologramCollection game object. Although that works, in hindsight it’s better to put that under the HologramCollection/Managers object. So please remove the script from the HologramCollection object, drag it anew from your assets on top of the Managers object, then drag the WortellCube prefab on top of the Cube field again.

HoloLens speech recognition 101

The speech recognition API for HoloLens in Unity3D is so simple that there’s basically not much else than 101. I had a look at Rene Schulte’s HoloWorld speech manager and was like… is that all? Well, apparently it is. So I created my own SpeechManager script, and added it to the Managers object. It’s basically a modified copy of Rene’s. Why re-invent the wheel when people smarter than yourself already have done so, right?

The speech manager at this point implements only two commands:

• “create new grid”
• “go to start”

and looks like this:

using UnityEngine;
using HoloToolkit.Unity;
using UnityEngine.Windows.Speech;

public class SpeechManager : MonoBehaviour
{
  public string GoToStartCommand = "go to start";

  public string NewGridCommand = "create new grid";

  private KeywordRecognizer _keywordRecognizer;

  private MainStarter _mainStarter;

  // Use this for initialization
  void Start()
  {
    _mainStarter = GetComponent<MainStarter>();
    _keywordRecognizer = new KeywordRecognizer(
      new[] { GoToStartCommand, NewGridCommand });
    _keywordRecognizer.OnPhraseRecognized += KeywordRecognizer_OnPhraseRecognized;
    _keywordRecognizer.Start();
  }

  private void KeywordRecognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
  {
    var cmd = args.text;

    if (cmd == NewGridCommand)
    {
      _mainStarter.CreateNewGrid();
    }

    if (cmd == GoToStartCommand)
    {
      if (GazeManager.Instance.Hit)
      {
        GazeManager.Instance.HitInfo.collider.gameObject.SendMessage(
          "OnRevert");
      }
    }
  }

  private void OnDestroy()
  {
    if (_keywordRecognizer != null)
    {
      if (_keywordRecognizer.IsRunning)
      {
        _keywordRecognizer.Stop();
      }
      _keywordRecognizer.Dispose();
    }
  }
}

In short – when the keywords “go to start” are recognized, “OnRevert” is called on the game object that you are looking at. We have seen this kind of message sending in the previous post already.If you say “create new grid” the CreateNewGrid method is called. This tries to find the MainStarter class as a component (it being at the same level in the Managers game object, it fill find it) and call the method directly. But neither methods are implemented, you will even notice the squiggly lines under CreateNewGrid. So let’s tackle that first, because now our project does not even compile.

(Re)creating a grid.

Creating a new grid is, simply put, deleting the old cubes are creating a new set. This implies that we must know which cubes are present now, something we don’t know now. This actually requires very little extra code:

private readonly List<GameObject> _cubes = new List<GameObject>();

public void CreateNewGrid()
{
  foreach (var c in _cubes)
  {
      Destroy(c);
  }
  _cubes.Clear();

  _distanceMeasured = false;
  _lastInitTime = DateTimeOffset.Now;
}

Very simple – all cubes created are stored in a list. So we need to destroy them one by one, and then we set the _distanceMeasured and _lastInitTime back to their start value – and the Update method that is called once per frame will do the rest. We also have to add one extra line to the CreateCube method in order to collect the created cube, at the end:

private void CreateCube(int id, Vector3 location, Quaternion rotation)
{
  var c = Instantiate(Cube, location, rotation) as GameObject;
  //Rotate around it's own up axis so up points TO the camera
  c.transform.RotateAround(location, transform.up, 180f);
  var m = c.GetComponent<CubeManipulator>();
  m.Id = id;
  _cubes.Add(c);
}

So now the only thing you need to do is drag the SpeechManager script on top of the Managers object too. When you are done, the Managers object should look like you see at the right.

Rebuild the UWP app from Unity, and deploy it using Visual Studio. And there we are. When you said “create new grid” the grid is immediately updated. Still without the ringing sound that shows in my initial video, but we will add that in a later stage.

Sending a cube back to where it came from

As we saw, we already implemented a call to an OnRevert method in the Speech Manager to recall the cube we are looking at to its original position, but we have not implemented it. To this intent, a cube needs to know where it came from and how it was rotated when it was created. So we add the following private fields to the cube:

private Vector3 _orginalPosition;

private Quaternion _originalRotation;

And we set those fields in Start by reading the information from the transform:

void Start()
{
  _rigidBody = GetComponent<Rigidbody>();
  _audioSource = GetComponent<AudioSource>();

  _orginalPosition = transform.position;
  _originalRotation = transform.rotation;
}

So now we need to implement the “OnRevert” method itself like this:

public IEnumerator OnRevert()
{
   var recallPosition = transform.position;
   var recallRotation = transform.rotation;

  _rigidBody.isKinematic = true;

  while (transform.position != _orginalPosition &&
         transform.rotation != _originalRotation)
  {
    yield return StartCoroutine(
      MoveObject(transform, recallPosition, _orginalPosition,
                 recallRotation, _originalRotation, 1.0f));
  }

  _rigidBody.isKinematic = false;
}

So first we we make sure we retain the current position and rotation. Then we set the isKinematic property of the rigid body to true. This effectively turns off the physics engine, so now we can move the cube ourselves. And then we loop over a so-called coroutine until the cube is back to it’s original position and rotation. I think of it as Unity’s equivalent of an async operation. Basically it says – animate the current transform smoothly from the current position to the original position, and at the same time rotate it from it’s current rotation to the original rotation, in 1.0 second. The coroutine itself is implemented like this:

// See http://answers.unity3d.com/questions/711309/movement-script-2.html
IEnumerator MoveObject(Transform thisTransform, Vector3 startPos, Vector3 endPos, 
  Quaternion startRot, Quaternion endRot, float time)
{
  var i = 0.0f;
  var rate = 1.0f / time;
  while (i < 1.0f)
  {
      i += Time.deltaTime * rate;
      thisTransform.position = Vector3.Lerp(startPos, endPos, Mathf.SmoothStep(0f, 1f, i));
      thisTransform.rotation = Quaternion.Lerp(startRot, endRot, Mathf.SmoothStep(0f, 1f, i));
      yield return null;
  }
}

How this works, is – as you can read in the source – is explained here. That sample only applies to moving an object - I have applied that knowledge to both moving and rotating. It basically is a way to smooth out the animation – if you  pay close attention, you will see that the cube starts slows, speeds up very fast, and then slows down again. I must admit that I am quite missing some of the finer points myself still, but this is how it can be done. Important, by the way, is that isKinematic gets set to false again once the cube is back on its place. Fun detail – if a cube that is moving back to it’s start position hits another cube, it is bumped out of the way, because for the cube that is hit, the physics engine is still working ;)

Finally, at the top of the OnCollision method we need to make sure returning objects don’t interfere with other cubes in terms of making sound and other stuff.

void OnCollisionEnter(Collision coll)
{
  // Ignore returning bodies
  if (_rigidBody.isKinematic) return;

And now, if you say “go to start” when you are looking at a specific cube, it will move to the location it came from.

Concluding remarks

In the previous post I have showed you how to move objects using the physics engine - this post has showed you have how to integrate speech commands and move objects via code – remarkably little code, yet again. Once again, you can see the code of the project so far here.