HoloLens CubeBouncer application part 3-air tapping the cubes away and adding spatial sounds

Preface

In the previous post I showed you how to create a neatly arranged grid aligned to your view in your HoloLens app. Now, it’s time to make a mess of it – I am going to show you how to employ air tap on the cubes to move the them around, and gaze to determine which way they go. Plus, we are going to add some sound to them – when they bounce against each other, and against the wall.

Steal some sounds

First of all, we need two short sound clips. One for two cubes hitting each other, one for a cube hitting a wall. Any clip will do, as long as it’s short. I took two from the “Free Casual Sounds SFX Pack” that you can find in the Unity Asset Store (Click Windows/Assets Store or hit CTRL+9)

 

 

Beware: importing directly from the store can get you more than you have bargained for, bloating your project with unnecessary stuff. I tend to create a new Unity3d project and import the package there first, to see what happens. And in this case I just cherry picked two sounds, that I called BounceCubel.wav and BounceBall.wav. Then I dragged into the Assets/Audio folder in Unity, as displayed here to the left.

Now it’s time for coding again. That is to say …

Steal some scripts

Writing code is awesome – not having to write code is even better. In the HoloLens toolkit there’s a script that almost does what we want. It’s called “GestureManager” and it’s in Assets\HoloToolkit\Input\Scripts. It’s a script that handles the air tap and sends a kind of message to a selected object. That is almost what we want. We need to copy and adapt it a little.

The next few steps are best done when Unity3d is not running, as it starts to parse scripts as soon as they appear in the folder:

• Copy Assets\HoloToolkit\Input\Scripts\GestureManager.cs to. Assets\Custom\Scripts
• Rename the file to RaySendingGestureManager.cs
• Open RaySendingGestureManager.cs and change the class name to RaySendingGestureManager
• Change “Singleton<GestureManager>” into “Singleton<RaySendingGestureManager>”
• Remove the namespace namespace HoloToolkit.Unity
• Add “using HoloToolkit.Unity” on top.
• Find the method GestureRecognizer_TappedEvent
• Change it to the following:

private void GestureRecognizer_TappedEvent(
     InteractionSourceKind source, int tapCount, Ray headRay)
{
    if (focusedObject != null)
    {
        focusedObject.SendMessage("OnSelect", headRay);
    }
}

Basically, the only thing you do, is pass on the headRay “Ray” as extra parameter to the focusedObject.SendMessage method. This means that Unity3d should search for a Script component in the selected GameObject, find a method “OnSelect” in there with a a single parameter of type object - and invoke that. This feels a bit Javascripty, only even less typed, and there is no punishment, either: if the method is not found, nothing happens. No error – just nothing. It that sense it’s like sending a message indeed – if no-one is listening, we don’t care. Talk about loose coupling. You can’t get much looser than this ;)

Why do we need to send the headRay? Because we want the selected object to know from which direction it’s being looked at when you tap. If you want to know the details of air tap, have a look at the rest of the script. For now, we should be content that there is a method called on the selected object – i.e. the WortellCube – when there is an air tap while the cursor is on it.

One more thing – we need to apply the script. Select the HologramCollection/Managers object in the Hierarchy pane, then hit the “Add Component” button, Type “Ray Sending” in the search box and add the Ray Sending Manager component. Net result should be this:

Message sent – now catch it!

Okay, so whenever someone air taps while their gaze is at ‘something’, Unity3D tries to call on “OnSelect” method. Let’s make sure that if that something is a WortellCube, it listens to it. Start as follows:

• Select the “Assets/Custom/Scripts folder
• Right click, hit Create/C# script
• Call it “CubeManipulator”
• Select the WortellCube Prefab
• Drag the new script on top Inspector tab, below the “Add Component” button.
• Hit File/Build Settings/Build and build the solution
• Open the Visual Studio solution (or reload it)

Open the CubeBouncer.cs script, and change it so it looks like this:

using HoloToolkit.Unity;
using System.Collections;
using UnityEngine;

public class CubeManipulator : MonoBehaviour
{
    private Rigidbody _rigidBody;

    public int ForceMultiplier = 100;


    // Use this for initialization
    void Start()
    {
        _rigidBody = GetComponent<Rigidbody>();
    }

    public void OnSelect(object ray)
    {
        if (!(ray is Ray)) return;
        var rayData = (Ray)ray;

        _rigidBody.AddForceAtPosition(
            new Vector3(
                rayData.direction.x * ForceMultiplier, 
                rayData.direction.y * ForceMultiplier, 
                rayData.direction.z * ForceMultiplier),
                GazeManager.Instance.Position);
    }
  }
}

Since the script is now part of a compound object, I can easily get a reference to other components in that objects. In this case, we can get a reference to the RigidBody component. For performance reasons (and because I find it more tidy looking), we store that in a private field that is initialized in the Start method.

And then there’s the OnSelect method. After first checking if we indeed get a Ray supplied in the ray parameter, we simply call the RigidBody’s AddForceAtPosition with a Vector3D made out of the direction of the Ray, times 100, and the position where the cursor is now. And that’s it. Unity gives the Cube a push in the direction you are looking on the place where your gaze is locked on the cube. And off it goes, spinning into the void, bouncing off other cubes and walls. Notice the cubes slowly slow down by themselves. That is all caused by the physical characteristics – the bouncyness of the Physic Material, the weight and drag of the Rigid Body. Having written software that calculates bouncing at angles myself, it almost feels like cheating. But this is the awesome power of a fleshed out physics engine.

To see this stage you actually have to rebuild the project from Unity first again or you will get an “[Position out of bounds!]” error when you try to run the app on a HoloLens or the emulator. This is because we have added a public field again, which translates into a property that can be filled by Unity. Every time you add public field to a script you will have to rebuild the project from Unity.

There is a still a thing missing. The cubes bounces off each other in dead silence. Time to fix that, and make the experience more immersive.

Let’s make some noise!

First, we need to add three public and one private field to the CubeManipulator script:

private AudioSource _audioSource;

public int Id;

public AudioClip BounceTogetherClip;

public AudioClip BounceOtherClip;

_audioSource gets initialized in Start as well:

void Start()
{
  _rigidBody = GetComponent<Rigidbody>();
  _audioSource = GetComponent<AudioSource>();
}

And then we have to act on a collision. That is just another simple private method that is automatically called by Unity. And wouldn’t you know it, it’s called “OnCollision” – duh ;). So let’s add that to the script:

void OnCollisionEnter(Collision coll)
{
  // Ignore hits by cursors
  if (coll.gameObject.GetComponent<CursorManager>() != null) return;

  // Play a click on hitting another cube, but only if the it has a higher Id
  // to prevent the same sound being played twice
  var othercube = coll.gameObject.GetComponent<CubeManipulator>();
  if (othercube != null && othercube.Id < Id)
  {
    _audioSource.PlayOneShot(BounceTogetherClip);
  }

  // No cursor, no cube - we hit a wall.
  if (othercube == null)
  {
    if (coll.relativeVelocity.magnitude > 0.1)
    {
      _audioSource.PlayOneShot(BounceOtherClip);
    }
  }
}

So if a cube collides with a GameObject that has a CursorManager – it’s a cursor, so ignore it. But if it has a CubeManipulator - it’s a cube too! Then play the BounceTogetherClip AudioClip. But only if the Id of this cube is lower than the cube we hit – to prevent double sounds. If it was not a cursor or another cube, it was most likely a wall – so play the BounceOtherClip, but only if the cube hit the wall ‘hard enough’.You do this by checking the collision magnitude. I took a quite arbitrary value. Once again – it feels like cheating. Unity takes care of mostly everything.

Aside – don’t try to be smart and change “if (othercube != null && othercube.Id < Id)” into
“if (othercube?.Id < Id)”. It will compile, it will run, and it will also mess up Unity3d. Avoid C# 6 constructs.

We only need to do a couple of things more to not only see but also hear our app at work. too. Save the script, then go back to Unity. Open the WortellCube prefab again, scroll down to the CubeManipulator script component. You will see – like expected – the script has three extra properties now: Id, Bounce Together Clip and Bounce Other Clip. So drag the Audio asset “BounceCube” on top of the Bounce Together Clip field, and BounceWall on top of Bounce Other Clips. Net result should be this:

Build the project again, and go back to Visual Studio. There is that tiny thing about the Id we have to solve. Because now we have an Id, but nothing is set yet. Remember that in the previous post the method CreateCube in MainStarter.cs had an id parameter that was not used? Well, now we are going to use it. We change that method a little by adding two lines:

private void CreateCube(int id, Vector3 location, Quaternion rotation)
{
  var c = Instantiate(Cube, location, rotation) as GameObject;
  //Rotate around it's own up axis so up points TO the camera
  c.transform.RotateAround(location, transform.up, 180f);
  var m = c.GetComponent<CubeManipulator>();
  m.Id = id;
}

And thus we assign the Id we need to make only one cube play the bounce together sound when two cubes hit. Deploy the project to a HoloLens or an emulator, and the cubes click when they bounce together, or emit a kind of boom when they hit the wall:

Also, if you have an actual HoloLens, try turning your head after launching some cubes – you will hear the sound actually coming from the right direction. This is because we have included an AudioSource component configured for Spatial Sound in the WortellCube prefab - and that travels along with the cube. Once again – it feels like cheating, but it works.

Concluding remarks

We have learned to intercept air taps, and apply directed force to a rigid body – making cubes all bounce by themselves (or actually, by Unity. We have also learned to act on collisions, and play spatial sound. Still by writing very, very little code. Next time – speech command and moving stuff yourself!

You can find, as always, the completed project so far here.