Building a floating audio player in Mixed Reality

Intro

As I promised in my previous blog post, I would write about how I created the floating audio player designed to easily demonstrate how to download and play audio files in Mixed Reality (or actually, just Unity, because the code is not MR specific). I kind of skipped over the UI side. In this post I am going to talk a little more about the floating audio player itself. This code is using the Mixed Reality Toolkit and so actually is Mixed Reality specific.

Dissecting the AudioPlayer prefab

The main game object

The AudioPlayer consists out of two other prefabs, a SquareButton and a Slider. I have talked about this button before, so I won’t go over that one in detail again. The main game object of the AudioPlayer has an AudioSource and two extra scripts. The simple version of the Sound Playback Controller was already described in the previous blog post, and will be handled in great detail here. The other script is a standard Billboard script from the Mixed Reality toolkit. It essentially keeps the object rotated towards the camera, so you will never see it from the side of the backside where it’s hard to read and operate. Note I have restricted pivot axis to Y, so it only rotates over a vertical axis.

The button

It’s a fairly standard SquareButton, and I have set the text and icon as I described here. Now that button only shows in the editor, the runtime text and the icon are set by a simple script that toggles icon and text, so that the button cycles between being a “Play” and a “Pause” button. That script is pretty easy:

using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class IconToggler : MonoBehaviour, IInputClickHandler
{
    public Texture2D Icon1;

    public Texture2D Icon2;

    public string Text1;

    public string Text2;

    private TextMesh _textMesh;

    private GameObject _buttonFace;

    void Awake ()
    {
        _buttonFace = gameObject.transform.
           Find("UIButtonSquare/UIButtonSquareIcon").gameObject;
        var text = gameObject.transform.Find("UIButtonSquare/Text").gameObject;
        _textMesh = text.GetComponent<TextMesh>();
        SetBaseState();
    }

    public void SetBaseState()
    {
       _textMesh.text = Text1;
       _buttonFace.GetComponent<Renderer>().sharedMaterial.mainTexture = Icon1;
    }

    private float _lastClick;

    public void OnInputClicked(InputClickedEventData eventData)
    {
        if (Time.time - _lastClick > 0.1)
        {
            _lastClick = Time.time;
            Toggle();
        }
    }

    public void Toggle()
    {
        var material = _buttonFace.GetComponent<Renderer>().sharedMaterial;
        material.mainTexture = material.mainTexture == Icon1 ? Icon2 : Icon1;
       _textMesh.text = _textMesh.text == Text1 ? Text2 : Text1;
    }
}

It has four public properties, as already is visible in the image: Image1 and Text1 for the default image and text (“Play”), Image 2 and Text 2 for the alternate image and text (“Pause”). The Awake method grabs some objects within the button itself, then sets the base state – which is, the default icon and text.

It also implements IInputClickHandler, so the user can tap it. In OnInputClicked it calls the Toggle method. That then toggles both text and image. Notice there’s simple time based guard OnInputClicked. This is to prevent the button from sending a burst of click events. In the Unity editor, I mostly get two clicks every time I press the XBox controller A button, and then nothing happens. Annoying, but easily mitigated this way.

The Slider

I can be short about that one. I did not create that, but simply nicked it from the Mixed Reality Toolkit Examples. It sits in HoloToolkit-Examples\UX\Prefabs. I like making stuff, but I like stealing reusing stuff even better.

The extended Sound Playback Controller

Let’s start at Start ;). Note: the BaseMediaLoader was handled in the previous blog post,

public class SoundPlaybackController : BaseMediaLoader
{
    public AudioSource Audio;

    public GameObject Slider;

    public GameObject Button;

    private SliderGestureControl _sliderControl;

    private IconToggler _iconToggler;

    public AudioType TypeAudio = AudioType.OGGVORBIS;

    void Start()
    {
        _sliderControl = Slider.GetComponent<SliderGestureControl>();
        _sliderControl.OnUpdateEvent.AddListener(ValueUpdated);
        Slider.SetActive(false);
        Button.SetActive(false);
        _iconToggler = Button.GetComponent<IconToggler>();
    }
}

In the Start method, we first grab a bunch of stuff. Note the fact that we not only turn off the slider control but also actually attach an event handler to that.

We continue with StartLoadMedia and LoadMediaFromUrl

protected override IEnumerator StartLoadMedia()
{
    Slider.SetActive(false);
    Button.SetActive(false);
    yield return LoadMediaFromUrl(MediaUrl);
}

private IEnumerator LoadMediaFromUrl(string url)
{
    var handler = new DownloadHandlerAudioClip(url, TypeAudio);
    yield return ExecuteRequest(url, handler);
    if (handler.audioClip.length > 0)
    {
        Audio.clip = handler.audioClip;
        _sliderControl.SetSpan(0, Audio.clip.length);
        Slider.SetActive(true);
        Button.SetActive(true);
        _iconToggler.SetBaseState();
    }
}

The override from StartLoadMedia in this version turns off the whole UI while we are actually loading data, and turns it on when we are done loading. Since that fails when we load MP3, the MP3 player in the demo project disappears and on startup. The others one disappear too, in fact, but immediately appear again since we are loading small clips. This goes so fast you can’t even see it.

LoadMediaFromUrl not only executes the request and sets the downloaded clip to the Audio Souce, as we saw before, but we also set the span of the Slider Control between 0 and the length of the AudioClip in seconds. Easy, right?

Now the Update method, which as you know is called 60 times per second, is the trick to keeping the slider equal to the the current time of the clips that’s now playing:

protected override void Update()
{
    base.Update();
    if (Audio.isPlaying)
    {
        _sliderControl.SetSliderValue(Audio.time);
    }
    if (Mathf.Abs(Audio.time - _sliderControl.MaxSliderValue) < 0.1f)
    {
        Audio.Stop();
        Audio.time = 0;
        _iconToggler.SetBaseState();
        _sliderControl.SetSliderValue(0);
    }
}

Thus if the audio clip plays, the slider moves along. It’s not quite rocket science. If the clip has nearly finished playing, it is stopped and everything is set to the base state: the icon, the time of the audio clip, and the slider is set to 0 again.

And finally – remember that event handler we added to the OnValueUpdated event of the slider? Guess what:

private void ValueUpdated()
{
    Audio.time = _sliderControl.SliderValue;
}

It’s the opposite of the third line of Update – now we set the Audio time to the Slider value.

Conclusion

And that’s it. You can simply use some out-of-the-box components in the Mixed Reality Toolkit and/or it’s examples to build a simple but effective control to play audio. You can grab the demo project (it’s still the same) from here.