Downloading holograms from Azure and using them in your Mixed Reality application

Intro

In my previous post I explained two ways of preparing holograms and uploading them to Azure: using a whole scene, and using just a prefab. In this post I am going to explain how you can download those holograms in a running Mixed Reality or HoloLens app and an show them.

Setting the stage

We uploaded two things: a single prefab of an airplane, with a behavior attached, and a scene containing a prefab – a house, also with a behavior attached. The house rotates, the airplane follows quite an erratic path. To access both we created a Shared Access Signature using the Azure Storage Explorer.

In the demo code there’s a Unity project called RemoteAssets. We have used that before in earlier posts. The third scene (Assets/App/Demo3/3RemoteScenes) is the scene that actually tries to load the holograms

If you open that scene, you will see two buttons: “Load House” and “Load Plane”

nicked from the Mixed Reality Toolkit Examples I nicked these buttons. The left button, “Load House”, actually loads the house. It does so because it’s Interactive script’s OnDownEvent calls SceneLoader.StartLoading

 

Loading a remote scene

This SceneLoader is not a lot of code and does a lot more than is strictly necessary:

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;
using UnityEngine.SceneManagement;

public class SceneLoader : MonoBehaviour
{
    public string SceneUrl;

    public GameObject Container;

    private bool _sceneIsLoaded;

    public void StartLoading()
    {
        if (!_sceneIsLoaded)
        {
            StartCoroutine(LoadScene(SceneUrl));
            _sceneIsLoaded = true;
        }
    }

    private IEnumerator LoadScene(string url)
    {
        var request = UnityWebRequest.GetAssetBundle(url, 0);
        yield return request.SendWebRequest();
        var bundle = DownloadHandlerAssetBundle.GetContent(request);
        var paths = bundle.GetAllScenePaths();
        if (paths.Length > 0)
        {
            var path = paths[0];
            yield return SceneManager.LoadSceneAsync(path, LoadSceneMode.Additive);
            var sceneHolder = GameObject.Find("SceneHolder");
            foreach (Transform child in sceneHolder.transform)
            {
                child.parent = Container.transform;
            }
            SceneManager.UnloadSceneAsync(path);
        }
    }
}

It downloads the actual AssetBundle using a GetAssetBundle request, then proceeds to extract that bundle using a DownloadHandlerAssetBundle. I already wrote about the the whole rigmarole of specialized requests and accompanying handlers in an earlier post. Then it proceeds to find all scenes in the bundle, picks the first once, and loads this one additive to the current scene. If you comment out all lines after the LoadSceneAsync and run the code, you will be actually able to see what’s happening – a second scene inside the current scene is created.

If you however run the full code, the SubUrb house will appear inside the HologramCollection and no trace of the BuildScene will remain. That’s because the code tries to find a “SceneHolder” object (and you can see that’s the first object in the BuildScene), moves all children (one, the house) to the Container object and once that is done, the additional scene will be unloaded again. But the hologram that we nicked from it still remains, and it even rotates. If you look very carefully in the editor if you click the button, you can actually see the scene appear, see the house being moved from it, and then disappear again.

The result: when you click “Load House” you will see the rotating house, 2 meters before you.

Success. Now on to the airplane.

Loading a remote prefab

This is actually less code, or at least – less code than the way I chose to handle scenes:

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;

public class PrefabLoader : MonoBehaviour
{

    public string AssetUrl;

    public GameObject Container;

    private bool _isLoaded;

    public void StartLoading()
    {
        if (!_isLoaded)
        {
            StartCoroutine(LoadPrefab(AssetUrl));
            _isLoaded = true;
        }
    }

    private IEnumerator LoadPrefab(string url)
    {
        var request = UnityWebRequest.GetAssetBundle(url, 0);
        yield return request.SendWebRequest();
        var bundle = DownloadHandlerAssetBundle.GetContent(request);
        var asset = bundle.LoadAsset<GameObject>(bundle.GetAllAssetNames()[0]);
        Instantiate(asset, Container.transform);
    }
}

The first part is the same, then we proceed to use bundle.LoadAsset to extract the first asset by name from the bundle as a game object (there is only one, so that’s always correct for this bundle). And then we instantiate the asset – a prefab, which is a game object, into the hologram collection.

If you click the “Load Plane” button the result is not what you might expect:

Uhm, what? We basically did the same as before, actually less. It turns out, the only reason the house rotated fine, was because I used the Horizontal Animator from my own HoloToolkitExtensions. That script is present in both the SceneBuilder project (that I used to create the bundles uploaded to Azure) and the target app, RemoteAssets, that downloads the assets bundles and tries to use them.

But for the airplane to move around, I created a script “MoveAround” that is only present in the SceneBuilder. It does not happen often, dear reader, but I intentionally checked in code that fails, to hammer home the following very important concept:

In an Asset Bundle you can put about anything – complete scenes, prefabs, images, materials and whatnot – everything but scripts.

In order to get this to work, the script and its meta file need to be copied to the target project. Manually.

It does not matter much where in the target project it comes, Unity will pick it up, resolve the script reference, the bundle will load successfully if you press the “Load Plane” button. I tend to place it next to the place where it’s used.

And lo and behold: and airplane moving once again like a drunken magpie.

Concluding words

I have shown you various ways to upload various assets – JSON data, images, videos and finally holograms to Azure, how to download them from you app, and what limitations you will need to consider.

Important takeaways from this and previous posts:

• Yes, I could have downloaded earlier assets using an Unity Asset Bundle as well, in stead of downloading images etc. via a direct URL. Drawback of using an Asset Bundle is you will always need Unity to build it. If you are building an app for a customer that wants to update training images or videos, it’s a big plus if you can just have them uploaded to Azure using the Storage Explorer or some custom (web) app. Whatever the customer can change or maintain themselves, the better it is.
• You can’t download dynamic behavior, only (static) assets. The most ‘dynamics’ a downloaded asset can have is referring to a script that exists both in the building and the target app. I have seen complex frameworks that tried to achieve downloadable behavior by storing properties into the project file but that usually is a lot of work to achieve only basic functionalities (like moving some parts or changing colors and stuff) but while that may for simple applications, approaches like that are complex to maintain, a lot of work to ‘program’ into your asset, brittle and hard to transfer between project. Plus, it still needs Unity, and your customer is not going to use that.
  Rule of thumb is always: if you want to change the way things look, you can download assets dynamically. If you need new behavior, you will need to update the app.

I hope you enjoyed this brain dump. The project can (still) be found here.