However, users interact mostly with the user interface, not the business logic. If the UI is flawed, even with correct business logic, it’s like having a car with all components tested except the connection between the steering wheel and the wheels. At some point, a user is going to messily encounter this oversight while taking an offramp doing 75.
In most cases, this is solved by a comprehensive manual testing plan. However, the MRTK3 contains a lot of UI interaction tests, and a host of classes that make simulated user input possible. And the greatest thing is: with some futzing around, you can actually use those classes yourself to make your own simulated UI test:
For this demo, I have put together a simple set of requirements:
The menu itself you could see (very briefly) in the intro ‘movie’ at the top of this article.
I am assuming a project set up for MRTK3 and having some functionality in it.
First, right-click on “Assets” in your project, then hit Create / Testing / Test Assembly Folder and give it a name. I called mine InteractionTests. Now, as I have written before, if you define assemblies yourself, Unity is not going to do you the pleasure of auto-referencing assemblies anymore, so you all have to do that yourself. Which ones, depends on what you need. I have found out that for my particular tests, we need to add the following assemblies as references:
The first two are added by default. Assemblies MRTK.Input.RuntimeTests and MRTK.Core.TestUtilities contain the actual utility classes we need to write code to simulate input, and the other three contain classes that we need to check results - like if a button is toggled or not.
The next step is very weird. You see, if you now write code in your test class and use the TestHand
class from the MixedReality.Toolkit.Input.Tests
namespace, Unity cannot find it. While it very clearly is there:
It wasn’t until I stumbled on the 17th comment on this post from 2019 in the Unity forums that I found out what I needed to do: go to the Packages folder, manually edit the manifest.json file, and add the following at the end:
"testables" :
[
"org.mixedrealitytoolkit.input",
"org.mixedrealitytoolkit.core"
]
You can’t make this up.
As always, if you know what you are looking for, the testables entry is actually mentioned in Unity’s infamously confusing documentation, but not with this critical piece of knowledge. Anyway, the result is A) you can now finally use the MixedReality.Toolkit.Input.Tests
utility classes, and B) all the unit tests in both assemblies now show up in your Test Runner, next to the ones you are going to add (in this picture, they already are).
The menu looks like this, and we need this information to be able to see how the menu responds to input actions.
For UX tests, we can utilize the BaseRuntimeHandInputTests
class. This is a subclass of the MRKT BaseRuntimeInputTests
, that takes care of a lot of things, like setting up a test scene with an MRTK XR Rig, and destroying it after the test.
public class ButtonsTests : BaseRuntimeHandInputTests
{
private const string MenuGuid = "e9ddf3517c4b9c7488c12bdec6a9917f";
private GameObject testGameObject;
private List<pressablebutton> allButtons;
At the top, you see the menu prefab guid, which you can find in the Menu.prefab.meta file:
fileFormatVersion: 2
guid: e9ddf3517c4b9c7488c12bdec6a9917f
PrefabImporter:
externalObjects: {}
userData:
assetBundleName:
assetBundleVariant:
As well as some other things we will need in the tests. Below that, the Init method creates the prefab and gathers some information about the prefab: its initial position and the buttons.
[SetUp]
public void Init()
{
testGameObject = InstantiatePrefab(MenuGuid);
allButtons = FindByName(testGameObject, "Buttons-GridLayout").
GetComponentsInChildren<pressablebutton>().ToList();
}
The Teardown just destroys the object. Note it does not need a [TearDown] attribute; the base class takes care of that.
public override IEnumerator TearDown()
{
yield return base.TearDown();
Object.Destroy(testGameObject);
}
The test that tests requirement 2 - only one button can be toggled - is the most complex. Or actually, it does the most.
[UnityTest]
public IEnumerator PressingTwoDifferentButtonsShouldOnlySelectTheLast()
{
var pressedButtons = new List<pressablebutton>();
var initialHandPosition = GetInitialHandPosition();
TestHand hand = null;
yield return GetHand(initialHandPosition, h => { hand = h; });
We need a list with buttons that are already pressed to make sure we don’t press the same button twice. Then we calculate get a first hand postion, which does not matter really, but we need an initial position. And then we test the initial condition: no buttons pressed.
First, we test if there are no toggled buttons. Then we move from hand to hand, poke the button, and every time there should only be one button toggled at any time.
Assert.AreEqual(0, GetToggledButtonCount());
foreach(var button in allButtons)
{
var handPosition =
GetInitialHandPositionBefore(button.gameObject, HandInFrontOfGameObject);
yield return MoveHandTo(hand, handPosition);
yield return PokeHand(hand, HandInFrontOfGameObject);
Assert.AreEqual(1, GetToggledButtonCount());
AddButtonToPressedList(pressedButtons);
}
A lot of the helper methods that I created to make things easier, are defined in base class BaseRuntimeHandInputTests
(that extends the MRTK3 class BaseRuntimeInputTests
, as I metioned before).
public abstract class BaseRuntimeHandInputTests : BaseRuntimeInputTests
{
protected const int HandMoveSteps = 1;
protected const int UpdateFrames = 1;
protected const float HandInFrontOfGameObject = 0.15f;
protected const float InitialHandInFrontOfUserDistance = 0.2f;
HandMoveSteps
is used in the methods that actually move the hand; the lower the number, the fewer steps are taken in moving the hand - so the hand moves faster. UpdateFrames
is the wait time after a hand move or creation. Here also goes: a lower number is a faster unit test. These numbers might be adapted to debug the test visually.
protected Vector3 GetInitialHandPosition(
float initialDistance = InitialHandInFrontOfUserDistance)
{
return InputTestUtilities.InFrontOfUser(Vector3.forward * initialDistance);
}
protected Vector3 GetInitialHandPositionBefore(
GameObject testGameObject,
float initialDistance = HandInFrontOfGameObject)
{
return testGameObject.transform.position - Vector3.forward * initialDistance;
}
There are basically two methods doing this: GetInitialHandPosition
gets a position before the user, GetInitialHandPositionBefore
get a position in front of a game object, so you can move the hand simply forward and press a button, for instance.
A bit of an oddball method - it creates the hand at the initial position. Since an IEnumerator can’t return a value and also can’t use ref out variables (I tried), I used a lambda to return the actual hand:
protected IEnumerator GetHand(Vector3 initialHandPosition, Action<testhand> action)
{
var hand = new TestHand(Handedness.Right);
yield return hand.Show(initialHandPosition);
yield return RuntimeTestUtilities.WaitForUpdates(UpdateFrames);
action(hand);
}
Note the RuntimeTestUtilities.WaitForUpdates
call. This needs to be done after every hand creation or move; otherwise, the test code will throw a “Cached unprocessed value unexpectedly became outdated for unknown reason, new value ‘0’ old value ‘3’” error.
With everything in place, now it’s very simple to move the hand.
protected IEnumerator PokeHand(TestHand hand, float distance)
{
yield return MoveHand(hand, Vector3.forward * distance);
yield return MoveHand(hand, -Vector3.forward * distance);
}
protected IEnumerator MoveHand(TestHand hand, Vector3 distance)
{
yield return hand.Move(distance, HandMoveSteps);
yield return RuntimeTestUtilities.WaitForUpdates(UpdateFrames);
}
protected IEnumerator MoveHandTo(TestHand hand, Vector3 location)
{
yield return hand.MoveTo(location, HandMoveSteps);
yield return RuntimeTestUtilities.WaitForUpdates(UpdateFrames);
}
PokeHand
moves the hand forward and backward, MoveHand
moves the hand forward over a specific vector (so relative from the current position), and MoveHandTo
moves the hand to a specific absolute location. These methods only add a RuntimeTestUtilities.WaitForUpdates
but it’s a bit annoying to have to add that yourself after every call.
The test method itself actually checks every time if only one button is toggled, but this would still work for pressing only two buttons (back and forth). To make sure every button press is a different press, I wrote this little routine:
private void AddButtonToPressedList(List<pressablebutton> pressedButtons)
{
var button = buttons.FirstOrDefault(b => b.IsToggled);
if (!pressedButtons.Contains(button))
{
pressedButtons.Add(button);
}
else
{
Assert.Fail("Button already pressed");
}
}
This I basically stole from existing code in the MRTK3 itself, with a little adaptation. This is how you load a prefab from a guid, then instantiate it.
private GameObject InstantiatePrefab(string guid)
{
var prefabPath = AssetDatabase.GUIDToAssetPath(guid);
var prefab = AssetDatabase.LoadAssetAtPath(prefabPath, typeof(Object));
return Object.Instantiate(prefab) as GameObject;
}
This is a rather standard routine that recursively looks for an a game object by name, below a starting object.
protected GameObject FindByName(GameObject parent, string name)
{
if (parent.name == name)
{
return parent;
}
foreach (Transform child in parent.transform)
{
var result = FindByName(child.gameObject, name);
if (result != null)
{
return result;
}
}
return null;
}
Yes, I know this is a trivial case. Yes, I know PressableButton
has methods that can simulate clicks, so you don’t need to go this roundabout way. Yes, I know the only-one-button-toggled logic should be driven by business logic that could be checked. Yes, I also know this is technically integration testing, not unit testing. That is not the point of this blog post: the point is to show how to set up and execute these kinds of automated UI tests using stuff that is already in the MRTK3. You can do all kinds of nifty things with hands, and this is very useful for finding events that are wired up in the editor but were broken later. The code in this blog post can be a useful starting point.
(Almost) everything in the ServiceFramework starts with a profile, and so does this service.
public class UserPresenceServiceProfile : BaseServiceProfile<IServiceModule>
{
[SerializeField]
private InputActionReference gazeTrackingState;
[SerializeField]
private float userAwayWaitTime = 3.0f;
[SerializeField]
private float userPresentWaitTime = 0.5f;
public InputActionReference GazeTrackingState => gazeTrackingState;
public float UserAwayWaitTime => userAwayWaitTime;
public float UserPresentWaitTime => userPresentWaitTime;
}
userAwayWaitTime
is the time (in seconds) the service needs to detect the continued user absence before it signals the outside world. This is to prevent events being triggered when users so much as blink, or the eye tracking loses track for a few moments.
userPresentWaitTime
is the time (also in seconds) the service needs to detect the continued return of the user after absence.
gazeTrackingState
is a reference to the Tracking State input action of the MRTK Default Input Actions. This is used to get the actual gaze state from the Interaction Manager in Unity’s XR Interaction Toolkit.
The service exposes only two items: the current user presence, and an event to tell the outside world the presence has changed.
public interface IUserPresenceService : IService
{
public bool IsUserPresent { get; }
public UnityEvent<bool> UserPresenceChanged { get; }
}
I will omit declarations and most of the constructor, as most of it is fairly standard for a Service Framework Service. The only thing worth noting is this line, where we pick up the reference to the gaze tracking state from the profile:
gazeTrackingState = profile.GazeTrackingState;
When the service is actually enabled, it only sets up a listener to the gaze tracking state’s action.performed
event. Note, this is actually not MRTK specific anymore - gazeTrackingState
is an InputActionReference
and that’s part of Unity’s Input System.
public override void Enable()
{
if (isInitialized)
{
return;
}
isInitialized = true;
gazeTrackingState.action.performed += GazeTrackingStateChanged;
}
private void GazeTrackingStateChanged(InputAction.CallbackContext ctx)
{
gazeStateResult = ctx.ReadValue<int>();
}
Now the thing to keep in mind is - if you run this in the editor, GazeTrackingStateChanged
gets called a few times and that’s it. When you run this on HoloLens 2, GazeTrackingStateChanged
gets really hammered with events. It seems like the eye tracker is firing this event all the time. That’s why the only thing we do is put the value in a field, and let the service’s Update loop handle the logic.
In the Update loop, we first check the state value. I have seen that “3” means “eye tracking detected”. I also have seen value “0” when I took off the device, so I have chosen to interpret “3” as “user present” and anything else as “not present”
public override void Update()
{
var newState = gazeStateResult == 3;
First step: if the detected new state is the same as the current state, remember that last requested state, and exit the method
if (newState == IsUserPresent)
{
lastRequestedState = newState;
return;
}
Second step: if there is, however, a new state, the method runs to the second if. If the lastRequestedState
does not match the newState
yet, that means a state change has taken place. This is registering noting the time the state change has taken place.
if( newState != lastRequestedState)
{
lastRequestedState = newState;
lastStateChangeTime = Time.time;
}
So. The first if doesn’t do anything anymore, as newState
and IsUserPresent
are not equal. But newState
and lastRequestedState
are equal, so lastStateChangeTime
stays fixed. Now the clock starts ticking in the last part start:
if( Time.time - lastStateChangeTime > (lastRequestedState ?
profile.UserPresentWaitTime : profile.UserAwayWaitTime))
{
IsUserPresent = lastRequestedState;
UserPresenceChanged.Invoke(IsUserPresent);
}
}
If the user does not do anything that makes the state flip again (at which point the first if kicks in again and ‘stops the clock’), IsUserPresent
is set and and the event is called. The time to wait before the event indicating state change is determined by whether the presence changes from true to false, or the other way around. It’s not really rocket science.
I have added a demo scene with a simple behaviour UserPresenceDisplayer
that shows how you might use this. The interesting parts are this:
public class UserPresenceDisplayer : MonoBehaviour
{
private async Task Start()
{
audioSource = GetComponent<AudioSource>();
await ServiceManager.WaitUntilInitializedAsync();
userPresenceService =
ServiceManager.Instance.GetService<IUserPresenceService>();
userPresenceService.UserPresenceChanged.AddListener(OnUserPresenceChanged);
}
private void OnUserPresenceChanged(bool currentPresence)
{
displayText.text = $"User is {(currentPresence ? "present" : "away")}";
audioSource.PlayOneShot(currentPresence ? userPresentClip : userAwayClip);
}
}
It waits for the Service Manager to be ready, then gets a reference to the service. When events are received, it shows an appropriate message on a floating text, plays a high note when the user presence changes from away to present, and a low note when it changes from present to away.
I have found it is best to set a slightly longer wait time (3-5 seconds) before firing the “user away” event before pausing whatever you want to pause, as false positives can be really annoying to the user. However, if the user returns, you want your app’s functionality back up to speed ASAP, so that is usually a shorter time. However, I can also imagine scenarios where a quick “user away” event is necessary, for instance when an app is used to monitor an exam or something. You can simply make multiple profiles for that without needing to change any code. That’s the beauty of the Service Framework.
Note: so far, this has been tested on HoloLens 2 only. For that, it will require the Eye Gaze Interaction profile being set in the OpenXR Eye Tracking interaction profiles.
And it will, obviously, only work with devices that actually support eye tracking. I will conduct experiments with Magic Leap 2 soon.
]]>This, of course, would not do. And, also of course, stubborn as I am, I banged against it until it worked again. Only this needs a little more code. But it’s also now a bit more beautiful, as the buttons actually are now animated:
I hope y’all can appreciate the unplanned cameo of a Starling on the bird feeder outside the window :).
If you are not interested in the why and the how:
NonNativeKeyboard.Instance.Open();
And you have a keyboard. The TouchableNonNativeKeyboard has been scaled to what I think is a usable size, and also has a MRTK3 RadialView to keep it in view, and my helper behaviour AppearInCenterViewController to make it appear in the center of your view. It works the same as the normal NonNativeKeyboard. In fact, it just is the normal NonNativeKeyboard, just with some added stuff.
The basic NonNativeKeyboardTouchAdapter
is very simple: it changes some settings to the audio component because I think this works better for typing. All MRTK3 buttons have no spatial sounds, so why should these have it. Then it simply loops over all the Button child objects - even the inactive ones - and adds a NonNativeKeyTouchAdapter
.
public class NonNativeKeyboardTouchAdapter : MonoBehaviour
{
private void Awake()
{
var defaultAudioComponent = GetComponent<AudioSource>();
defaultAudioComponent.playOnAwake = false;
defaultAudioComponent.spatialize = false;
}
private void Start()
{
var buttons = GetComponentsInChildren<Button>(true);
foreach (var button in buttons)
{
// The search box has an incorrect collider and should not act as a
// button anyway
if (button.gameObject.name != "search")
{
button.gameObject.EnsureComponent<NonNativeKeyTouchAdapter>();
}
}
}
}
The NonNativeKeyTouchAdapter
actually does all the work - for every key there’s one. Except “search”. I think that actually has a Button
behaviour by accident, as its OnClick method goes nowhere.
At Awake, it calculates a few things related to the button’s animation - where it should start, and where it should end.
private void Awake()
{
defaultPosition = transform.localPosition;
animatedPosition = defaultPosition + new Vector3(0, 0, AnimationMovementDelta);
}
In OnEnable, we make sure the button always is at the default position, because it might have been halfway when the key disappeared. The keyboard consists of several panes, and another one appears if you press the “ABC” button or the “&123” button. For the same reason, we must reset the lastClickTime (we don’t want a button to be pressable again too quickly, otherwise it will quickly repeat) and the location of the button collider should also be set. Why this is, I will explain later.
private void OnEnable()
{
transform.localPosition = defaultPosition;
lastClickTime = Time.time;
if (isInitialized)
{
buttonCollider.center = buttonColliderDefaultCenter;
}
Initialize();
}
Initialization should only be done once - and why I used OnEnable
to kick it off instead of Awake
or Start
I will explain later as well.
First order of business is, of course, check if we did not already initialize in a different round of OnEnable
. Then we make a collider around the button that is a bit smaller than the actual size, to make it less likely the user hits two keys at once. The collider also is moved off-center; a bit ‘backwards’ so you won’t accidentally press ‘through’ the button easily.
private void Initialize()
{
if (isInitialized)
{
return;
}
isInitialized = true;
var rectTransform = GetComponent<RectTransform>();
buttonCollider = gameObject.EnsureComponent<BoxCollider>();
var size = new Vector3(
rectTransform.rect.size.x - ColliderMargin,
rectTransform.rect.size.y - ColliderMargin,
ColliderThickness);
buttonCollider.size = size;
buttonColliderDefaultCenter = new Vector3((size.x + ColliderMargin) / 2.0f,
(-size.y - ColliderMargin) / 2.0f, ColliderZDelta);
buttonCollider.center = buttonColliderDefaultCenter;
The resulting collider, when made visible, looks like this:
Before we do that, we first grab some stuff we will need:
image = GetComponent<Graphic>();
var defaultColor = image.color;
var button = GetComponent<Button>();
Then we add a StatefulInteractable
and set up the first event. Using my own blog about the available events, I took the firstSelectEntered
event, which is fired when something enters the collider. When this event is launched by a PokeInteractor
(i.e. your index finger) and it has not been clicked in the last ReClickDelayTime
seconds (1 by default), it fires the normal button
’s event, as if it was been clicked ‘the old way’. It also starts a coroutine to animate the button to its pressed position
interactable = gameObject.EnsureComponent<StatefulInteractable>();
interactable.firstSelectEntered.AddListener(selectArgs =>
{
if (selectArgs.interactorObject is not PokeInteractor ||
Time.time - lastClickTime < ReClickDelayTime)
{
return;
}
button.onClick.Invoke();
StartCoroutine(MoveButton(defaultPosition, animatedPosition));
});
Its mirror image is of course: when the user stops pressing the button, it moves the button back to its ‘unpressed’ position
interactable.lastSelectExited.AddListener(_ =>
{
StartCoroutine(MoveButton(animatedPosition, defaultPosition));
});
And then there’s this little line:
button.interactable = false;
We don’t want the Button
behaviour to interfere with us when using touch. We literally only hijack its OnClick
event and then turn it off. Otherwise, hover events go off and hand ray interaction is still possible, giving a pretty confusing experience.
Unfortunately, turning off the Button
behaviour disables all hover events, which is not nice, so we add that back simply using this, using the Button
’s highlightedColor
property
interactable.firstHoverEntered.AddListener(hoverArgs =>
{
SetColorOnHoverPoke(hoverArgs.interactorObject,
button.colors.highlightedColor);
});
interactable.lastHoverExited.AddListener(hoverArgs =>
{
SetColorOnHoverPoke(hoverArgs.interactorObject, defaultColor);
});
}
private void SetColorOnHoverPoke(IXRHoverInteractor interaction, Color color)
{
if (interaction is PokeInteractor)
{
image.color = color;
}
In the standard non-native keyboard, there is no animation at all, and neither did my MRTK2 behaviour add that. In the meantime, I am like 4 years along in my experience and ideas about UX, so I thought it cool to make the keys actually move. And they do, using this simple routine using a simple Vector3.Lerp
.
private IEnumerator MoveButton(Vector3 startPos, Vector3 endPos)
{
if (transform.localPosition == endPos)
{
yield break;
}
const float rate = 1.0f / AnimationTime;
var i = 0.0f;
while (i < 1.0f)
{
i += Time.deltaTime * rate;
var newPos = Vector3.Lerp(startPos, endPos, Mathf.SmoothStep(0f, 1f, i));
transform.localPosition = newPos;
buttonCollider.center =
buttonColliderDefaultCenter - (newPos - defaultPosition);
yield return null;
}
}
I actually had to look up how to use that Lerp in one of my very first HoloLens blogs from July 2016 because these days I always use LeanTween - but I did not want to create a dependency on that now. Also: there is something funky about this routine, as you might have noticed - it does something with the button’s position alright, but also something with the collider’s center position. This is because I quickly found out that when you move a button backwards, the collider is dragged along. And if you then just touch the button, you get this stupid effect:
So that’s why that when the button moves backward, the collider moves forward in exactly the opposite direction - with the net result the collider stays in the exact same place, while the visible graphics do not, and the ping-pong effect as shown above does not show.
Now to make sure button and collider positions don’t get messed up by buttons disappearing and re-appearing we keep a reference to the initial position of both collider and button itself, and that is why they are always reset to their starting position at the start of OnEnable.
For that, you actually have to do the spelunking in the MRTK3 I talked about. You see, every button also has a NonNativeValueKey
behaviour. This behaviour is a child class of NonNativeKey
and that does something funky in its Awake
method
protected virtual void Awake()
{
if (Interactable == null)
{
Interactable = GetComponent<StatefulInteractable>();
}
// If there is a StatefulInteractable, that is used to trigger the FireKey
// event. Otherwise the Button is used.
if (Interactable != null)
{
Interactable.OnClicked.AddListener(FireKey);
}
else
{
if (KeyButton == null)
{
KeyButton = GetComponent<Button>();
}
if (KeyButton != null)
{
KeyButton.onClick.AddListener(FireKey);
}
}
}
It checks if there’s a StatefulInteractable
around, and if so, it wires the click event not to the button, but to the interactable. And we want it to work like it did, so we can hijack the Button
’s OnClick
and not have it mess with the StatefulInteractable
. This is not a problem for the first keyboard_Alpha panel, as that is active by default, so the Awake
events for those buttons already done before we can add the StatefulInteractable
. But for the panels that are default inactive, there’s suddenly a StatefulInteractable
when they awake. As a workaround, the initialization of my NonNativeKeyTouchAdapter
is done on OnEnable
, so whatever happens - whenever the button’s NonNativeValueKey
buttons awake, they will never find a StatefulInteractable
and work as we want them to work.
There is actually a bug in the keyboard: the sounds only play for the keyboard_Alpha panel. This is because the MRTK3 KeyboardAudio behaviour only looks for active buttons and as I said before, the other panels are inactive. So on TouchableNonNativeKeyboard I have disabled KeyboardAudio and added behaviour FixedKeyboardAudio. It’s almost identical, but it looks for all buttons (using GetComponentsInChildren<Button>(true)
instead of just GetComponentsInChildren<Button>()
).
So why would you use this? Well, I think the days of only building single device apps are gone. These days, you try to target as many devices as possible. The Non Native Keyboard gives a consistent look & feel for text input, and is also easily to control as to where it appears and how - which native keyboards not always allow. In my approach, I have tried to mess with the original NonNativeKeyboard as little as possible - the TouchableNonNativeKeyboard prefab is a simple Prefab Variant from NonNativeKeyboard, with literally only a bit of scaling changed, as well as three added behaviours: NonNativeKeyboardTouchAdapter, and the already mentioned RadialView and AppearInCenterViewController.
I hope you will find it useful. The demo project, once again, is at GitHub
]]>The idea is that you can use this to have multiple ‘experiences’ connected to multiple locations, for instance, if you want people to learn about a particular machine, building, or any other thing you can use to stick a QR code on. The QR code makes it possible to tie that experience to a particular location and align it potentially to the QR code’s orientation. You can see the airplane and the capsule taking both position and orientation of the QR code, while the ‘custom experience’ only takes the position and places the ‘experience’ above the QR code, using world orientation.
I am going to give a short recap of how things work, referencing my original articles from early 2021 where possible.
If you want to know in detail how it works: basically the same as I described here
There are a few changes:
AutoEnable
property, which makes the service start scanning for QR codes as soon as it starts.In the demo project, you will see the service now uses an “AutoStartQRCodeTrackingServiceProfile” with AutoEnable
set to true
This automatically starts the service. As you can see in the piece of code below. Basically, only the last three lines are added.
public override void Update()
{
if (qrTracker == null && accessStatus ==
QRCodeWatcherAccessStatus.Allowed)
{
SetupTracking();
}
}
private void SetupTracking()
{
qrTracker = new QRCodeWatcher();
qrTracker.Updated += QRCodeWatcher_Updated;
IsInitialized = true;
Initialized?.Invoke(this, EventArgs.Empty);
SendProgressMessage("QR tracker initialized");
if( profile.AutoEnable)
{
Enable();
}
}
The scene setup is simple. For every QR code, there’s a Tracker and a TrackerDisplayer.
Its main behaviour is called “ContinuousQRTrackerController
” and I took the word “continuous” because, unlike in my previous samples, it does not stop the service once it has found a QR code, and you can individually reset it. You can see it looks for a QR code with payload “HLItem1”. You can, of course, change that to any value you like, as long as it matches the payload of the QR code you want to have tracked. Basically, the tracker does the following:
QRCodeFound
eventlocationQrValue
serialized fieldAlthough the last part is actually done by the Spatial Graph Coordinate Setter behaviour.
Some highlights of ContinuousQRTrackerController
’s code. At startup, it turns off the marker (more on that later), gets a reference to the QR tracking service, and waits a rather arbitrary 0.25 seconds to give the service time to start up. Then it sets up some events:
private IQRCodeTrackingService QrCodeTrackingService =>
qrCodeTrackingService ??= ServiceManager.Instance.GetService<IQRCodeTrackingService>();
private async Task Start()
{
markerHolder = spatialGraphCoordinateSystemSetter.gameObject.transform;
markerDisplay = markerHolder.GetChild(0).gameObject;
markerDisplay.SetActive(false);
ResetTracking(false);
// Give service time to start;
await Task.Delay(250);
if (!QrCodeTrackingService.IsSupported)
{
return;
}
QrCodeTrackingService.QRCodeFound += ProcessTrackingFound;
spatialGraphCoordinateSystemSetter.PositionAcquired += SetPosition;
}
When a QR code is found, we first check if the message has data at all, if the marker is already displayed, or if it has just been reset by the user, in which case we don’t do anything at all. We also check if we haven’t just processed this QR code in the last 200 ms, and then and only then we ask the spatialGraphCoordinateSystemSetter
to actually align the marker to the QR code.
private void ProcessTrackingFound(object sender, QRInfo msg)
{
if (msg == null || markerDisplay.activeSelf || resetTime > Time.time)
{
return;
}
lastMessage = msg;
if (msg.Data == locationQrValue &&
Math.Abs((DateTimeOffset.UtcNow -
msg.LastDetectedTime.UtcDateTime).TotalMilliseconds) < 200)
{
spatialGraphCoordinateSystemSetter.SetLocationIdSize(msg.SpatialGraphNodeId,
msg.PhysicalSideLength);
}
}
The marker is the blueish thing you see appear over the QR code:
ResetTracking is called from the little menus floating over the ‘experiences’, and they allow you to make the particular QR code trackable again. It gives you a two seconds grace period to get away from the QR code - this makes sense, otherwise, the QR code is immediately tracked again and immediately shows again.
public override void ResetTracking()
{
ResetTracking(true);
}
private void ResetTracking(bool delayed)
{
if (delayed)
{
resetTime = Time.time + 2;
}
markerDisplay.SetActive(false);
}
Basically, this is nearly 100% equal to what I described earlier in this article about upgrading the whole shebang to OpenXR.. It sits on the gameobject below the continuous tracker
This is a behaviour that ties an object to a tracked QR code.
As you can see, the QRPoseTrackController for the Jet tracks the ContinuousTracker1. It starts as follows:
public class QRPoseTrackController : MonoBehaviour
{
[SerializeField]
private BaseTrackerController trackerController;
[SerializeField]
private bool setRotation = true;
private AudioSource audioSource;
private Transform childObj;
private void Start()
{
audioSource = GetComponentInChildren<AudioSource>(true);
childObj = transform.GetChild(0);
childObj.gameObject.SetActive(false);
trackerController.PositionSet.AddListener(PoseFound);
}
Note it actually refers to a BaseTrackerController
rather than a ContinuousQRTrackerController
- this allows for building other controller logic. It also has an option to not only set location but also rotation. For the airplane and the capsule, this is set to true, for the ‘custom experience’ to false. On startup, it gets the child object, tries to find an optional AudioSource
, and adds a listener to the TrackerController’s PositionSet
event.
When a position is found, it shows the objects on the QR code’s location, optionally aligns it, and plays a sound. The Task.Yield
thing is necessary because the AudioSource
is on a game object that is initially disabled (in the Start
method it says childObj.gameObject.SetActive(false)
, right?) and apparently Unity needs a frame to actually activate an AudioSource
before it can play the sound.
private void PoseFound(Pose pose)
{
if (setRotation)
{
childObj.SetPositionAndRotation(pose.position, pose.rotation);
}
else
{
childObj.position = pose.position;
}
childObj.gameObject.SetActive(true);
Task.Run(PlaySound);
}
private async Task PlaySound()
{
await Task.Yield();
if(audioSource != null && audioSource.clip != null)
{
audioSource.Play();
}
}
The only thing left is this little method
public void Reset()
{
trackerController.ResetTracking();
childObj.gameObject.SetActive(false);
}
This is the method called by the floating reset menu each ‘experience’ has, it basically deletes the whole experience, and resets the controller (giving you the 2-second grace time).
QR codes are a powerful and simple way to have objects appear at particular locations without having to set up all kinds of holograms in advance. This way, you can quickly set up an ‘experience’, a training scenario, or a kind of guided tour. A bit like a poor man’s Microsoft Dynamics 365 Guides ;). There are a few things to consider when using this code, though:
Have fun playing with it. The demo project is in this branch of the QRCodeService repo. This concludes my blogging for 2023, I wish you a happy 2024 both in Mixed and real Reality :)
]]>Anyway, after showing how to get the position of the hand while doing an air tap, I thought I was done on this subject. Nope: two different developers wanted to know if I could tell them how to get where the hand ray was projecting on.
Well, I don’t know if I have found the right way, or even the best way, but I at least have found a way. I modified my previous sample a bit (again), so now it not only shows for each hand where the hand itself is during a tap, but also where the end of the hand ray is. This is actually a pretty simple adaptation of the previous code. The start is more or less the same:
private void Start()
{
handsAggregatorSubsystem =
XRSubsystemHelpers.GetFirstRunningSubsystem<IHandsAggregatorSubsystem>();
leftHand.SetActive(false);
rightHand.SetActive(false);
findingService = ServiceManager.Instance.
GetService<IMRTK3ConfigurationFindingService>();
but then comes the interesting part. See, my MRTK3ConfigurationFindingService
does not only provide events to check if left or right hands are triggering in some way, but also direct access to the hands themselves. And the hands have a LineRender
component in their children:
var rightLineRenderer = findingService.RightHand.
gameObject.GetComponentInChildren<LineRenderer>(true);
var leftLineRenderer = findingService.LeftHand.
gameObject.GetComponentInChildren<LineRenderer>(true);
which happens to be the hand ray. And if you want to know where the end of the ray is: simply ask it the position of its last point like this:
var rayPos = leftLineRenderer.
GetPosition(leftLineRenderer.positionCount -1);
The whole thing that is triggered when you do an air tap with your left hand:
findingService.LeftHandStatusTriggered.AddListener(t=>
{
leftHand.SetActive(t);
if (t)
{
var rayPos = leftLineRenderer.
GetPosition(leftLineRenderer.positionCount -1);
textMesh.text = $"Left hand position: {GetPinchPosition(findingService.LeftHand)}";
textMesh.text +=
$"{Environment.NewLine} Left hand ray position: {rayPos}";
leftHand.transform.position = rayPos;
}
});
The code for the right hand is omitted, as it’s nearly identical. On a HoloLens 2, it looks like this:
To make sure the ray also hits physical objects (i.e., the spatial map), I have added an ARMeshManager to the project’s camera, as I described here. The caveat is - this hits everything with a collider - not only the spatial map, but also the cube floating in the air. If you want to distinguish with that, you will have to do ray casts along the direction of the LineRender yourself.
Demo project can be downloaded from this MRTKAirTap project branch.
]]>Il2CppOutputProject\Source\il2cppOutput\Symbols\il2cppFileRoot.txt does not exist Il2CppOutputProject\Source\il2cppOutput\Symbols\LineNumberMappings.json does not exist
There is very little to find online; about the only real source of information I found was this one, from June 2023, in the Unity forums. Basically, the solution is: edit the “Unity Data.vcxproj” and remove the entries describing those files. This, of course, does not work when you make a fresh build or run into this in a CI/CD pipeline, which was exactly how I learned about this when I upgraded Augmedit’s Lumi to a new major Unity solution.
The short version: if you are not interested in the how and why and want this issue fixed and you want it fixed now, just download this file, put it somewhere in your project, build the project again and be done with it forever.
So the problem is: Unity generates links in the Unity Data.vcxitems to files that are simply not there. And as the solution suggested in the Unity forums says, we can do without. So after my colleague Niek brought the Unity PostProcessBuild
attribute to my attention, off I went:
[PostProcessBuild(1)]
public static void FixVcxItemsFile(BuildTarget target, string pathToBuiltProject)
{
if (target == BuildTarget.WSAPlayer)
{
var vcxItemsFile =
Directory.GetFileSystemEntries(pathToBuiltProject,
"Unity Data.vcxitems",
SearchOption.AllDirectories).FirstOrDefault();
if (vcxItemsFile != null)
{
FixVcxItemsFile(vcxItemsFile);
}
}
}
When the build is done, Unity calls methods in static classes decorated with PostProcessBuild
. The number indicates the order in which they are to be called, should you have more than one, but since we don’t, that number is inconsequential. The method gets a build target and the path where the project is built to as parameters. We then search for the file “Unity Data.vcxitems” and if we find it, we are going to fix it.
The offending lines are pretty long and look like this, at least in Lumi:
<None Include="$(MSBuildThisFileDirectory)..\Il2CppOutputProject\Source\il2cppOutput\Symbols\il2cppFileRoot.txt">
<DeploymentContent>true</DeploymentContent>
<ExcludeFromResourceIndex>true</ExcludeFromResourceIndex>
</None>
<None Include="$(MSBuildThisFileDirectory)..\Il2CppOutputProject\Source\il2cppOutput\Symbols\LineNumberMappings.json">
<DeploymentContent>true</DeploymentContent>
<ExcludeFromResourceIndex>true</ExcludeFromResourceIndex>
</None>
And they sit inside an “ItemGroup” element. So the trick is to load the document into an XML processing API, find all “None” elements inside the project group that have an Include attribute that ends with either “il2cppFileRoot.txt” or “LineNumberMappings.json”. So I asked CoPilot to write me an algorithm, that compiled great and even ran beautifully, but unfortunately didn’t do anything - and it also did this very inefficiently. But at least it showed me what API to use in this context, and I came to the following code:
private static void FixVcxItemsFile(string vcxItemsFile)
{
var projectFile = XDocument.Load(vcxItemsFile);
var itemsToDelete = projectFile.Descendants().
FirstOrDefault(node => node.Name.LocalName == "ItemGroup")?.
Descendants().Where
(node => node.Name.LocalName == "None" &&
(node.IncludeAttributeEndsWith("il2cppFileRoot.txt") ||
node.IncludeAttributeEndsWith("LineNumberMappings.json")));
if (itemsToDelete != null)
{
foreach (var item in itemsToDelete.ToList())
{
item.Remove();
}
projectFile.Save(vcxItemsFile);
}
}
So it does exactly what I just wrote - it finds offending items, then tells them to delete themselves, and saves the remaining.
IncludeAttributeEndsWith
is a simple extension method that I wrote because the Linq statement is already complex enough:
private static bool IncludeAttributeEndsWith(this XElement element, string contents)
{
var attr = element.Attribute("Include");
if (attr == null) return false;
return attr.Value.EndsWith(contents);
}
And that’s all. Should you ever run into this strange error, you can get around it by using this little helper.
]]>For contrast, this is the HoloLens 2 profile. Here the OpenXR Hands API is selected instead of the Magic Leap one.
Magic Leap 2 also supports keyword recognition - there is a custom API for that in their SDK. It’s things like this that make cross-platform development difficult. Therefore I could not get the speech command in my port of Augmedit’s Lumi product to work. If only someone had thought of making a KeywordRecognitionSubsystem implementation so you could just as easily use keyword recognition as on HoloLens 2…
Well, good news. Someone just did. Yours truly.
I will not even start to pretend I completely understand how subsystems work, but I have a kind of an idea now. In theory, they can consist of 5 classes and an interface:
For KeywordRecognitionSubsystems, there’s already a lot of heavy lifting done. We actually only need to provide a subsystem and a provider and can do so by extending existing base classes for keyword recognition. The entire subsystem therefore looks like this:
[Preserve]
[MRTKSubsystem(
Name = "MRTKExtensions.MagicLeap.SpeechRecognition",
DisplayName = "MRTK MagicLeap KeywordRecognition Subsystem",
Author = "LocalJoost",
ProviderType = typeof(MagicLeapKeywordRecognitionProvider),
SubsystemTypeOverride = typeof(MagicLeapKeywordRecognitionSubsystem))]
public class MagicLeapKeywordRecognitionSubsystem :
KeywordRecognitionSubsystem
{
#if MAGICLEAP
[RuntimeInitializeOnLoadMethod(
RuntimeInitializeLoadType.SubsystemRegistration)]
static void Register()
{
var cinfo = XRSubsystemHelpers.
ConstructCinfo<MagicLeapKeywordRecognitionSubsystem,
KeywordRecognitionSubsystemCinfo>();
if (!Register(cinfo))
{
Debug.LogError($"Failed to register the {cinfo.Name} subsystem.");
}
}
#endif
}
The MRTKSubsystem
describes the subsystem and indicates which Subsystem and Provider are to be connected together. Optionally you can also configure a ConfigType in this attribute. The subsystem itself then only needs a static method decorated with a RuntimeInitializeOnLoadMethod
attribute to actually get launched on startup. It uses KeywordRecognitionSubsystemCinfo
cinfo class - which is part of the MRTK3, so we don’t have to create this ourselves. As I said before, I don’t know why this is necessary, but sometimes you simply have to do some cargo cult programming follow existing patterns.
The provider does the actual work. The provider derives from the abstract class KeywordRecognitionSubsystem.Provider
, which requires us to implement the following methods:
UnityEvent CreateOrGetEventForKeyword(string keyword);
void RemoveKeyword(string keyword);
void RemoveAllKeywords();
IReadOnlyDictionary<string, UnityEvent> GetAllKeywords();
Also, there are several life cycle methods we can override coming from KeywordRecognitionSubsystem.Provider
- methods that are largely the same as in a behaviour.
In the Start
method override, I have implemented some code to initialize the voice recognition. Attentive readers will notice that this - and a lot of the following code - is basically a modified version of the runtime voice intents sample on the Magic Leap developer docs - with a few additions by me to take care of some idiosyncrasies I ran into.
[Preserve]
internal class MagicLeapKeywordRecognitionProvider :
KeywordRecognitionSubsystem.Provider
{
private int commandId = 0;
private MLVoiceIntentsConfiguration voiceConfiguration;
public override void Start()
{
base.Start();
if (voiceConfiguration == null)
{
voiceConfiguration =
ScriptableObject.CreateInstance<MLVoiceIntentsConfiguration>();
voiceConfiguration.VoiceCommandsToAdd =
new List<MLVoiceIntentsConfiguration.CustomVoiceIntents>();
voiceConfiguration.AllVoiceIntents =
new List<MLVoiceIntentsConfiguration.JSONData>();
voiceConfiguration.SlotsForVoiceCommands =
new List<MLVoiceIntentsConfiguration.SlotData>();
}
if (!running)
{
MLVoice.OnVoiceEvent += OnVoiceEvent;
}
}
}
The line voiceConfiguration.SlotsForVoiceCommands
= … is one of those additions that proved to be necessary - if I didn’t add that, I got a null reference error. Note that running
is a read-only base class property that is set internally.
You can see that events and keywords are both added to the internal dictionary, as well as ‘intent’ to the Magic Leap API. GetAllKeywords
simply returns the keyword/event dictionary
public override UnityEvent CreateOrGetEventForKeyword(string keyword)
{
if (!keywordDictionary.ContainsKey(keyword))
{
keywordDictionary.Add(keyword, new UnityEvent());
AddIntentForKeyword(keyword);
SetupVoiceIntents();
}
return keywordDictionary[keyword];
}
public override IReadOnlyDictionary<string, UnityEvent> GetAllKeywords()
{
return keywordDictionary;
}
keywordDictionary
, by the way, is once again a base class property. It is a simple Dictionary<string, UnityEvent>
.
Here the same pattern: remove from the internal dictionary, remove intent. Oh, and disconnect any listeners from the event before we toss it out.
public override void RemoveKeyword(string keyword)
{
if(keywordDictionary.TryGetValue(keyword, out var eventToRemove))
{
eventToRemove.RemoveAllListeners();
keywordDictionary.Remove(keyword);
voiceConfiguration.AllVoiceIntents.Remove(
voiceConfiguration.AllVoiceIntents.First(k=> k.value == keyword));
SetupVoiceIntents();
}
}
public override void RemoveAllKeywords()
{
foreach( var eventToRemove in keywordDictionary.Values)
{
eventToRemove.RemoveAllListeners();
}
keywordDictionary.Clear();
voiceConfiguration.AllVoiceIntents.Clear();
SetupVoiceIntents();
}
If the subsystem is halted or destroyed, we need to handle things in the Stop
and Destroy
life cycles method, which are now pretty easy to make:
public override void Stop()
{
base.Stop();
MLVoice.OnVoiceEvent -= OnVoiceEvent;
}
public override void Destroy()
{
base.Destroy();
RemoveAllKeywords();
Stop();
}
Now we have set up the framework, there’s some little implementation details left. In the Start
method, we connected the OnVoiceEvent
method to the MLVoice.OnVoiceEvent
event. The implementation is pretty simple: we simply check if a keyword is recognized and if so find the event belonging to it - and then invoke that event, notifying possible external listeners.
private void OnVoiceEvent(in bool wasSuccessful,
in MLVoice.IntentEvent voiceEvent)
{
if (wasSuccessful)
{
if (keywordDictionary.TryGetValue(voiceEvent.EventName,
out var value))
{
value?.Invoke();
}
}
}
AddIntentForKeyword
creates the actual intent. The commandId
needs to be unique, so we simply use an incrementing integer
private void AddIntentForKeyword(string keyword)
{
var newIntent = new MLVoiceIntentsConfiguration.CustomVoiceIntents
{
Value = keyword,
Id = (uint)commandId++
};
voiceConfiguration.VoiceCommandsToAdd.Add(newIntent);
}
The last method is quite weird. In all methods, you can see that after adding or removing intents, the method SetupVoiceIntents
is called. This is because for any change of intents to be recognized, MLVoice.SetupVoiceIntents
needs to be called. At least, that seems to be the case. Here’s another idiosyncrasy I found: if an MLVoiceIntentsConfiguration
contains zero intents, MLVoice.SetupVoiceIntents throws an exception. So if it’s empty, I make sure there is at least one dummy intent - without an event
private void SetupVoiceIntents()
{
if (!voiceConfiguration.AllVoiceIntents.Any())
{
AddIntentForKeyword("dummyxyznotempty");
}
MLVoice.SetupVoiceIntents(voiceConfiguration);
}
To show it actually works, I have added a small piece of demo code that allows you to verify this actually works. If you have configured MagicLeapKeywordRecognitionSubsystem
as your keyword recognition system in the MRTK3 settings and deploy it to the Magic Leap, you can use see this user interface:
By default, it recognizes three phrases. You can add a “Hello there”, remove it again, remove all, restore the initial commands, and toggle on/off the whole recognizer. Adding commands goes like this:
public void InitStandardPhrases()
{
RemoveAll();
keywordRecognitionSubsystem.CreateOrGetEventForKeyword("Good morning").
AddListener(() => ShowRecognizedCommand("Good morning"));
keywordRecognitionSubsystem.CreateOrGetEventForKeyword("Nice weather").
AddListener(() => ShowRecognizedCommand("Nice weather"));
keywordRecognitionSubsystem.CreateOrGetEventForKeyword("Mixed Reality is cool").
AddListener(() => ShowRecognizedCommand("Mixed Reality is cool"));
UpdateRecognizedCommands();
}
Removing keywords goes like this:
public void RemoveHello()
{
keywordRecognitionSubsystem.RemoveKeyword("Hello there");
UpdateRecognizedCommands();
}
public void RemoveAll()
{
keywordRecognitionSubsystem.RemoveAllKeywords();
UpdateRecognizedCommands();
}
and controlling the keyword recognizer like this:
public void ToggleKeywordRecognition()
{
if (keywordRecognitionSubsystem.running)
{
keywordRecognitionSubsystem.Stop();
}
else
{
keywordRecognitionSubsystem.Start();
}
}
Using this keyword recognizer, you can use exactly the same MRTK3 API for creating and responding to keywords as you were used to using on HoloLens, with the WindowsKeywordRecognitionSubsystem
. The Magic Leap API is neatly encapsulated, and as far as speech control for your app goes, it doesn’t matter whether it’s running on a HoloLens 2 or a Magic Leap 2. As a consequence,Augmedit Lumi now does support speech recognition on Magic Leap - without any code change for that.
When I started developing for Quest and Magic Leap 2 as well, I noticed no such file was available. When I was experimenting with getting Lumi - Augmedit’s flagship product for brain surgery - to run on the Magic Leap 2, there were some things that only occasionally went wrong, and only at run time. How helpful it would be if that app also would dump its log messages in a file, so I could check after the fact. Indeed. And so I wrote a little ServiceFramework service that does exactly that.
The service sports a profile that allows you to set a few properties:
This all can be conveniently set via the inspector.
The service sports only two methods, that you don’t actually even need to use - provided you set the AutoStart profile property to true, but hey, you can do it all yourself if you want
public interface IFileLoggerService : IService
{
public void StartLogging();
public void StopLogging();
}
It’s a bog standard ServiceFramework Service. Messages are dumped in a queue whenever they arrive, and written in the log file one by one by the Update
method, to prevent overloading the system with write actions - and keeping the messages in order.
public class FileLoggerService : BaseServiceWithConstructor, IFileLoggerService
{
private readonly string nl = Environment.NewLine;
private readonly FileLoggerServiceProfile serviceProfile;
private StreamWriter currentLogFile = null;
private readonly Queue<string> logMessages = new();
public FileLoggerService(string name, uint priority,
FileLoggerServiceProfile profile)
: base(name, priority)
{
serviceProfile = profile;
}
}
nl
is just shorthand for Environment.NewLine
which we will see back later. Next up is the basic startup/shutdown logic that really doesn’t show much surprises:
public override void Start()
{
if (serviceProfile.AutoStart)
{
StartLogging();
}
}
public override void Destroy()
{
StopLogging();
base.Destroy();
}
public void StartLogging()
{
if (currentLogFile == null)
{
Application.logMessageReceivedThreaded += LogListener;
}
}
public void StopLogging()
{
Application.logMessageReceivedThreaded -= LogListener;
if (currentLogFile != null)
{
currentLogFile.Close();
currentLogFile = null;
}
}
It also shows the crux of the whole story: the event Application.logMessageReceivedThreaded
is used to intercept everything Unity wants to log - this is a rather standard trick - and pass it to the LogListener
method. This method formats any log message that is not filtered either by phrase or type and dumps the result in the logMessages
queue…
private void LogListener(string logString, string stacktrace, LogType lType)
{
if ( ShouldLog(logString, lType))
{
logMessages.Enqueue(GetLogString(logString, stacktrace, lType));
}
}
private string GetLogString(string message, string stacktrace, LogType lType)
{
var timeStamp = DateTimeOffset.UtcNow.ToString("yyyy-MM-dd HH:mm:ss.fff");
var trace = !string.IsNullOrEmpty(stacktrace) ?
$"StackTrace: {stacktrace}{nl}" : string.Empty;
return
$"Time: {timeStamp}{nl}Log: {lType}{nl}Msg: {message}{nl}{trace}====={nl}";
}
… and like I wrote before, those log messages are written in the log file by the Update
method, that is called once every frame by the Service Framework.
public override void Update()
{
base.Update();
if (logMessages.Any())
{
LogFile.WriteLine(logMessages.Dequeue());
}
}
The LogFile
property creates an auto flushing StreamWriter
or returns the existing one for append:
private StreamWriter LogFile
{
get
{
if (currentLogFile == null)
{
var logFilePath = Path.Combine(Application.persistentDataPath,
$"{serviceProfile.LogFilePrefix}{DateTimeOffset.UtcNow:yyyyMMddHHmmss}.log");
currentLogFile = new StreamWriter(
new FileStream(logFilePath, FileMode.Append, FileAccess.Write));
currentLogFile.AutoFlush = true;
}
return currentLogFile;
}
}
Cleaning up the old log files, as we saw being called from the Initialize
method, is simply done thus:
private void DeleteOldLogs()
{
foreach (var file in GetOldLogFilesToDelete())
{
try
{
File.Delete(file);
}
catch (Exception e)
{
Debug.LogError($"Log {file} could not be deleted: {e}");
}
}
}
private IEnumerable<string> GetOldLogFilesToDelete()
{
return Directory.GetFiles(Application.persistentDataPath,
$"{serviceProfile.LogFilePrefix}*.log").Where(file =>
Math.Abs((DateTimeOffset.Now - File.GetCreationTime(file)).Days) >
serviceProfile.RetainDays);
}
All files older than RetainDays
are deleted. Or at least, we try to - and if that fails, it will also be logged in the current log file. How meta ;)
The only thing left then is the filtering, in which we see some simple keyword filtering and some very funky bit shifting stuff.
private bool ShouldLog(string logString, LogType lType)
{
var logTypeFlagInt = 1 << (int)lType;
return ((int)serviceProfile.LogTypes & logTypeFlagInt) != 0 &&
!serviceProfile.FilterPhrases.Any(logString.Contains);
}
So what is the deal with that? Well, Unity has defined the log levels as such in an Enum:
namespace UnityEngine
{
public enum LogType
{
Error,
Assert,
Warning,
Log,
Exception,
}
}
The issue with that is that if you define a property of type LogType
, you can only select one value at a time in the inspector, so you could trap only one type of log to trap. While what I want was this: being able to choose any combination of log types.
So I defined my own Enum
[Flags]
public enum LogTypeFlags
{
Error = 1,
Assert = 1 << 1,
Warning = 1 << 2,
Log = 1 << 3,
Exception = 1 << 4
}
And using the fact that an Enum can be cast to an int, and bit shifting a 1 by the int value of LogType
, I can compare my own Enum with the Unity one. This, of course, entirely depends on the order of the Unity Enum and the assumption (or hope, whatever you want to call it) they will never change this.
If you connect your Magic Leap 2, Quest or other android device, you can now simply find logs. Every session gets its own log file. On Magic Leap 2, you can find it at
Internal shared storage\Android\data\[your.package.name.com]\files
I have created a little demo project that shows how it works, using an extremely lame piece of demo code:
public class DemoLogger : MonoBehaviour
{
private int logFrameCount;
void Update()
{
if (logFrameCount++ % 240 == 0)
{
Debug.Log($"DemoLogger Update {logFrameCount}");
}
}
}
It basically dumps a debug message in the log every 240 frames (which is about 4 seconds). This must be about the least visually compelling demo I ever created, as it shows absolutely nothing. The only effect is the appearance of log files ;). After a run, you will see log files appear on the device, in this case Magic Leap 2:
This service is very useful in finding those weird ‘Heisenbugs’ that sometimes occur, and actually - it’s useful on HoloLens 2 as well, because how often does a user restart the app after something has gone wrong - overwriting the log file immediately. Of course, you can also use Application Insights, AppCenter logging or some other remote logging service - but this always works, connection or not, is simple to use, and has very little impact. Be aware, however, that excessive logging can also have an impact on performance.
Anyway, as indicated before, all code involved can be found here.
]]>So I took a look at the repo from my blog post about this from June 2022 - that was still on some old MRTK3 Pre-release - and first updated it. Then I added the requested functionality. The way to do it turns out to be very simple. First, you have to get a reference to a hands aggregator subsystem:
handsAggregatorSubsystem =
XRSubsystemHelpers.GetFirstRunningSubsystem<IHandsAggregatorSubsystem>();
And then you get the actual pinch position like this:
private Vector3 GetPinchPosition(ArticulatedHandController handController)
{
return handsAggregatorSubsystem.TryGetPinchingPoint(handController.HandNode,
out var jointPose)
? jointPose.Position
: Vector3.zero;
}
The only piece of the puzzle you then need is finding the ArticulatedHandController
objects for left and right hand, which you can do using my updated MRTK3ConfigurationFindingService
.
All things combined, you can use it like this:
findingService =
ServiceManager.Instance.GetService<IMRTK3ConfigurationFindingService>();
findingService.LeftHandStatusTriggered.AddListener(t=>
{
leftHand.SetActive(t);
if (t)
{
textMesh.text = $"Left hand position: {GetPinchPosition(findingService.LeftHand)}";
}
});
findingService.RightHandStatusTriggered.AddListener(t=>
{
rightHand.SetActive(t);
if (t)
{
textMesh.text = $"Right hand position: {GetPinchPosition(findingService.RightHand)}";
}
});
It still works like before, on HoloLens 2 and Quest, but it now also shows the position on which you actually click, and with what hand, in text.
You can find the updated code and new functions in this repo, branch crossplatairtap-position
]]>However, this was … quite a venture.
When I wrote about using the Spatial Map on Magic Leap 2, I was surprised to learn Magic Leap hadn’t implemented ARMeshManager. However, using a simple behaviour filled that void. That was nothing compared to what I ran into when I wanted to do something I assumed to be easy: capturing the camera image. After all, I needed it to feed the model. On HoloLens 2, all you need to do to get a webcam camera view is this:
var webCamTexture = new WebCamTexture(requestCameraSize.x, requestCameraSize.y,
cameraFPS);
webCamTexture.Play();
When I tried this on Magic Leap 2, nothing happened. I could actually retrieve cameras and camera sizes using the WebCamTexture
API, but I did not get any image. After conferring with a Magic Leap engineer, I got more or less the same message as with the Spatial Map: this part of the Unity stuff is not implemented (yet), so I also had to use Magic Leap specific code here as well. He was kind enough to point me to the “Simple Camera Example” in the Magic Leap 2 developer docs. The name is a bit misleading because there are actually two samples there. I used the simplest one of the two. That ‘Simple Camera Example’ is 300 lines long.
I repeat: that ‘Simple Camera Example’ is 300 lines long.
It would also require me to completely rewrite the logic of the app. This, of course, would not do.
I am not a software architect just for the showy name, so I put on my ‘architect hat’ and called the Reality Collective Service Framework to the rescue. If there are two or more very different APIs aiming to basically achieve the same goal, I try to define a service handling those differences. Instead of relying on either WebCamTexture or the Magic Leap camera API, I made myself an Image Acquiring Service, implementing one interface, with two implementations: one for HoloLens, and one for Magic Leap 2.
The interface is hilariously simple:
public interface IImageAcquiringService : IService
{
void Initialize(Vector2Int requestedImageSize);
Vector2Int ActualCameraSize { get; }
Task<Texture2D> GetImage();
}
The main YoloObjectLabeler
behaviour first held all the settings and did the image processing: now the settings have all moved to service profiles and it just gets a reference to an IImageAcquiringService
, sends it the Yolo model image size, loads the actual provided camera size, and then repeatedly calls GetImage()
to get the latest image.
To be able to give every implementation its own settings, I have defined a profile with the following settings:
public class ImageAcquiringServiceProfile : BaseServiceProfile<IServiceModule>
{
[SerializeField]
private int cameraFPS = 4;
[SerializeField]
private Vector2Int requestedCameraSize = new(896, 504);
public int CameraFPS => cameraFPS;
public Vector2Int RequestedCameraSize => requestedCameraSize;
}
The FPS is only functional on the HoloLens 2 (or any other platform that implements WebCamTexture
). The default value shown here is for HoloLens 2 - Magic Leap 2 actually provides lots more different camera sizes, and I have set it to 640x480, as this is closest to the 320x256 the Yolo V8 model in the app uses.
Although this is about running it on Magic Leap 2, I wanted to show you how a WebCamTexture using implementation looks like, if only to show how simple and clear it is:
public class ImageAcquiringService : BaseServiceWithConstructor, IImageAcquiringService
{
private readonly ImageAcquiringServiceProfile profile;
private WebCamTexture webCamTexture;
private RenderTexture renderTexture;
public ImageAcquiringService(string name, uint priority,
ImageAcquiringServiceProfile profile)
: base(name, priority)
{
this.profile = profile;
}
public void Initialize(Vector2Int requestedImageSize)
{
renderTexture = new RenderTexture(requestedImageSize.x, requestedImageSize.y,
24);
webCamTexture.Play();
}
public override void Start()
{
webCamTexture = new WebCamTexture(profile.RequestedCameraSize.x,
profile.RequestedCameraSize.y, profile.CameraFPS);
ActualCameraSize = new Vector2Int(webCamTexture.width, webCamTexture.height);
}
public Vector2Int ActualCameraSize { get; private set; }
public async Task<Texture2D> GetImage()
{
if (renderTexture == null)
{
return null;
}
Graphics.Blit(webCamTexture, renderTexture);
await Task.Delay(32);
var texture = renderTexture.ToTexture2D();
return texture;
}
}
Start
is called by the ServiceFramework, creates the WebCamTexture
with the requested size, then retrieves the actual size (this might differ - I can ask HoloLens for 640x480 but I won’t get it - it will default to the closest camera size).
Initialize
sets the RenderTexture
’s initial size (320x256 for this model) and starts generating images.
And then, by calling GetImage()
I can get the latest image from the WebCamTexture. Mind you, this generates a new Texture2D
. It’s the caller’s responsibility to destroy that after it’s done with it (this already was the case in the previous sample, but then it all happened in one class)
I will limit myself to a few snippets, as the complete service is 267 lines long. I have basically taken the “Simple Camera Sample”, removed a lot of the global variables and changed them into parameters. At the top of the service, it says this:
public void Initialize(Vector2Int requestedImageSize)
{
this.requestedImageSize = requestedImageSize;
}
public override void Start()
{
StartCameraCapture();
}
Basically the same as the HoloLens 2 implementation, only it now fires off a lot of Magic Leap specific code, instead of a WebCamTexture.Play()
. This kicks off a whole batch of things. As far as I can see it:
OnCaptureRawVideoFrameAvailable
to the OnRawVideoFrameAvailable
OnCaptureRawVideoFrameAvailable
then calls UpdateRGBTexture
Texture2D
“videoTexture
”, which is about the only global variable left. It also sets the ActualCameraSize
(this is available only after the first frame).The GetImage implementation for Magic Leap 2 remarkably looks like that of the HoloLens 2:
public async Task<Texture2D> GetImage()
{
if (videoTexture == null)
{
return null;
}
if (renderTexture == null)
{
renderTexture = new RenderTexture(imageSize.x, imageSize.y, 24);
}
Graphics.Blit(videoTexture, renderTexture);
await Task.Delay(32);
return FlipTextureVertically(renderTexture.ToTexture2D());
}
… but for one crucial detail: the image is flipped upside down.
So there’s this final method that actually takes care of that by flipping it. I nicked it off StackOverflow, of course
private static Texture2D FlipTextureVertically(Texture2D original)
{
var originalPixels = original.GetPixels();
var newPixels = new Color[originalPixels.Length];
var width = original.width;
var rows = original.height;
for (var x = 0; x < width; x++)
{
for (var y = 0; y < rows; y++)
{
newPixels[x + y * width] = originalPixels[x + (rows - y - 1) * width];
}
}
original.SetPixels(newPixels);
original.Apply();
return original;
}
Apart from the refactoring of the image aquistion into a service, the app itself is largely unchanged. So how does it work in real life on the Magic Leap 2? Well, it recognizes the airplanes, more or less in the right place, just like HoloLens 2. It also recognizes the DC3 Dakota, even though I never added a picture of that to the training set - but I guess this is more a Unity Barracuda accomplishment than a Magic Leap 2 one. As performance goes on this one, I would say, about on par with HoloLens 2. This surprised me, as I expected it to be faster, having a more beefy processor, but it actually seems to be just that bit slower and have a lower recognition rate than HoloLens 2.
Don’t get me wrong - there’s definitely room for improvement here - in my code, that is. I will not even begin to pretend I understand this device as well as I do HoloLens 2. I notice the image I get is a bit grainier than HoloLens 2 delivers. Also, the FOV of the webcam seems to be bigger, which gives some more distortion at the edges - in the sense that the 3D object and the image don’t exactly overlap anymore, and therefore my rather crude approach to locating objects in 3D space based upon the location in a 2D image doesn’t work that well anymore.
I am excited to see Magic Leap 2 can do computer vision in the way as well, which in the light of the current AI wave is a very important feature. In this regard, I think it would be nice if the Magic Leap SDK would implement the WebCamTexture as well. However, since this is a bit of an unusual use case, I can also imagine this having not the highest priority.
A branch of the YoloHolo for Magic Leap 2 can be downloaded here
]]>