What is the idea?

The Spectacles themselves have no user authentication that we, as lens developers, can piggyback on. So if we want any kind of “prove it’s really you” before using the stored token, we have to do it ourselves inside the lens. The simplest thing that could possibly work is, of course, a PIN code: the user picks one the first time they log in, we use it as an encryption key for the token, and from that point on the lens can only recover the token if the user types the PIN again.

The nice side effect is that the token on disk is no longer readable at all without the PIN, which should calm your Compliance Officer down a bit (although as I wrote last time, I still don’t think anything but the lens itself can read that storage anyway, but let’s not argue with Compliance Officers).

So this is what I built. After you have authenticated, you get asked to enter your pincode twice, then you login:

And when you start the lens for the second ir later time time you can login with just your pin.

That PIN is actually used in the encryption. No PIN, no decryption, no token, no service calls. And you can even use the Spectacles app on your phone to type in the pin, which usually is easier than using a floating keyboard.

The only new interface you need to know about

The whole story really boils down to one new service interface:

export interface IPincodeRequestService {
    askForPinCode(requireConfirm: boolean): Promise<string>;
}

export function IPincodeRequestService() {}

This is the contract between the token store (which needs a PIN to do its job) and whatever part of your app happens to know how to talk to the user. The requireConfirm flag is the difference between “pick a new PIN” (ask twice, make sure the user entered the same thing both times) and “enter your existing PIN” (ask once).

The empty function at the bottom is the service token I wrote about in my service-driven development article.

The new token store

With that interface in hand, the new EncryptedPersistentStorageTokenStore is pretty readable. Here is the main part:

import AESEncryptionHandler from "LocalJoost/Security/AESEncryptionHandler";
import { AccessToken } from "./AccessToken";
import { ITokenStore } from "./ITokenStore";
import { IPincodeRequestService } from "./IPincodeRequestService";
import { ServiceManager } from "../ServiceManager";

export class EncryptedPersistentStorageTokenStore implements ITokenStore {
    private store: GeneralDataStore = global.persistentStorageSystem.store;
    private readonly tokenKey: string = "encrypted_access_token";
    private pinCode = ""
    private pinCodeRequestServiceCache: IPincodeRequestService;
    private readonly maxRetries : number = 3;

    public async getToken(): Promise<AccessToken | null> {
        const token = await this.getEncryptedTokenFromStore();
        if (token) {
            return AccessToken.fromJSON(JSON.parse(token));
        }
        return null;
    }

    public async setToken(token: AccessToken): Promise<void> {
        if(!this.pinCode) {
            this.pinCode = await this.pinCodeRequestService.askForPinCode(true);
        }
        var encryptedToken = AESEncryptionHandler.encryptWithPin(JSON.stringify(token), this.pinCode);
        this.store.putString(this.tokenKey, encryptedToken);
    }

    public async clearToken(): Promise<void> {
        this.store.remove(this.tokenKey);
        this.pinCode = "";
    }
    ...
}

A few things to notice:

• The PIN is kept in memory after the first successful entry. So you only get prompted once per lens session, not on every token refresh. This prevents the decryption having to happen every time the PIN is asked. On clearToken (that is, on logout) we wipe it again.
• setToken with requireConfirm: true - when a token is being stored, we’re either doing it for the first time ever, or the user has just logged in again after a logout. Either way they need to set a PIN, not recall one, so we make them confirm it.
• getToken with requireConfirm: false - the user is trying to unlock an existing token, so asking once is enough.
• The pinCodeRequestServiceCache is just a lazy lookup into the ServiceManager. This is because the IPincodeRequestService is typically registered by some UI component, and the token store may already exist before that UI is ready. Looking it up on first use avoids initialization-order headaches.

The actual encryption happens in AESEncryptionHandler.encryptWithPin. More about that in a minute. First, the other half of getEncryptedTokenFromStore:

private async getEncryptedTokenFromStore(): Promise<string> {
    const encryptedToken = this.store.getString(this.tokenKey);
    if (!encryptedToken) {
        return null;
    }
    var retries: number = this.maxRetries + 1;

    var unencryptedCode : string
    while (!unencryptedCode && retries > 0) {
        if(!this.pinCode) {
            this.pinCode = await this.pinCodeRequestService.askForPinCode(false);
        }
        unencryptedCode = AESEncryptionHandler.decryptWithPin(encryptedToken, this.pinCode);
        if(!unencryptedCode) {
            this.pinCode = "";
        }
        retries--;
        if( retries === 0) {
            this.clearToken();
            throw new Error("Maximum retries reached. Please re-authenticate the device.");
        }
    }
    return unencryptedCode;
}

The logic is:

• If there’s no token in storage, there’s nothing to decrypt. Return null, and the service will fall through to a fresh device code flow.
• If there is a token, ask for the PIN and try to decrypt.
• Wrong PIN? decryptWithPin returns null (more on that in a minute), we throw away the cached PIN, and ask again.
• After three wrong PINs, give up, wipe the encrypted token from storage, and throw. The user is forced to do a fresh device code flow, which is exactly what you want: a thief with a stolen pair of Spectacles doesn’t get unlimited guesses.

The encryption itself

Spectacles doesn’t ship with a crypto API, so I dropped in a slighty modified version of crypto-js as a vendored library under LocalJoost/Security/crypto-js. Around it sits AESEncryptionHandler, which does two things: plain AES-CBC (given a key and IV), and a PIN-based variant that derives the key from the PIN.

The PIN-based variant is what the token store actually uses:

public static encryptWithPin(plainText: string, pin: string): string {
    try {
        const salt = CryptoJS.lib.WordArray.random(16);
        const iv = CryptoJS.lib.WordArray.random(16);

        const key = CryptoJS.PBKDF2(pin, salt, {
            keySize: 256 / 32,
            iterations: 1000
        });

        const ivHex = iv.toString();
        const encryptedCiphertext = AESEncryptionHandler.encrypt(
            plainText,
            "hex:" + key.toString(),
            "hex:" + ivHex
        );
        if (!encryptedCiphertext) {
            return null;
        }

        // Format: salt (32 hex chars) + iv (32 hex chars) + ciphertext (hex)
        return salt.toString() + ivHex + encryptedCiphertext;
    } catch (error) {
        print("Encryption error: " + error);
        return null;
    }
}

For the non-cryptographers: a PIN code like “123456” is way too short and predictable to use as an encryption key directly. PBKDF2 takes the PIN and a random salt, chews on them for 1000 iterations, and spits out a proper 256-bit key. The salt is different every time you encrypt, which is why you can’t just store key = f(pin) somewhere and be done with it - you need the salt alongside the ciphertext to reconstruct the key later.

The IV (initialization vector) is also random per encryption and also needs to be stored alongside. So the output format is simply salt_hex + iv_hex + ciphertext_hex, all glued into one string. Decryption reverses it:

public static decryptWithPin(encryptedText: string, pin: string): string {
    try {
        // Extract salt (first 32 hex chars = 16 bytes), IV (next 32), and ciphertext (rest)
        const salt = CryptoJS.enc.Hex.parse(encryptedText.substring(0, 32));
        const ivHex = encryptedText.substring(32, 64);
        const ciphertextHex = encryptedText.substring(64);

        const key = CryptoJS.PBKDF2(pin, salt, {
            keySize: 256 / 32,
            iterations: 1000
        });

        return AESEncryptionHandler.decrypt(
            ciphertextHex,
            "hex:" + key.toString(),
            "hex:" + ivHex
        );
    } catch (error) {
        print("Decryption error: " + error);
        return null;
    }
}

Chop the stored string into three pieces, re-derive the key from the PIN and the salt, and ask AES to decrypt. If the PIN was wrong, CBC with PKCS7 padding will typically fail the padding check and crypto-js will throw - which is why the whole thing is wrapped in a try/catch that returns null on failure. That’s what the retry loop in the token store keys off of.

There’s also a test() method in there that does a round-trip with "123456" and verifies a wrong PIN returns null. Handy for when you’re poking at it in the editor and wondering why nothing works - usually it’s because you forgot to actually register the service.

A little ripple effect on the interface

Making the token store wait for user input means ITokenStore itself had to become async. It used to be:

interface ITokenStore {
    getToken(): AccessToken | null;
    setToken(token: AccessToken): void;
    clearToken(): void;
}

Now it’s:

interface ITokenStore {
    getToken(): Promise<AccessToken | null>
    setToken(token: AccessToken): Promise<void>;
    clearToken(): Promise<void>;
}

Which means EntraDeviceCodeFlowAuthenticationService.authenticate needs a couple of awaits sprinkled in, and, more importantly, a try/catch around the initial getToken, because that’s the call that can now throw the “maximum retries reached” error I just described:

public async authenticate(): Promise<AccessToken> {
    var currentToken: AccessToken | null = null;
    try{
        currentToken = await this.tokenStore.getToken();
    } catch (error) {
        var errorMessage = error instanceof Error ? error.message : String(error);
        this.trace(errorMessage);
        this.UserActionRequiredEvent.invoke(errorMessage);
        await this.awaitableSleep.sleep(3);
    }
    ...
}

If the user fails the PIN three times, we show them the error via UserActionRequiredEvent, wait three seconds so they can actually read it, and then fall through to the normal “no valid token, start device code flow” path. Which is the correct behaviour: you got locked out, so go log in again.

The old PersistentStorageTokenStore is still there, by the way. It just got its method signatures updated to async/Promise. If you don’t want PIN protection you can still use it - just revert the line in EntraBootstrapper that we’ll look at next.

Wiring it all up

The bootstrapper is almost the same as before. The one-line difference:

var entraService = new EntraDeviceCodeFlowAuthenticationService(
    this.tenantId, this.clientId, new EncryptedPersistentStorageTokenStore(), true);

EncryptedPersistentStorageTokenStore instead of PersistentStorageTokenStore. That’s it. Everything else, the tenant ID, the client ID, the service registration, all unchanged.

The other bit of wiring is on the UI side. Something has to actually implement IPincodeRequestService and pop up a keyboard when the token store calls askForPinCode. For the demo I let EntraSampleUIWindow do double duty: it’s both the thing that shows the welcome messages and the thing that asks for the PIN. In a real app you would probably make a dedicated component, but for a sample this keeps the moving parts to a minimum.

In the scene, I added a TextInputField (one of the new UIKit components) to the prefab and hooked it up to a new @input:

@input private pincodeInput: TextInputField;

In onAwake we register ourselves as the implementation:

var serviceManager = ServiceManager.getInstance();
serviceManager.register(IPincodeRequestService, this);
this.entraService = serviceManager.get(IEntraDeviceCodeFlowAuthenticationService);

And then the actual implementation of the interface method:

public async askForPinCode(requireConfirm: boolean): Promise<string> {
    var validPinCodeObtained = false;
    this.pincodeInput.sceneObject.enabled = true;
    var pinCode: string = "";
    do {
        this.showMessage(requireConfirm ? "Please enter a PIN code to encrypt your token:" : "Please enter your PIN code to decrypt your token:");
        pinCode = await this.waitForInput();
        if (!requireConfirm) {
            validPinCodeObtained = true;
        }
        else {
            await this.awaitableSleep.sleep(0.25);
            this.showMessage("Please confirm your PIN code by entering it again:");
            var confirmPinCode = await this.waitForInput();
            if (pinCode !== confirmPinCode) {
                this.showMessage("PIN codes do not match. Please try again.");
                await this.awaitableSleep.sleep(3);
            } else {
                validPinCodeObtained = true;
            }
        }
    }
    while (!validPinCodeObtained)
    await this.awaitableSleep.sleep(2);
    this.pincodeInput.sceneObject.enabled = false;
    return pinCode;
}

Enable the input field, show the right prompt, wait for the user, and if we’re in “pick a new PIN” mode, ask again and loop until the two entries match. When we’re done, hide the input field so it doesn’t clutter the rest of the flow. Note again the reappearance of the trusty AwaitableSleep.

The only mildly interesting bit is waitForInput:

private async waitForInput(): Promise<string> {
    this.pincodeInput.text = "";
    this.enterTestPinCode();
    return new Promise((resolve) => {
        const handler = () => {
            const input = this.pincodeInput.text.trim();
            if (input.length >= 6) {
                this.pincodeInput.onTextChanged.remove(handler);
                global.textInputSystem.dismissKeyboard();
                resolve(input);
            }
        };
        this.pincodeInput.onTextChanged.add(handler);
    });
}

The UIKit TextInputField fires onTextChanged on every keystroke. We wait until there are at least 6 characters, then call it done, remove the handler, dismiss the keyboard, and resolve the promise. No explicit “submit” button needed.

And finally a confession - typing a PIN on the virtual keyboard in the Lens Studio editor is annoying, so I cheated:

private enterTestPinCode() : void {
    if (!global.deviceInfoSystem.isEditor()) return;
    const delayedEvent = this.createEvent("DelayedCallbackEvent");
    delayedEvent.bind(async () => {
        this.pincodeInput.text = "123456";
    });
    delayedEvent.reset(2);
}

Two seconds after the prompt appears, if we’re running in the editor, the field is auto-filled with 123456. On the actual device this does nothing. Small quality-of-life hack that will save you a lot of typing while developing.

Trying it out

Run the lens once, go through the device code flow on your computer as before. After you log in, you’ll get prompted to pick a PIN. Enter it twice. The token is now encrypted in storage.

Stop the lens. Start it again. You’ll get asked for your PIN. Enter it, and you’ll get the welcome message without having to go through device code flow again. Enter the wrong PIN three times and it will kick you out and restart the device code flow.

In the editor you’ll actually see things happening in the debug console if you leave verbose on, which is kind of fun. On device it all just works.

One last caveat

I want to be very clear: this is a solution, not the solution. A 6-digit PIN encrypted with PBKDF2-1000 is not going to hold up against a determined attacker with the encrypted blob in hand. 1000 iterations is on the low side by 2026 standards and a six-digit numerical PIN has only a million possible values. Also, I think there needs to be protection against stupidly simple PIN codes like 000000, 111111 and the 123456 I showed in the demo because you know people are going to pick those stupid codes if you let them. But on Spectacles, as I wrote last time, there is (as far as I know) no way to get the encrypted blob out of the lens’ persistent storage in the first place, so brute-forcing it is not a realistic attack. What this does protect against is the much more mundane scenario: someone picks up your glasses, puts them on, and fires up the lens. Without the PIN, they get nothing, and your enterprise backend is not going to hand out data to a stranger. That’s a lot better than where we were last month.

If your threat model is scarier than that, you probably want a proper secure enclave, hardware attestation, and a team of people with ‘CISSP’ on their LinkedIn. I wish you all the best.

Code, as always, can be found at GitHub - in the pincode-protected branch.

Entra authentication, the general idea

For normal authentication, the flow is usually:

1. You or your app pops up a window or a browser that requires you to enter credentials. This can be either username and password (followed by some 2FA prompt on your phone - you have set that up right?), a passkey, or a Windows Hello recognition.
2. If satisfactory credentials are supplied, you get a package of data that includes your name, ID (usually email address), and most importantly: an access token and a refresh token.
3. The access token can be used as a bearer token to access secure resources, for instance some API.
4. The refresh token can be used to get a new access token when the old one expires.

Device Code Flow

This all works fine on a computer or a phone, but on Spectacles we don’t have popup browser windows, passkeys, or Windows Hello, and although the virtual keyboard is pretty decent and you can type in usernames and passwords using your phone as well, anyone who has done so repeatedly (like me) can tell you it’s a pretty tedious PITA to do it that way. The ETF OAuth working group, who created standards around OAuth, fortunately saw this as well and defined something: OAuth 2.0 Device Authorization Grant. It’s essentially a way to do authentication for what they call so beautifully an ‘input-restricted device’. The Microsoft implementation is commonly referred to as Device Code Flow. This works as follows:

1. The device calls a particular API provided by the identity authority (in this case by Microsoft).
2. The API returns a data structure containing, amongst other things, a code and a verification URL.
3. The device instructs the user to go to a particular website (in Microsoft’s case this is usually https://login.microsoft.com/device) on any device they see fit. I usually just take a browser on a computer, as typing on a keyboard while wearing a see-through device is perfectly possible.
4. You log in normally there, using whatever authentication you or your admin has set up.
5. The device, in the meantime, has been polling the verification URL every few seconds. While the user has not logged in on the computer, it gets a 400 error and should retry after a couple of seconds. As soon as the user has logged in, it gets a 200 status plus a data structure containing user data as well as an access token, an expiration date, and a refresh token. The access token you can add as a normal bearer token to your HTTP request to a secured resource; the refresh token can be used to get a new token when the old one expires, as I wrote before.

The login procedure looks like this:

This kind of login is used for Xbox for instance, but also for things like streaming services, as no one ever liked entering passwords with a TV remote. Because it keeps your token and refresh token, you (fortunately) don’t have to log in every time you use those devices. I have implemented this for Spectacles and Entra using - of course - a service.

Basic usage

The service needs to be initialized with a tenant ID, a client ID, an ITokenStore implementation for token persistence (I will explain that later), and an option to get verbose logging. And believe me, verbose it is when you ask it to be so. Getting an Azure tenant ID, setting up a client in Azure to get an ID, and deploying a secured resource to test access with is outside of the scope of this article, but rest assured: if you ask one of the AI chatbots of today, it will guide you through the process pretty easily.

But enfin, the initialization, in the EntraBootstrapper component:

var serviceManager = ServiceManager.getInstance();
var entraService = new EntraDeviceCodeFlowAuthenticationService(
    this.tenantId, this.clientId, new PersistentStorageTokenStore(), true);
serviceManager.register(IEntraDeviceCodeFlowAuthenticationService, entraService);

Service bootstrapper I explained before in my original article about services in Lens Studio. If you look it up in the scene, you see two text boxes where you can configure the Azure tenant ID and your Azure client app ID.

The service has a very simple interface:

interface IEntraDeviceCodeFlowAuthenticationService {
    authenticate(): Promise<AccessToken>;
    UserActionRequiredEvent: Event<string>;
    logout(): void
}

You get yourself a reference to the service:

this.entraService = ServiceManager.getInstance().get(IEntraDeviceCodeFlowAuthenticationService);

You subscribe to the service’s UserActionRequiredEvent, and when it’s triggered, call a method that shows a message to the user indicating what to do or what happened:

this.entraService.UserActionRequiredEvent.add((message: string) => {
    this.showMessage(message);
});

Call authenticate and wait for a token to appear - or an error to occur:

try {
    var token: AccessToken = await this.entraService.authenticate();
    this.showMessage(`Welcome ${token.userName}`);
} catch (e) {
    this.showMessage(`Authentication failed: ${e}`);
    return;
}

And then you call a test URL:

// this needs to be in the declaration part of your calling class
private readonly http: InternetModule = require("LensStudio:InternetModule");

const httpRequest: RemoteServiceHttpRequest = RemoteServiceHttpRequest.create();
httpRequest.method = RemoteServiceHttpRequest.HttpRequestMethod.Get;
httpRequest.url = this.testApiUrl;
httpRequest.setHeader("Authorization", `Bearer ${token.accessToken}`);
this.http.performHttpRequest(httpRequest, (resp) => {
    this.showMessage(`Test URL status code: ${resp.statusCode}`);
});

Finally, if you want the app to forget about you, simply call the service’s logout method.
And that’s about it. How it actually works is a little bit more complicated. Although the basic service is simple enough, there are some nasty hairy bits. It also seems simpler than it is because by now I am building on top of quite a number of nifty utility classes that make things a lot simpler. I won’t explain every little detail because that would take an even longer article (and this one is already absurdly long and will most likely only be read by LLM training scrapers in its entirety), but at least I will take you through the flow of the code. But be sure to follow up the code into its veins because there’s some handy stuff in here, other than the Entra authentication alone.

Start of the service

The top of the service class contains all the stuff we need, including some things Microsoft requires us to use and know. Don’t worry too much about the why - this is what you need and how it’s done. The constructor just takes the arguments we have seen, fills in some derivatives, and then waits for the user to start an authentication flow.

export class EntraDeviceCodeFlowAuthenticationService implements IEntraDeviceCodeFlowAuthenticationService {
    private static readonly OAUTH_PATH: string = "oauth2/v2.0";
    private static readonly ENTRA_HOST: string = "https://login.microsoftonline.com";
    private static readonly CONTENT_TYPE_FORM: string = "application/x-www-form-urlencoded";

    private readonly http: InternetModule = require("LensStudio:InternetModule");
    private readonly verbose: boolean;
    private readonly clientId: string;
    private readonly authority: string;
    private readonly scope: string;
    private active: boolean = false;
    private backoff: number = 5;
    private awaitableSleep: AwaitableSleep = new AwaitableSleep(); 
    private readonly tokenStore: ITokenStore;

    constructor(tenant: string, clientId: string, tokenStore: ITokenStore, verbose: boolean = false) {
        this.clientId = clientId;
        this.verbose = verbose;
        this.authority = `${EntraDeviceCodeFlowAuthenticationService.ENTRA_HOST}/${tenant}/${EntraDeviceCodeFlowAuthenticationService.OAUTH_PATH}`;
        this.scope = `api://${clientId}/user_impersonation offline_access`;
        this.tokenStore = tokenStore;
    }

Note, though, this.authority. This is the base URL where we are going to request device codes and tokens, using the service’s postAsync method.

A little aside

I would like to draw your attention to AwaitableSleep. Coming from Unity and C#, I really like to use something like Task.Delay to wait without having to deal with the TypeScript promises baloney, so I made something that works like this.

export class AwaitableSleep {
    waitToken: CancelToken | null = null;
    public sleep(durationSeconds: number): Promise<void> {
        return new Promise<void>((done) => {
            this.waitToken = setTimeout(() => done(), durationSeconds * 1000);
        });
    }

    public cancel(): void {
        if (this.waitToken !== null) {
            clearTimeout(this.waitToken);
            this.waitToken = null;
        }
    }
}

If I want to wait for 5 seconds, I can simply do

await this.awaitableSleep.sleep(5);

and cancel it as well, from another thread, if that needs be.

The heart of the matter: authentication

It looks simple enough:

public async authenticate(): Promise<AccessToken> {
    let currentToken = this.tokenStore.getToken();
    if (currentToken) {
        this.trace("Existing token found, checking expiration");
        if (!currentToken.isExpired) {
            this.trace("Using cached access token");
            return currentToken;
        }
        else {
            currentToken = await this.refreshToken(currentToken.refreshToken);
            if (currentToken) {
                this.tokenStore.setToken(currentToken);
                return currentToken;
            }
        }
    }
    this.trace("No valid token available, starting new device code flow authentication");
    const deviceCodeResponse = await this.requestDeviceCode();
    this.trace(`User code: ${deviceCodeResponse.user_code}`);
    this.UserActionRequiredEvent.invoke(`Please go to ${deviceCodeResponse.verification_uri} and enter code: ${deviceCodeResponse.user_code}`);
    currentToken = await this.pollForToken(deviceCodeResponse.device_code);
    this.tokenStore.setToken(currentToken);
    return currentToken;
}

• Does the token store have a token and is it not expired? We are done! Return the token.
• If there is a token in the store but it is expired, refresh it and return it (and store it for next time).
• If there was no token, get a device code.
• Start polling for a token.
• If we get a token, we store it, then return it.

Requesting a device code

The first time, there won’t be a token at all, of course. So we end up at the bottom of authenticate and have to get a device token first:

private async requestDeviceCode(): Promise<IDeviceCodeResponse> {
    const result: IHttpResult = await this.postAsync("devicecode", {
        client_id: this.clientId,
        scope: this.scope,
    });

    if (result.statusCode !== 200) {
        throw new TokenRequestError(
            `Device code request returned HTTP ${result.statusCode}`,
            result.body
        );
    }

    const parsed = JSON.parse(result.body) as IDeviceCodeResponse;
    this.trace(`Received user code: ${parsed.user_code}`);
    return parsed;
}

Curiously we do this by HTTP POST (while we are basically getting something, but this is apparently how it’s done). It returns a structure containing a device_code and a verification_code, which we need in the next step.

Polling for token

So hurray, we have a device code. The user has been informed to go somewhere and log in with this code and their Entra credentials. Now we have to poll to see if Microsoft is giving their blessing and will grace our code with an actual access token. This also happens via an HTTP POST:

private async pollForToken(deviceCode: string): Promise<AccessToken> {
    this.active = true;
    this.backoff = 5;

    const tokenParams: FormParams = {
        client_id: this.clientId,
        grant_type: "urn:ietf:params:oauth:grant-type:device_code",
        device_code: deviceCode,
    };

    while (this.active) {
        this.trace("Polling for token...");
        const result: IHttpResult = await this.postAsync("token", tokenParams);
        const payload = JSON.parse(result.body);

        if (result.statusCode === 200) {
            this.halt();
            var tokenResponse = payload as ITokenResponse;
            return AccessToken.create(tokenResponse);
        }

        const tokenError = payload as ITokenError;

        switch (tokenError.error) {
            case "authorization_pending":
                this.trace("Waiting for user to authorize...");
                break;
            case "slow_down":
                this.backoff += 5;
                this.trace(`Increased polling interval to ${this.backoff}s`);
                break;
            default:
                this.handlePollingError(tokenError);
        }

        await this.awaitableSleep.sleep(this.backoff);
    }
    throw new TokenRequestError("Polling was cancelled");

I basically poll every five seconds, or slower when requested. If we actually get a result, it’s an ITokenResponse, that contains these fields:

export interface ITokenResponse {
  token_type: string
  scope: string
  expires_in: number
  access_token: string
  refresh_token?: string
  id_token?: string
}

You might notice, by the way, there’s never an implementation of ITokenResponse (or any other of the responses). Apparently this is possible in TypeScript and, as far as I understand, the ‘canonical way to do it’.

Creating a token from the token

Anyway, ITokenResponse’s access_token field contains some encoded data we want to know as well, like the user’s ID (usually the email address) and the actual name, so we can welcome them. Also, we want to know when the token expires.

import StringUtils from "LocalJoost/Utils/StringUtils";
import { ITokenResponse, ITokenData } from "./TokenResponses";

export class AccessToken {
    public accessToken: string;
    public refreshToken: string;
    public expiration: number;
    public userName: string;
    public oid: string;

    public static create(tokenResponse: ITokenResponse): AccessToken {
        const token = new AccessToken();
        token.accessToken = tokenResponse.access_token;
        token.refreshToken = tokenResponse.refresh_token;
        token.expiration = Date.now() / 1000 + tokenResponse.expires_in;
        var tokenData = this.getTokenDataFromAccessToken(tokenResponse.access_token);
        token.userName = tokenData.name;
        token.oid = tokenData.oid;
        return token;
    }

    public static fromJSON(json: object): AccessToken {
        const token = new AccessToken();
        Object.assign(token, json);
        return token;
    }

    public get isExpired(): boolean {
        return Date.now() / 1000 >= this.expiration;
    }

    private static getTokenDataFromAccessToken(accessToken: string): ITokenData {
        const splitToken: string[] = accessToken.split('.');
        const correctedBase64: string = StringUtils.base64UrlToBase64String(splitToken[1]);
        const decoded: string = StringUtils.base64Decode(correctedBase64);
        const tokenData: ITokenData = JSON.parse(decoded);
        return tokenData;
    }
}

The whole prying apart happens in getTokenDataFromAccessToken and that goes into all kinds of Base64 decoding and taking into account the fact that JWT access tokens are not encoded in standard Base64 but slightly differently - this method transforms it into standard Base64 so it can be decoded. StringUtils contains some nice methods for encoding and decoding strings, which are also useful in other scenarios. The fromJSON method is used to be able to retrieve it from storage, more about that later.

Anyway, at the end of pollForToken we have a token we can use, but that might expire. This is where the refreshToken method comes in, which is fairly simple:

private async refreshToken(refreshToken: string): Promise<AccessToken | null> {
    this.trace("Refreshing access token");
    const result: IHttpResult = await this.postAsync("token", {
        client_id: this.clientId,
        grant_type: "refresh_token",
        refresh_token: refreshToken,
        scope: this.scope,
    });

    if (result.statusCode !== 200) {
        this.trace(`Token refresh failed with HTTP ${result.statusCode}`);
        return null;
    }

    this.trace("Access token refreshed successfully");
    const tokenResponse = JSON.parse(result.body) as ITokenResponse;
    return AccessToken.create(tokenResponse);
}

It posts a refresh token request, gets a new token back - with a new refresh token and everything. It’s like you re-login but without you needing to do something. The fun thing is, you can just keep calling authenticate every time you need to access the secured API, because it will automatically retrieve a stored token, prompt the user for credentials to create a token, or refresh it -all the song and dance around the actual authentication happens behind the curtain.

Some helper methods

To make communication with the Entra authentication API a bit easier, this postAsync method was created.

private postAsync(endpoint: string, params: FormParams): Promise<IHttpResult> {
    const httpRequest: RemoteServiceHttpRequest = RemoteServiceHttpRequest.create();
    httpRequest.method = RemoteServiceHttpRequest.HttpRequestMethod.Post;
    httpRequest.url = `${this.authority}/${endpoint}`;
    httpRequest.body = FormParamUtils.serializeForm(params);
    httpRequest.setHeader("Content-Type", EntraDeviceCodeFlowAuthenticationService.CONTENT_TYPE_FORM);

    return new Promise<IHttpResult>((resolve) => {
        this.http.performHttpRequest(httpRequest, (resp) => {
            resolve({ statusCode: resp.statusCode, body: resp.body });
        });
    });
}

This saves the calling code a lot of plumbing for creating three HTTP POST requests (device code, poll, and refresh) and the interpretation of the result. It also uses the FormParams type and a helper method to make it easier to build an HTTP body payload.

export type FormParams = Record<string, string>;

export class FormParamUtils {
    public static serializeForm(params: FormParams): string {
        const parts: string[] = [];
        for (const key of Object.keys(params)) {
            parts.push(`${key}=${encodeURIComponent(params[key])}`);
        }
        return parts.join("&");
    }
}

Persistence and storage

Of course, if you have to log in this way every time you start the lens, it kind of defies the purpose. So once you have logged in, there needs to be a form of persistence. This is where an ITokenStore comes into play. The EntraDeviceCodeFlowAuthenticationService simply defers storing and retrieving to an object that implements a very simple interface:

interface ITokenStore {
    getToken(): AccessToken | null;
    setToken(token: AccessToken): void;
    clearToken(): void;
}

To demonstrate the principle, I have created a very simple implementation that stores the AccessToken as JSON in Spectacles’ persistent storage system:

import { AccessToken } from "./AccessToken";
import { ITokenStore } from "./ITokenStore";

export class PersistentStorageTokenStore implements ITokenStore {
    private store: GeneralDataStore = global.persistentStorageSystem.store;
    private readonly tokenKey: string = "access_token";

    getToken(): AccessToken | null {
        const token = this.store.getString(this.tokenKey);
        if (token) {
            return AccessToken.fromJSON(JSON.parse(token));
        }
        return null;
    }

    setToken(token: AccessToken): void {
        this.store.putString(this.tokenKey, JSON.stringify(token));
    }

    clearToken(): void {
        this.store.remove(this.tokenKey);
    }
}

This works fine, but you have to take one important issue into consideration. As long as the refresh token is valid, the lens will log in with your credentials, regardless of who wears the device, until you log out. And Spectacles itself does not have login or authentication we can use to protect it. So if your Spectacles get nicked, the thief can use the lens on your behalf and access your data. This is no different than someone stealing your Xbox where you logged in to. For an actual enterprise deployment, it behooves to think a bit about securing the login in a simple way, like with a PIN code or something. And store the data encrypted, although I wonder how useful that is, because as far as I know, there is no way to access the data in a lens’ persistent storage by anything but the lens itself. However, your Compliance Officer might get very nervous about it. I might write a better ITokenStore fix in a follow-up post - this one is long enough as it goes.

Demo code

I have added an EntraSampleUIWindow prefab that is a very simple UI and some code to actually start the authentication, to prove that it works. It can be configured with a simple test API URL that proves that it can actually call the secured URL.

To show it all works, I have written the following demo code in the equally named EntraSampleUIWindow component that shows the following things happening in your lens:

The code looks like this:

private async demoAuthentication(): Promise<void> {
    try {
        this.entraService.UserActionRequiredEvent.add((message: string) => {
            this.showMessage(message);
        });
        var token: AccessToken = await this.entraService.authenticate();
        this.showMessage(`Welcome ${token.userName}`);
        await this.awaitableSleep.sleep(4);

        this.showMessage(`Token valid until ${new Date(token.expiration * 1000)}`);
        await this.awaitableSleep.sleep(4);

        const httpRequest: RemoteServiceHttpRequest = RemoteServiceHttpRequest.create();
        httpRequest.method = RemoteServiceHttpRequest.HttpRequestMethod.Get;
        httpRequest.url = this.testApiUrl;
        httpRequest.setHeader("Authorization", `Bearer ${token.accessToken}`);
        this.http.performHttpRequest(httpRequest, (resp) => {
            this.showMessage(`Test URL status code: ${resp.statusCode}`);
        });
        await this.awaitableSleep.sleep(4);

        // Intentionally expire the token to demonstrate refresh flow
        token.expiration = Date.now() / 1000;
        var newToken = await this.entraService.authenticate();
        this.showMessage(`Token refreshed. New expiration time: ${new Date(newToken.expiration * 1000)}`);
        await this.awaitableSleep.sleep(4);

        httpRequest.setHeader("Authorization", `Bearer ${newToken.accessToken}`);
        this.http.performHttpRequest(httpRequest, (resp) => {
            this.showMessage(`Test URL status code after refresh: ${resp.statusCode}`);
        });

        this.button.sceneObject.enabled = true;
    } catch (e) {
        this.showMessage(`Authentication failed: ${e}`);
        return;
    }
}

• First, it subscribes to the UserActionRequiredEvent.
• Then it waits for authentication, using the whole rigmarole of code we have just seen.
• It shows a welcome text with the user’s name that we have plucked out of the access token.
• 4 seconds later it tries to call the test API and shows the result.
• Then it intentionally messes up the token’s expiration date to force a refresh. If you have the debug on in the editor, you can actually see this working.
• It calls the test API again to check that still works.
• In the final step, it enables the logout button.

If you now stop the lens, and start it again (whether on Spectacles or on Lens Studio) you will not see the device flow prompt, but it will immediately welcome you, because it pulled the token from the persistent store. However, if you press the logout button, it will clear the token persistent store. The very simple code for that sits in the onAwake method of the EntraSampleUIWindow component:

this.button.sceneObject.enabled = false;
this.button.onTriggerDown.add(() => {
    this.entraService.logout();
    this.demoAuthentication();
});

It starts the authentication again after logging out, thus showing you the device code prompt again, proving you have indeed logged out.

Conclusion

Although it’s not quite the intended use case for Spectacles, I am pretty sure lots of enterprises are keeping a keen eye on whatever Snap is doing. As Snap founder Evan Spiegel indicated on stage at Lens Fest 2025 during Q&A: Snap regularly gets (and denies) requests for Spectacles bulk purchasing from companies, but he was not surprised by getting those requests, because “there are not very many compelling devices on the market, and most of them are probably in this room”. This was a very valid point, and it became even more valid recently: we already knew HoloLens 2 is being deprecated without a successor, but now also Magic Leap has thrown in the towel for Magic Leap 2. So this makes me wonder what will happen when the 2026 commercial version of Spectacles will come to market. I know it’s possible to integrate Spectacles with enterprise services - and so do you now.

“Spiegel” means “mirror” in Dutch. I wonder what future for XR Mr. Spiegel has seen in his (magic) mirror. We will learn soon enough later this year.

Code, as always, can be found at GitHub

Early HoloLens days, I learned an important lesson quickly: one of the scariest things you could do is upgrade your apps to a new Unity version - even just a point release could hose your app. As fellow Mixed Reality developer Olivér Vasvári noted in comments on my August blog about running MRTK3 apps on Apple Vision Pro, my solution (based on Unity 6000.0.49) was still working on 6000.0.53, but fell apart on 6000.0.62. Specifically, the XR camera view was ‘stuck’ to your head - that is, every virtual object was moving along with your head on the Vision Pro.

This kind of thing is unfortunately part of the life of a Unity Mixed Reality developer. So I set out to find if I could revive my solution. Spoiler - I could.

Starting point was my last Vision Pro MRTK port. I went for broke and did not just upgrade to the version where Oliver noticed the problem, but jumped to the newest Unity 6.3 version (which was 6000.3.2f1 at the point I wrote that).

Upgrade project to 6.3

Straightforward enough: open the solution with 6000.3.2f1. This apparently needs an URP upgrade:

Click OK, and then you will be greeted by another issue:

Click open Package Manager and you will notice my hacked 2.2.4 Vision OS XR Plugin is no longer accepted.

Uninstall it, and Unity will proceed to install the newest Vision Pro packages:

Upgrade MRTK3 (optional)

In my previous Vision Pro post I used MRTK3 4.0.0-pre.1, but in the meantime 4.0.0-pre.2 has been released. It would be a shame not to use all the good work done by Kurtis Eveleigh, the proud MRTK custodian. The MRTK Feature Tool has been abandoned, and never worked on the Mac anyway, so the only way to upgrade I can think of is to download the packages manually, and put them in the Packages/MixedReality folder. The quickest way to effectuate the upgrade is to open the manifest.json in a text editor, do a search & replace on pre.1, and replace it with pre.2. You can then, of course, remove the pre.1 tgz files.

Unity will then popup this warning:

This you can safely ignore.

Fix nothing visible

The app can now be deployed on the Vision Pro, but when you run it, you will most likely see nothing. That is because the cubes are created 1.6m above your head. To fix this, you will need to change the Y value of the Camera Offset in the MRTK XR Rig to 0.

This is normally not a problem, but apparently it is now. If it’s Unity that does not like this, the Vision Pro packages, or something else, I don’t know.

Fix view being stuck to camera

If you now run the app, the cubes will appear all right, but if you move your head, all the cubes move with your head instead of hanging in space after the initial spawn, as displayed on the gif a the start of this post. This is one of the most peculiar changes we need to make:

As you can see, I have disabled the original Tracked Pose Driver and added a second one. This one has some hard-coded Action definitions in stead of an Action References. I got this wisdom by studying the Vision OS template v 3.0.2. Why I didn’t use Action References, and made a new Input Actions asset, like the default MRTK Default Input Actions, tailored for Vision Pro? Believe me, I tried, but either I don’t understand those Input Action assets correctly, or it simply doesn’t work in the Unity 6.3/Vision Pro combo. However, this works. But this leads to a new problem.

Fix cubes appearing on floor

The cubes are now no longer stuck to your head, but instead of appearing in the view, they appear on the floor and even partially in the floor, with only one or two rows visible. The rest is invisible due to occlusion being applied to the Spatial Map. This issue, although seemingly very simple, took me quite some time to figure out. My best guess is: whatever layer Unity is putting on top of the Apple stuff - it apparently needs some time to initialize and figure out where the headset actually is before it properly sets the main camera’s transform position.

The simplest way to fix that was to change my own startup code in CubeManager from:

private void Start()
{
    audioSource = GetComponent<AudioSource>();
    CreateGrid();
}

to

private async Task Start()
{
    audioSource = GetComponent<AudioSource>();
    await Task.Delay(1000);
    CreateGrid();
}

Simply wait a second till the Unity Player get it’s act together. I don’t know if this is optimal, but for my demo app, it does the trick.

Fix hand menu

Now the only thing missing is the hand menu - that does not appear when you hold up your hand. The root cause is still the same: Vision Pro apparently does not track your palm, and that’s what the hand menu (and other things in the MRTK) apparently rely on. So I went back to the trick Guillaume Vauclin from EADS, France created - adapt the code of the VisionOSHandProvider.cs file so that it no longer reports the palm cannot be tracked, but returns the wrist Pose instead. There is only one thing - I could not get Unity to accept a hacked version of the Apple Vision OS XR Plugin with a changed file inside it. So I went back to an old trick I used in November 2022 for patching the MRTK2 - replace the file in the Library folder after Unity has unpacked all the library files.

You will find the fixed VisionOSHandProvider.cs in the Patches folder, together with a small shell script I created (actually I asked Claude to write it) that simply takes all C# files in the current folder, finds the location of each correspondingey named file in the Library folder, and replaces it:

#!/bin/bash

# Get the current directory
CURRENT_DIR="$(pwd)"
LIBRARY_DIR="$(pwd)/../Library"

# Check if Library directory exists
if [ ! -d "$LIBRARY_DIR" ]; then
    echo "Error: $LIBRARY_DIR directory not found"
    exit 1
fi

# Find all C# files in current directory
for cs_file in *.cs; do
    # Skip if no .cs files found
    if [ ! -f "$cs_file" ]; then
        echo "No C# files found in current directory"
        exit 0
    fi
    
    echo "Processing: $cs_file"
    
    # Find all matching files in Library and subdirectories
    while IFS= read -r target_file; do
        echo "  Replacing: $target_file"
        cp "$cs_file" "$target_file"
    done < <(find "$LIBRARY_DIR" -type f -name "$cs_file")
done

echo "Done!"

Make sure the project is opened by Unity at least once, wait until it is fully done loading and installing all packages. Then open the Patches folder in Terminal, and run the following commands:

chmod +x replace_files.sh
./replace_files.sh

The first command you will need to run only the first time. Also, remember to run this command in your CI build scripts. In my previous jobs, I wrote CI scripts that did the following steps when a patch like this was needed:

• Pull code
• Open in Unity
  • basically do nothing
  • quit
• Run patch script
• Open in Unity again
  • run tests
  • perform actual build
  • etc

Concluding words

It’s remarkable how flexible and malleable MRTK3 is - I have got it to run on all Mixed Reality headsets I managed to get my hands on, with minor tweaks. This makes for a very consistent experience and retains as much of your investments in Mixed Reality as possible. Which is an important thing, given the HoloLens 2 deprecation.

Updated QuestBouncer project for Apple Vision Pro can be found here.

Unfortunately, we also lost a bit. For instance, we lost sound effects. In addition, the PinchButton contained an Interactable, which allowed us to control the button’s behavior in quite some detail:

However, it turns out that the UIKit button scripts are quite busybodies. If you select the “Inspect Preview” button on your preview panel, you’ll see what’s actually going on behind the scenes:

Aha! In fact, most of what the three UIKit buttons do is create all the components you need to make a functioning button on the fly, including an Interactable, to save you a lot of work. This, now, we can use to our advantage!

As you can see on the MyButton prefab I made in the demo project for the scrollable menu, there is an UIKitButtonController script added below the RectangleButton. Not coincidentally, this has exactly the same properties as an Interactable.

This is a little script I created. Actually, most of it I just nicked from Interactable itself: all the properties and UI widget declarations. The only part I added was this:

onAwake() {
    this.tryGetInteractable();
}

private tryGetInteractable(attempts: number = 0): void {
    const interactable = this.getSceneObject().getComponent(Interactable.getTypeName()) as Interactable;

    if (interactable) {
        this.updateInteractableProperties(interactable);
    } else if (attempts < 10) { // Max 10 attempts
        const delayedEvent = this.createEvent("DelayedCallbackEvent");
        delayedEvent.bind(() => this.tryGetInteractable(attempts + 1));
        delayedEvent.reset(0.1); // Wait 100ms between attempts
    } else {
        print("UIKitButtonController: Failed to find Interactable component after 10 attempts");
    }
}

private updateInteractableProperties(interactable: Interactable): void {
    interactable.targetingMode = this.targetingMode;
    interactable.targetingVisual = this.targetingVisual;
    interactable.ignoreInteractionPlane = this.ignoreInteractionPlane;
    interactable.keepHoverOnTrigger = this.keepHoverOnTrigger;
    interactable.enableInstantDrag = this.enableInstantDrag;
    interactable.isScrollable = this.isScrollable;
    interactable.allowMultipleInteractors = this.allowMultipleInteractors;
    interactable.enablePokeDirectionality = this.enablePokeDirectionality;
    interactable.acceptableXDirections = this.acceptableXDirections;
    interactable.acceptableYDirections = this.acceptableYDirections;
    interactable.acceptableZDirections = this.acceptableZDirections;
    interactable.useFilteredPinch = this.useFilteredPinch;
}

After onAwake, it tries ten times within a second to see if the button script has already finished creating the Interactable, and if so, it transfers all the properties you set in the Inspector to the actual Interactable. Not quite rocket science, but very useful.

And by the way, if you want to have the sound effects back that you got using the SIK buttons, just add an Audio component and a properly configured InteractableAudioFeedback below the button script, and it will be picked up automatically by the Interactable that will be created at runtime by the UIKit button scripts, like this:

All code and samples are already in the demo project used in my previous two posts, so I’m not going to add a specific new project for this. You can simply nick the UIKitButtonController script from there.

The ScrollMenu prefab

I usually build up my prefabs in such a way that the top-level SceneObject has a kind of controller script, while there is always one child SceneObject that holds the actual visible part (in this case, a menu). That way, I can let the controller script handle the actual display state and the way things work by calling a script method, without having to mess with the actual internal structure of the prefab, potentially turning off parts that have vital controlling scripts on it, messing up the workings of the app, and potentially creating issues that way.

The controller script in this case is called - very originally - UIKitScrollMenuController. It features a few input fields. You can change the first three (in fact, you must do so with the Scroll Button Prefab field after you dragged it onto your scene, as I explained in part 1). The last three should best be left undisturbed.

The first field is the vertical size a button uses (including padding), the second the horizontal size. The control makes buttons in two columns and as many rows as necessary. If you want more columns, you will have to adapt the code. Since you will have to press them by finger, I don’t anticipate much narrower buttons, so I guess you don’t have to change Column Size that often, but Y Offset you might. It is now tailored toward my sample button.

The MenuFrame component contains the Frame script showing the UI canvas, as well as the HeadLock and the Billboard script keeping the UI more or less in view.

One thing of note - if you are using Billboard, please remember to disable “Allow translation”, otherwise you can still grab and move the floating window, but you will more or less be fighting the HeadLock and the Billboard scripts, which is not desirable. Either the user decides where a window goes, or the system - but not both.

Some other details:

• ScrollWindowAnchor determines where on the floating screen the scroll window will appear. This you can use mostly to decide the vertical starting point, should you require to change that.
• ScrollWindow itself decides the actual size of the scroll area.
• Scrollbar determines the vertical position of the scrollbar.
• Slider determines the size of the scrollbar.

If you change either ScrollWindowAnchor or ScrollWindow, be prepared to fiddle with Scrollbar and ScrollbarSlider until it all fits nicely together again, with sizes aligning visually, etc.

Scripts

The whole thing works using only three custom scripts:

• BaseScrollButtonData (which I already explained in the previous post)
• BaseUIKitScrollButtonController
• UIKitScrollMenuController

So let’s start with the easy part:

BaseUIKitScrollButtonController

This is a fairly simple script, but still requires some explanation. Let’s start with the header and the events.

@component
export class BaseUIKitScrollButtonController extends BaseScriptComponent {
    @input buttonText: Text;
    @input uiKitButton: BaseButton;

    private onButtonPressedEvent = new Event<BaseScrollButtonData>();
    public readonly onButtonPressed = this.onButtonPressedEvent.publicApi();

    public onHoveredEvent = new Event<boolean>();
    public onHovered = this.onHoveredEvent.publicApi();

Remember this can be used as a parent class component for your own button script. Here you can see what it does behind the curtains.

• Text should contain the button’s text to be set by the data fed to this button’s setButtonData method (as explained before)
• It exposes an onButtonPressed event that is triggered when the button is pressed, and returns the BaseScrollButtonData that was used to create this button in the first place.
• It also exposes an event onHovered that tells the interested listener whether the button is hovered over by the user.

Although both events are public, they are typically only used internally, by the UIKitScrollMenuController, as will become clear later.

The setButtonData is used by UIKitScrollMenuController to feed the actual button data to the button that is to be created:

public setButtonData(scrollButtonData: BaseScrollButtonData): void {
    if (this.uiKitButton != null) {
        this.uiKitButton.onHoverEnter.add(() => this.onHoveredEvent.invoke(true));
        this.uiKitButton.onHoverExit.add(() => this.onHoveredEvent.invoke(false));
        this.uiKitButton.onTriggerDown.add(() => this.onButtonPressedEvent.invoke(scrollButtonData));
        this.buttonText.text = scrollButtonData.buttonText;
        this.applyCustomSettings(scrollButtonData);
    }
}

protected applyCustomSettings(scrollButtonData: BaseScrollButtonData): void {
}

It wires up the button’s internal events to the BaseUIKitScrollButtonController’s onButtonPressed and onHovered, both of which will be consumed by the UIKitScrollMenuController. It also sets the button’s text, then finally calls the (here empty) applyCustomSettings method that you can override in a child class should you need to do so, to perform some custom actions for your custom button. I showed an example of that here.

UIKitScrollMenuController

This is basically the magic wand that all ties it together. The start is simple enough:

@component
export class UIKitScrollMenuController extends BaseScriptComponent {
    @input yOffset: number = 5;
    @input columnSize: number = 4;
    @input scrollButtonPrefab: ObjectPrefab;
    @input scrollWindow: ScrollWindow;
    @input menuRoot: SceneObject;
    @input closeButton: BaseButton;

    private onButtonPressedEvent = new Event<BaseScrollButtonData>();
    public readonly onButtonPressed = this.onButtonPressedEvent.publicApi();
    private scrollArea: SceneObject;

On top we see the six inputs already discussed before, then again a onButtonPressed that can inform interested listeners what button in the list was pressed (as shown here). The scrollArea we will need for a peculiar thing later.

Next is the setting up in onAwake:

private onAwake(): void {
    this.scrollArea = this.scrollWindow.getSceneObject();
    this.setMenuVisible(false);
    const delayedEvent = this.createEvent("DelayedCallbackEvent");
    delayedEvent.bind(() => {
        this.initializeUI();
    });
    delayedEvent.reset(0.1);
}

We get a reference to the actual scrollWindow, we hide the menu for now, then start a delayed event for initializing the UI. This is necessary because for some reason the close button is not awake at onAwake yet, so you get the “Component not yet awake” error otherwise.

The methods for initializing the UI, as well as opening and closing the menu are as follows:

protected initializeUI(): void {
    this.closeButton.onTriggerDown.add(() => this.closeMenu());
}

closeMenu() {
    const delayedEvent = this.createEvent("DelayedCallbackEvent");
    delayedEvent.bind(() => {
        this.setMenuVisible(false);
    });
    delayedEvent.reset(0.25);
    this.setMenuVisible(false);
}

public setMenuVisible(visible: boolean): void {
    this.menuRoot.enabled = visible;
}

In the closeMenu I keep a standard 0.25 seconds delay so the ‘click’ sound of the button has time to play; otherwise it will not play or be clipped as menuRoot SceneComponent, that holds all the UI, is hidden. The menuRoot component should be set to the first child in the prefab, as said before:

Otherwise the UIKitScrollMenuController will essentially disable itself.

The meat of the matter is the createButtons method, which essentially creates all buttons and the structure to support events to the outside world. Your own code should call it, feeding it an array of BaseScrollButtonData (or a child class of that). It starts as follows:

public createButtons(scrollButtonData: BaseScrollButtonData[]): void {
    var lines = Math.ceil(scrollButtonData.length / 2);
    var initOffset = lines % 2 != 0 ? this.yOffset : this.yOffset / 2;
    var yStart = Math.ceil(lines / 2) * this.yOffset - initOffset;
    var line = 0;
    this.scrollWindow.onInitialized.add(() => {
        this.scrollWindow.setScrollDimensions(new vec2(0, lines * this.yOffset));
    });
    this.setMenuVisible(true);

It first calculates how many rows of buttons are required, then the initial offset from the center, and with that at what y coordinate the buttons need to start (this will be used later). Then the scroll window scroll dimensions is set to the vertical size times the number of lines. Since we are only scrolling vertically, we only need to set the y part of it.

In hindsight I should have called yOffset “rowSize”, but what the heck.

for (let i = 0; i < scrollButtonData.length; i++) {
    var button = this.scrollButtonPrefab.instantiate(this.scrollArea);
    var buttonTransform = button.getTransform();
    var xPos = (i % 2 == 0) ? -this.columnSize : this.columnSize;
    buttonTransform.setLocalPosition(
      new vec3(xPos, yStart - this.yOffset * line, 0.1));
    button.enabled = true;
    if (i % 2 != 0) {
        line++;
    }

    const buttonController =
       getComponent<BaseUIKitScrollButtonController>(button,
                   BaseUIKitScrollButtonController);
    buttonController.setButtonData(scrollButtonData[i]);
    buttonController.onHovered.add((p) => {
        this.scrollWindow.vertical = !p;
    });
    buttonController.onButtonPressed.add((data) =>
       this.onButtonPressedEvent.invoke(data));
}
this.updateScrollPosition();

So, for every entry in scrollButtonData this code:

• Creates a button
• Places it on the left or right side of the list based on whether it’s an odd or even button
• Enables the button
• Increases the line after every two buttons
• Gets a reference to BaseUIKitScrollButtonController
• Feeds the BaseScrollButtonData entry to its setButtonData method - this will hook up the button
• Makes sure the window scrolling is disabled when you hover over a button
• Routes a button’s pressed event to the outside so you can listen to all buttons in one place
• And finally calls updateScrollPosition

The third-to-last thing deserves a bit of explanation. I noticed it was pretty hard to press a button on a scrollable list, especially in not very good lighting conditions, as you tend to accidentally drag/move the list if you just miss the buttons, or the Spectacles camera misses you trying to press it. This makes it a lot more usable.

updateScrollPosition now is also a bit of a hacky thing. Because if you fill up a UIKit scroll list, it tends to set its scroll button halfway down the list. Why that is, I don’t know.

private updateScrollPosition(): void {
    const delayedEvent = this.createEvent("DelayedCallbackEvent");
    delayedEvent.bind(() => {
        this.scrollWindow.scrollPositionNormalized = new vec2(0, 1);
        this.menuRoot.getTransform().setLocalScale(new vec3(1, 1, 1));
    });
    delayedEvent.reset(1);
}

It basically sets the scrollPositionNormalized to 0, 1, which translates to “vertical top scroll position”. 0, 0 is vertical center, 0, -1 is vertical bottom. If you don’t add updateScrollPosition, instead of the desired left screen, you get the right screen.

Conclusion

So that’s kind of it. I do hope it’s useful for you and also gives you some insights into how you can cajole some UIKit elements into the shape you want. There is actually another little helper class in here, but I will deal with that later in yet another blog post.

The demo project is (still) here at GitHub.

This will be a two part blog post. This first post is relatively short and explains how to use it, the next one will describe more in detail how it works. If just you want to use it, this post should get you going.

How does it look?

Well, like this.

And if you press buttons, you see per button two messages in the logger:

Both the button and the menu itself print out what button is pressed. That is all for now - it’s yours to decide what you actually want the events to do. The menu is a just frame, also stays neatly in view, but that’s all standard Spectacles Interaction Kit components. The only thing I added is a custom close button.

How can it be used?

Very fast and high over:

• Make a prefab with a UIKit BaseButton, and make sure is has a BaseUIKitScrollButtonController component (or a child class of that)
• Drag the ScrollMenu’s prefab onto the scene
• Drag your button prefab on the input field “Scroll Button Prefab” of the ScrollMenu’s UIKitScrollMenuController component.
• Create a bootstrapper script
• Give that a reference (@input) to a UIKitScrollMenuController
• Drag the ScrollMenu’s UIKitScrollMenuController on that input
• The bootstrapper script somehow gets a feed of data, and turns in into a list of BaseScrollButtonData - or a list of BaseScrollButtonData child classes.
• It feeds that list into UIKitScrollMenuController.createButtons
• And the buttons will be created.

Now if that went a bit too fast, I will go over it into more detail

Button and button controller

The UIKitScrollMenuController, which I will talk about later, instatiates buttons within its scroll list, so it needs something to instatiate. If you look into the demo project, you will see an example of how such an instantiable button prefab might look. My example is called, very orignally, MyButton. This has a couple of rather standard components, and it also has a MyButtonController.

It is defined like this:

import { BaseScrollButtonData } from "LocalJoost/Ui/ScrollWindow/Scripts/BaseScrollButtonData";
import { BaseUIKitScrollButtonController } 
  from "LocalJoost/Ui/ScrollWindow/Scripts/BaseUIKitScrollButtonController";

@component
export class MyButtonController extends BaseUIKitScrollButtonController {
    protected applyCustomSettings(scrollButtonData: BaseScrollButtonData): void {
        super.applyCustomSettings(scrollButtonData);
        this.uiKitButton.onTriggerDown.add(() => {
            print("From MyButtonController: button pressed: " + scrollButtonData.buttonText);
        });
    }
}

Your button controller must inherit BaseUIKitScrollButtonController. It inherits a Text and a UIKitButton @input field from it, and those must be set in the editor. The base class does a lot of useful things, setting the button text amongst others. It also gives you an applyCustomSettings method to override, in this case it rather trivially prints out to the logger that it is pressed, and shows the button text from the BaseScrollButtonData that it has been fed.

BaseScrollButtonData

This is a very simple data class from which buttons are created and every buttons gets fed such a BaseScrollButtonData object (or a child class)

export class BaseScrollButtonData {
    public buttonText: string;
}

As stated, you can also make a child class that contains lots more data fields, and do something interesting with it to the button in the override of applyCustomSettings of your custom equivalent of MyButtonController. For instance, you can add a color attribute and give every button a different color. Or, like I did, you can add attributes that hold location, altitude, name and the IATA airport code of an airport and use that to select a location on a map when a button is pressed.

Configure ScrollMenu prefab

The whole menu is fit within the ScrollMenu prefab. Drag this prefab onto your scene. The prefab has a couple of inputs you can fill, all of which can be ignored - apart from the Scroll Button Prefab input: that must contain a prefab. Drag your Button prefab on that field

and the ScrollMenu is almost ready to go.

Feed the menu and listen to the menu

As wrote before, the menu needs a list of BaseScrollButtonData data objects containing at least the button text to be able to generate a list of buttons. This is a sample of a bootstrapper script that does just that:

import { BaseScrollButtonData } from "LocalJoost/Ui/ScrollWindow/Scripts/BaseScrollButtonData";
import { UIKitScrollMenuController } from "LocalJoost/Ui/ScrollWindow/Scripts/UIKitScrollMenuController";

@component
export class ScrollButtonDataLoader extends BaseScriptComponent {
    @input scrollMenuController: UIKitScrollMenuController

    private onAwake(): void {
        const buttonDataArray: BaseScrollButtonData[] = [];
        for (let i = 1; i <= 20; i++) {
            const buttonData = new BaseScrollButtonData();
            buttonData.buttonText = "Button " + i;
            buttonDataArray.push(buttonData); 
        }
        this.createButtons(buttonDataArray);
        this.scrollMenuController.onButtonPressed.add((data) => {
            print("From UIKitScrollMenuController: button Pressed: " + data.buttonText);
        });
    }
}

This script needs an reference to the menu’s UIKitScrollMenuController so in can call it’s method createButtons method.

In real use, buttonDataArray would be filled by reading a data stream from the web or some kind of configuration file. However, in this simple sample I just make them locally. But the scrollMenuController also exposes and event onButtonPressed to which you can subscribe and do something with, although here we also just print out the data that was fed to the button, just to show it works.

Some bits and pieces

The UIKitScrollMenuController also features a setMenuVisible method which you can use to turn the menu of or off from the outside, without having to reload the data using the ScrollButtonDataLoader or whatever you want to call your loader script.

Make sure your scripts lives in a scene object under the actual ScrollMenu prefab, otherwise you most likely will run into the dreaded “Compent not yet awake” error.

For the sake of completness: this menu needs both the Spectacles Interaction Kit and UIKit to be installed.

The demo project with all the neccesary components can be found here.

someSceneObject.getComponent(MyComponentClass.getTypeName()) as MyComponentClass;

And what is even worse, it does not work at all when when you need to find the reference but you only have the component’s base class. Suppose I have this trivial base controller component:

@component
export class MyBaseClass extends BaseScriptComponent {

    public hello(): void {
        print("Hello from MyBaseClass");
    }
}

However, I have multiple prefabs types with a slightly different child class suited for their particular purpose, for instance:

import { MyBaseClass } from "./MyBaseClass";

@component
export class MyChildClass extends MyBaseClass {

    public hello(): void {
        print("Hello from MyChildClass");
    }
}

If I add this child class to my prefab, then try to find a reference to its controlling component using the MyBaseClass class

myPrefab.getComponent(MyBaseClass.getTypeName()) as MyBaseClass;

I wil get a null value result! This is highly confusing for someone coming from Unity, and rather limiting as well.

However, there is a solution. I have extended my SceneUtils with this little method:

export function getComponent<T>(
    sceneObject: SceneObject,
    myClassType: new (...args: any[]) => T): T | null {
    for (const component of sceneObject.getComponents("Component") as Component[]) {
        if (component instanceof myClassType) {
            return component as T
        }
    }
    return null;
}

I made a little test script in the demo project to show how it works:

import { MyBaseClass } from "./MyBaseClass";
import { getComponent } from "LocalJoost/Utilities/SceneUtils";

@component
export class TestScript extends BaseScriptComponent {
    @input prefab : ObjectPrefab;
   
    onAwake() {
        var instance = this.prefab.instantiate(this.getSceneObject());
        var getComponentResult = instance.getComponent(MyBaseClass.getTypeName()) as MyBaseClass;
        print("getComponentByClass: " + (getComponentResult != null));

        var findScriptComponentResult = getComponent<MyBaseClass>(instance, MyBaseClass);
        print("findScriptComponentByClass: " + (findScriptComponentResult != null));
        findScriptComponentResult.hello();
    }
}

Here’s the logger output that actually proves it works

Although I search using the base class, I get a reference to a child class object. Which is exactly what you would expect. Or at least what I would expect. It’s only five lines of code, so I wonder why Snap didn’t implement it this way in their own getComponent. Maybe they wanted to leave in some challenges for grumpy old skool software engineers to solve 😁. However, this works and it easy to use.

• findSceneObjectByName
• findComponentInChildren
• findAllComponentsInChildren
• findScriptComponentInChildren
• findAllScriptComponentsInChildren
• findComponentInParents
• findAllComponentsInParents
• findScriptComponentInParents
• findComponentInSelfOrParents
• findAllScriptComponentsInSelfOrParents

If you are coming from Unity, you might easily fall into the trap (as I did) of seeing not differences between Component and ScriptComponent, use findComponentInChildren or findAllComponentsInChildren, notice that that won’t even compile because it wants a “keyof ComponentNameMap” in stead of a type - and waste a lot of time on that. So if you want to find ScriptComponent/behaviours that you have placed in the scene and your team mate has moved it somewhere, you should rather use the *findScriptComponent* methods than the *findComponent* methods.

However, with the exception of findSceneObjectByName, these only work if you provide a top SceneComponent. So if your lovely team mates have created multiple root scene objects, good luck finding your scripts.

So I created this very little helper function that does that for you:

import { findAllScriptComponentsInChildren } 
  from "SpectaclesInteractionKit.lspkg/Utils/SceneObjectUtils";
import { DEFAULT_MAX_CHILD_SEARCH_LEVELS } 
  from "SpectaclesInteractionKit.lspkg/Utils/SceneObjectUtils";

export function findAllScriptComponentsInScene<T extends ScriptComponent>(
  scriptClass: new (...args: any[]) => T,
  maxDepth: number = DEFAULT_MAX_CHILD_SEARCH_LEVELS): T[] | null {
    const foundComponents: T[] = [];
    const rootCount = global.scene.getRootObjectsCount();
    try {
        for (let i = 0; i < rootCount; i++) {
            const so = global.scene.getRootObject(i);
            const components = findAllScriptComponentsInChildren<T>(so, scriptClass, maxDepth);
            if (components && components.length > 0) {
                foundComponents.push(...components);
            }
        }
        return foundComponents.length > 0 ? foundComponents : null;
    } catch (error) {
        print("Error while searching for components in scene: " + error);
        return null;
    }
}

I created a little demo project that demonstrates this. For this I made a trivial TestTarget ScriptComponent that I sprinkled through the scene:

@component
export class TestTarget extends BaseScriptComponent {

    public getInfo(): string {
        return "I am a TestTarget component on " + this.getSceneObject().name + "!";
    }
}

And an equally trivial TestFinder ScriptComponent that finds them:

import { findAllScriptComponentsInScene } from "LocalJoost/Utilities/SceneUtils";
import { TestTarget } from "TestTarget";

@component
export class TestFinder extends BaseScriptComponent {
    onAwake() {
        const foundTargets = findAllScriptComponentsInScene<TestTarget>(TestTarget);
        if (foundTargets) {
            for (const target of foundTargets) {
                print(target.getInfo());
            }
        }
    }
}

Result in the Lens Studio Logger panel shows it actually works:

A word of warning: this is highly inefficient as you are bascially traversing the whole scene to find script components. This can be useful at startup, for connecting the dots initially, especially in the circumstances I described. But do not use this very often, like from an Update loop, because especially in a large scene it can be very resource intensive and slow.

Lens Studio unfortunately doesn’t know the concept of prefab variants like Unity does, so if I want another prefab that is almost the same, I need to copy it. Let’s call it MyOtherPrefab. I add an extra text. The top scene object is still called MyAwesomePrefab, so let’s rename that, and hit Apply

You might think we are done, but this is were it gets odd. Because if you drag both MyAwesomePrefab and MyOtherPrefab in the scene…

They both show up as MyAwesomePrefab.

This, of course, can be highly confusing. Maybe it’s a bug, or maybe I simply not understand how this is supposed to work. However, this is how I fix it. Fortunately, all files a Lens Studio creates are text files. I don’t know what you call this format, but somewhere you will find this:

Change that to MyAwesomePrefab, save the file, and if you now drag MyAwesomePrefab on the scene…

Important note: save your project first before starting the rename. If you haven’t saved it, Lens Studio makes a kind of temporay project on a in a temporary location and that may cause issues when you do things like this. Even better - commit your project to Git before doing scary things like this. A prudent developer always makes small steps so it’s easy to recover from oopsies.

No code this time, as there is no code to share ;)

However, this posed a few problems when I was trying to deploy on Spectacles from Windows using the relatively new “wired connection” option. It became pretty clear no one at Snap used it much - few probably even tried - because it caused some headaches trying to get it to work. Still, this is where Snap’s dedication to their product shows. After posting my woes on the Spectacles Reddit, I was contacted by one of their engineers, Pasha Antonenko, and we had a well-over-an-hour-long video call (at a rather ungodly early hour for Pasha) in which we established the baseline for successfully connecting and deploying a Spectacles app to a Spectacles using a USB cable from Windows.

In short - the list of requirements is this:

1. Whenever you connect a Spectacles to a Windows machine, you get the “USB device malfunctioned” error. Always. However confusing that is, you can actually ignore that. You might first see a message that it set up a “composite USB device” first.
2. You must have the newest version of the ABD tools and have that set up in Lens Studio to get a stable connection between Lens Studio and the Spectacles. Do not - I repeat, do not - be tempted to re-use an ADB tools install from some Unity installation, as this developer initially tried because it seemed the logical thing to do.
3. Double-check that wired connection is turned on in the settings. I was sure they were, but they were not. Maybe a result of the recent upgrade on my phone. I use Android, and I guess the Snap folks are primarily on iOS too 😉.
4. You can only connect to a native USB-C port, so you need a cable with USB-C plugs on both sides. A converter cable with a USB-A plug on one side will not work properly, regardless of how fast the port or cable are rated.

Ad 2: do not forget to actually point to the place where you unpacked the ADB tools in preferences (Lens Studio / Preferences):

Ad 3: you need to set this in the Spectacles app on your phone:

Only when you see this in your Logger is the Spectacles successfully connected to your Windows computer, and the USB cable can be used for deployment.

No demo project this time, because there is no code to share.