An advanced gaze-following behaviour to place objects on top or in front of obstructions (and scale if necessary)

Intro

In my previous post I described some complicated calculations using a BoxCastAll to determine how to place a random object on top or in front of some obstruction in the looking direction of the user, be it another object or the Spatial Mesh. Because the post was long enough as it was, I described the calculations separately. They are in an extra method "GetObjectBeforeObstruction" in my HoloToolkitExtension LookingDirectionHelpers, and I wrote a very simple Unity behaviour to show how it could be used. But that behaviour simply polls every so many seconds (2 is default) where the user looks, then calls GetObjectBeforeObstruction and moves the object there. This gives a kind of nervous result. I promised a more full fledged behaviour, and here it is: AdvancedKeepInViewController. It’s basically sitting in a project that looks remarkably like the demo project in the previous post: the same ‘scenery’, only there’s a 4th element you can toggle using the T button or by saying “Toggle”.

Features

The behaviour

• Only moves the object if the head is rotated more than a certain number of degrees per second, or the user moves a certain number of meters per second. It use the CameraMovementTracker from Walk the World that I described in an earlier post.
• Optionally fine tunes the location where the object is placed after doing an initial placement (effectively doing a BoxCastAll twice per movement)
• Optionally scales the object to have a more or less constant viewing size. This is indented for 'billboards' like objects - i.e. floating screens.
• Optionally makes an object appear right in front of the user if it's enabled, in stead of moving it in view the first time from the last place where it was before it got disabled.
• Optionally makes the object disappear when the user is moving a certain number of metes per second, to prevent objects from blocking the view or providing distractions. This is especially useful when running an app in a HoloLens while you are on a factory floor where you really want to see things like handrails, electricity cables, or holes in the floor (possibly with a smelter in it).

The code is not that complicated, but I thought it best to explain it step by step. I skip the part where all the editor-settable properties are listed - you can find them in AdvancedKeepInViewController's source in the demo project. I have added explanatory tooltip description to almost all of them.

Starting up

The start is pretty simple:

void Start()
{
    _objectMaterial = GetComponentInChildren<Renderer>().material;
    _initialTransparency = _objectMaterial.color.a;
}

void OnEnable()
{
    _startTime = Time.time + _delay;
    DoInitialAppearance();
    _isJustEnabled = true;
}

private void DoInitialAppearance()
{
    if (!AppearInView)
    {
        return;
    }

    _lastMoveToLocation = GetNewPosition();
    transform.position = _lastMoveToLocation;
}

We get the material of the first Render's material we can find and it's initial transparency, as we need be able to revert to that later. Then we need to check if the user has selected to object to initially appear in front, and if so, do the initial appearance. At the end you see GetNewPosition being called, that's a simple wrapper around LookingDirectionHelpers.GetObjectBeforeObstruction. It tries to project the object to hit an obstruction at a certain max direction; if there is no obstruction in that range, just give a point at the maximum distance. Since it's called multiple times and I am lazy, I made a little method of it

private Vector3 GetNewPosition()
{
    var newPos = LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
        DistanceBeforeObstruction, LayerMask, _stabilizer);
    if (Vector3.Distance(newPos, CameraCache.Main.transform.position) < MinDistance)
    {
        newPos = LookingDirectionHelpers.CalculatePositionDeadAhead(MinDistance);
    }
    return newPos;
}

Moving around

The main thing is, of course, driven by the Update loop. The Update method therefore the heart of the matter:

void Update()
{
    if (_startTime > Time.time)
        return;
    if (_originalScale == null)
    {
        _originalScale = transform.localScale;
    }

    if (!CheckHideWhenMoving())
    {
        return;
    }

    if (CameraMovementTracker.Instance.Distance > DistanceMoveTrigger ||
        CameraMovementTracker.Instance.RotationDelta > DeltaRotationTrigger || 
        _isJustEnabled)
    {
        _isJustEnabled = false;
        MoveIntoView();
    }
#if UNITY_EDITOR
    if (_showDebugBoxcastLines)
    {
        LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
            DistanceBeforeObstruction, LayerMask, _stabilizer, true);
    }
#endif
}

After the startup timeout (0.1 second) has been expired, we first gather the original scale of the object (needed if we actually scale). If the user is moving fast enough, hide the object and stop doing anything. Else, use the CameraMovementTracker that I wrote about two posts ago to determine if the user has moved or rotated enough to warrant a new location for the object (and the first time the code gets here, repositioning should happen anyway). And then it simply shows the Box Cast debug lines, that I already extensively showed off in my previous post.

So the actual moving around is done by these two methods (using once again good old LeanTween), and the second one is pretty funky indeed:

private void MoveIntoView()
{
    if (_isMoving)
    {
        return;
    }

    _isMoving = true;
    var newPos = GetNewPosition();
    MoveAndScale(newPos);
}

private void MoveAndScale(Vector3 newPos, bool isFinalAdjustment = false)
{
    LeanTween.move(gameObject, newPos, MoveTime).setEaseInOutSine().setOnComplete(() =>
    {
        if (!isFinalAdjustment && EnableFineTuning)
        {
            newPos = GetNewPosition();
            MoveAndScale(newPos, true);
        }
        else
        {
            _isMoving = false;
            DoScaleByDistance();
        }
    });
    _lastMoveToLocation = newPos;
}

So the move MoveIntoView method first check if a move action is not already initiated. Then it gets a new position using - duh - GetNewPosition again, and calls MoveAndScale. MoveAndScale moves the object to it's new position, then calls itself an extra time. The idea behind this is a follows: the actual bounding box of the object might have changed between the original cast in MoveIntoView and the eventual positioning if the object you move is locked to be looking at the Camera while it moved, using something like the Mixed Reality Toolkit's BillBoard or (as in my sample) my very simple LookAtCamera behaviour . So a second 'finetuning' call is done, using the 'isFinalAdjustment' parameter. And if we are done moving, optionally we do some scaling. And this looks like this:

You might also notice the cubes appear from the camera’s origin, the floating screen appears initially in the right place. This is another option you can select.

Scale it up. Or down

For an object like a floating screen with text, you might want to ensure readability. So if your text is projected too far away, it might become unreadable. If it is projected too close, the text might become huge, the user can only see a small portion of it - and effectively it's unreadable too. Hence this little helper method

private void DoScaleByDistance()
{
    if (!ScaleByDistance || _originalScale == null || _isScaling)
    {
        return;
    }
    _isScaling = true;
    var distance = Vector3.Distance(_stabilizer ? _stabilizer.StablePosition : 
        CameraCache.Main.transform.position,
        _lastMoveToLocation);
    var newScale = _originalScale.Value * distance / MaxDistance;
    LeanTween.scale(gameObject, newScale, MoveTime).setOnComplete(() => _isScaling = false);
}

I think it only makes sense for 'text screens', not for 'natural' objects, therefore it's an option you can turn off in the editor. But if you do turn it on, it determines the scale by multiplying the original scale by the distance divided by the MaxDistance, assuming that is the distance you want to see you object on it's original scale as defined in the editor. Be aware that the autoscaling can make the screen appear inside other objects again, so use wisely and with caution.

Fading away when necessary

This method should return false whenever the object is faded out, or fading in or out – that way, MoveIntoView does not get called by Update.

private bool CheckHideWhenMoving()
{
    if (!HideWhenMoving || _isFading)
    {
        return true;
    }
    if (CameraMovementTracker.Instance.Speed > HideSpeed &&
        !_isHidden)
    {
        _isHidden = true;
        StartCoroutine(SetFading());
        LeanTween.alpha(gameObject, 0, FadeTime);
    }
    else if (CameraMovementTracker.Instance.Speed <= HideSpeed && _isHidden)
    {
        _isHidden = false;
        StartCoroutine(SetFading());
        LeanTween.alpha(gameObject, _initialTransparency, FadeTime);
        MoveIntoView();
    }

    return !_isHidden;
}

private IEnumerator SetFading()
{
    _isFading = true;
    yield return new WaitForSeconds(FadeTime + 0.1f);
    _isFading = false;
}

Basically this method says: if this thing should be hidden at high speed and is not already fading in or out:

• If the user’s speed is higher than the threshold value and the object is visible, hide it
• If the user’s speed is lower than the threshold value and the object is invisible, show it.

The way of hiding and showing is once againdone with LeanTween, but I found that using the .setOnComplete was a bit unreliable for detecting the fading in or out came to an end. So I simply use a coroutine that sets the blocking _isFading, waits a wee bit longer than the FadeTime, and then clears _isFading again. That way, no multiple fades can start or stop.

The tiny matter of transparency

The HideWhenMoving feature has a dependency – for it to work, the material needs to support transparency. That is to say – it’s rendering mode needs to be set to transparent (or at least not opaque). As you move around quickly, the semi transparent box and the double boxes will fade out and in nicely:

But if you move around and the floating screen wants to fade, you will see only the text fade out – the window outline stays visible. This has a simple reason: the material’s rendering mode is set is opaque, not transparent

The background of the screen with the button fades out nicely though because it uses a different material – actually a copy, but with only the rendering mode set to transparent:

But if you look really carefully, you will see not the entire screen fades out. Part of the button seems to remain visible. The culprit is the button’s backplate:

Now it’s up to you – you can change the opacity this material, and then it will be fixed for all buttons. The problem is that this material is part of the Mixed Reality Toolkit. So if you update that, it will most likely be overwritten. And then you will have to keep track of changes like this. Or you can manually change every backplate of every button, or do that once and make your own prefab button. There are multiple ways to Rome in this case.

Nice all those Unity demos…

But how does it look in real life? Well, like in this video. First it shows you the large semi-transparent cube actually disappears if you move quickly enough, then it shows the moving and scaling of the "Hello World" screen, but it also shows that when you move quickly enough, it will try to fade, but only the text will fade. The two cubes show nothing special other than that they appear more or less on the spatial mesh, and the "Screen with button" shows shrinking and growing as well, and it will fade completely - but the back plate. I have told you how to fix that.

Some tidbits

If you try to run the project in an HoloLens or Immersive Headset and wonder where the hell the cubes, capsule and other 'scenery' is that is clearly visible in the Unity editor - the are explicitly turned off by something called the HideInRuntime behaviour that sits in the "Scenery" game object, where all the 'scenery'  resides it. This is because in a HoloLens, you already have real obstructions. If you want to try this in an Immersive headset, please remove or disable this feature otherwise you will be in a void with almost nothing to test the behaviour at all.

Conclusion

Unlike the previous one, this behaviour makes full use of the possibilities GetObjectBeforeObstruction offers. I think there’s still room for improvement here and tweaks. For instance, if you want to use this behaviour to move and place stuff, simply disable the the behaviour when it’s done. But this behaviour as it is, is very usable and in fact I use it myself in various apps now.