Understanding the Gaze and Adding a Gaze Input Cursor into your Unity 3D Holographic App

This is the sixth post in the series of Holographic App Development Using Microsoft HoloLens. In one of my previous post, we have discussed How to build a Holographic Apps with Unity 3D and Microsoft HoloLens.  Now let’s try to interact with the same holographic app we made so far. To start with, we will focus on Gaze Input.  Like we use a mouse to move a cursor, it is now Gaze for the HoloLens. Gaze is one form of input on HoloLens. Gaze Input in HoloLens indicates where the user is currently focusing. A cursor is used to indicates the direction of the Gaze. The cursor follows the point of attention as users heads move in the scene.

 

Gaze
Gaze Input in HoloLens

To determine the point of user’s attention or user currently focusing on,  each frame does a raycast into the scene and place cursor on the object where the raycast is intersected with the objects. The same formula goes even when the user is moving along with HoloLens, and thus the focus is maintained.

Raycast into the Scene
Raycast into the Scene

Watch this Video to get a quick glance on what is Gaze Input and how it works.

Just to summarize, here is the list of  points that were highlighted in this video.

  • Cursor Indicates the Gaze Direction.
  • Raycast determines point of attention.
  • Cursor can hug surfaces it intersects.
  • Non-directional cursors often work best.
  • Arrow can lead user to action.
  • Cursor can be temporarily  hidden.

To start with Adding a Gaze cursor into our application, we will refer to the previous developed 3D Holographic application.. from a bit early stage and HoloToolkit – Unity

Developing Holographic Apps with Unity 3D and Microsoft HoloLens

HoloApp


Start Using HoloToolkit – Unity

The HoloToolkit – Unity is the repository that contains all Unity specific components and scripts to make development faster. The HoloToolkit is a collection of scripts and components intended to accelerate development of holographic applications targeting Windows Holographic…

HoloToolkit contains the following feature areas:

  • Input
  • Sharing
  • Spatial Mapping
  • Spatial  Sound
  • Utilities

.. Read More about HoloToolkit Here

Including HoloToolkit in to the Unity Solution

You can include the HoloToolkitUnity, easily in to your Unity Project.

Adding HoloToolkit in your Unity Project
Adding HoloToolkit in your Unity Project

Adding a Gaze Input Cursor

To add the Gaze Input Cursor from HoloToolkit  we need follow two steps

  • Placing the Gaze Manager Script
  • Placing the Cursor in the Scene

Placing the Gaze Manager Script

The Gaze Manager script used to calculates the Raycast hit position and normal. If the Raycast hits a hologram, it set the position and normal to match the intersection point.

Add an Empty GameObject in the Scene, and named it as “GameRoot

Adding Game Script in Root
Adding Game Script in Root

Navigate to HoloToolkit –> Input and drag the GazeManager.cs script on to GameRoot object in the object Hierarchy.

Adding Gaze Manager
Adding Gaze Manager

Once the Script is added, you should be able to see this as a components in the Object Inspector.

Verify the Gaze Manager Script associations
Verify the Gaze Manager Script associations

Placing the Cursor in the Scene

Cursor is the Visual Indication for the Gaze. When we  Raycast hits the Hologram, we need to display this cursor to indicate the gazed object.

Adding the Cursor
Adding the Cursor

Navigate to HoloToolkit –> Prefabs –> Input, and add the cursor into the object hierarchy

Adding the Cursor
Adding the Cursor
Adding the Cursor
Adding the Cursor

At this point of time if you inspect the Cursor Object, you will find “Cursor Manager” is already attached with it.

Cursor Manager Script to Control The Cursor
Cursor Manager Script to Control The Cursor

CursorManager class takes Cursor GameObjects and  shows the appropriate Cursor when a Hologram is hit.  This hit is calculated by the GazeManage Raycast.

Gaze Manager Interaction with Cursor Manager
Gaze Manager Interaction with Cursor Manager

That’s it for now ! We can run and test the cursor appearance when Gaze.


Run and Test the Gaze Input Cursor

Run the application and you should be able to view the cursor as soon as focus is on any of the Cube. When there is no focused object, you won’t see the cursor appearing.

Test Gaze Cursor
Test Gaze Cursor

Behind the scene… How it works ?

If you want to check what is happening in behind, the entire things are managed by the Gaze Manager and Cursor Manager scripts. On each frame updates, the Update() method for Gaze Manager is invoked, which calling the UpdateRaycast() method to find the Raycast hit information.  When Raycast hot the Hologram, LateUpdate() method from the Cursor Manager is invoked, that takes control of showing and hiding the cursor.

Update - Update Raycast - Last Update Interaction
Update – Update Raycast – Last Update Interaction

Gaze Handler –  Adding Visual Effects When Gaze Enter / Exit

Let’s take it a step forward and giving a Visual effects on the object when it is being gazed.

Add a new Script  and name  it as “’Gaze Handler”  and have it as a Singleton implementation.

image

Add Two method OnGazeEnter() and OnGazeExit() respectively for Gaze Enter and Gaze Exit.

image

Now, let’s try to change the color of the object as soon as Gaze enter, and revert to the original color when Gaze off.  Following code snippet are just trying to get the material color used for objects and updating with a fixed (yellow) color and when Gaze Exit, the previously stored color is getting reverted.

Gaze - Enter / Exit
Gaze – Enter / Exit

Here is how the entire script class looks like…

public class GazeHandler : Singleton<GazeHandler>
{
    private Color startColor;

    // Use this for initialization
    void Start()
    {
    }

    // Update is called once per frame
    void Update()
    {
    }

    void OnGazeEnter()
    {
        var com = gameObject.GetComponent<Renderer>();
        startColor = com.material.color;
        com.material.color = Color.yellow;
    }

    void OnGazeExit()
    {
        var com = gameObject.GetComponent<Renderer>();
        com.material.color = startColor;
    }
}

Now refer to the script over here Git repository / Or Explore it in Visual Studiohttps://github.com/Microsoft/HoloToolkit-Unity/blob/master/Assets/HoloToolkit/Input/Scripts/GazeManager.cs

You will find how the Gaze manager is checking the change of current Hit Object. Just update the method name that we write recently for Gaze Exit and Gaze Entry.

Invoking Gaze Enter / Exit
Invoking Gaze Enter / Exit

Add this script as components to all the cube object.

Adding Gaze Handler Script
Adding Gaze Handler Script

That’s it.

Now, as soon as an old object is changed and the new object gets focus, the Gaze Enter and Exit will start invoking. Here is a complete set of code map that shows the entire flow starting from Raycast to On Gaze Enter.

Gaze Manager - Gaze Handler Interaction Flow
Gaze Manager – Gaze Handler Interaction Flow

Here is how it look like – Video

HoloToolkit is very useful and can make your HoloLens App development very fast ! Start exploring it!

Hope this help !

Advertisements

Written by Abhijit Jana

I am a Modern and Cloud App Development Consultant working at Microsoft. Presently I am focusing on adopting and delivering world class solution on N+1 Technologies, where I mainly drive engagements on Universal Windows Platforms, Microsoft Azure, IoT and Cross Platform and Device related application Development. I am a former Microsoft MVP, Author of Kinect for Windows SDK and HoloLens BluePrints book and Run Daily .NET Tips . To know more about me click here.

12 comments

  1. I followed your steps but its not working in my case. Few differences i found in Hololens kit and your script. For instance under gazeManager we SendMessageOptions.RequireReceiver in your sample but in Hololens kit its showing SendMessageOptions.DontRequireReceiver.
    I’m bit confuse here.
    Would you please help?

    Like

    1. Hi Amit, What is not working ? Are you able to see the cursor ? did you tried with breakpoint and see if the script are getting called as expected.

      With Respect of SendMessageOptions, the Requirer Receiver is used when there is a listener. In the next article I have created my own handler to handle the gaze. you may look at that as well. Please do let me know if that resolve your issues. Or else, I can help you further. Thanks again for asking.

      Cheers !

      Like

  2. I also followed your steps. My Cube color is changing to yellow when I gaze over the object. But when I leave, the color still remains yellow. I debugged the code and I saw that the color in “starColor” ist always the same. Even after changing the the color of the cube in unity. So i guess its saving the yellow color. Do you know what I am missing?

    Thanks
    Kevin

    Like

    1. Hello Kevin, at OnGazeEnter() method, could you please check what value are you receiving in the Start Color for first time ?
             var com = gameObject.GetComponent();
              startColor = com.material.color;

      Like

  3. hello,
    I’m very new in unity and hololens .
    There’s no cursor manager inside the holotoolkit assets, and when I want to just importing the “cursor manager”script from academy tutorial, an error 0246 occurs.
    where’s the cursor manager? And how to add gaze without this script?

    Like

    1. With the new version of HoloToolkit, you don’t required Cursor manager. Have a Empty Root object, Add Input Manager, Gaze Manager and Gaze Stabilizer.
      Add your 3D Object
      Add Cursor Prefabs from the Input > Prefabs
      Run the application, You should start seeing the Cursor.

      For Gesture, Use below interface and implement them in the class.

      NewBehaviourScript : MonoBehaviour, IFocusable, IInputClickHandler

      Let me know if you still have problem.

      Like

      1. It works!!!Thank you! I can see the cursor on and off my assembly!
        Thank you!
        I still have problem with gestures. My task is very basic and simple, I got a solidworks assembly, the effect what I need is, I can see the assembly hologram and interact with it, like move, rotate, explode the parts, etc in my hololens. Just like the effect in academy 210 and 211:”Interactible(choose different of assembly)”,”Navigation(rotate assembly in every direction)”,”Manipulation(move assembly)”,”Model expansion(expand the parts of assembly)”.
        For now, I know how to transfer the assembly to unity, and with your help I can add gaze input. But you said:”For Gesture, Use below interface and implement them in the class.
        NewBehaviourScript : MonoBehaviour, IFocusable, IInputClickHandler” confused me, it’s totally different between the academy tutorial and latest holotoolkit, As a very new beginner, I don’t know what to do with the holotoolkit.

        Thank you again!!!

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s