Understanding the Gaze and Adding a Gaze Input Cursor into your Unity 3D Holographic App

This is the sixth post in the series of Holographic App Development Using Microsoft HoloLens. In one of my previous post we have discussed about How to build a Holographic Apps with Unity 3D and Microsoft HoloLens.  Now let’s try to interact with the same holographic app we built so far. To start with, we will focus on Gaze Input.  Like we use a mouse to move a cursor, it is now Gaze for the HoloLens. Gaze is one form of input on HoloLens. Gaze Input in HoloLens indicate where the user is currently focusing on. A cursor is used to indicate the direction of the Gaze. The cursor follows the point of attention as users heads moves in the scene.

 

Gaze

Gaze Input in HoloLens

To determine the point of user’s attention or user currently focusing on,  each frame do a Raycast into the scene  and place cursor on the object where the Raycast is intersect with the objects. The same formula goes even when user is moving along with HoloLens and thus the focus is maintained.

Raycast into the Scene

Raycast into the Scene

Watch this Video to get a quick glance on what is Gaze Input and how it works.

Just to summarize, here is the list of  points that were highlighted in this video.

  • Cursor Indicates the Gaze Direction.
  • Raycast determines point of attention.
  • Cursor can hug surfaces it intersects.
  • Non-directional cursors often work best.
  • Arrow can lead user to action.
  • Cursor can be temporarily  hidden.

To start with Adding a Gaze cursor in to our application we will refer to the previous developed 3D Holographic application .. from a bit early stage and HoloToolkit – Unity

Developing Holographic Apps with Unity 3D and Microsoft HoloLens

HoloApp


Start Using HoloToolkit – Unity

The HoloToolkit – Unity  is the repository that contains all Unity specific components and scripts to make development faster. The HoloToolkit is a collection of scripts and components intended to accelerate development of holographic applications targeting Windows Holographic…

HoloToolkit contains the following feature areas:

  • Input
  • Sharing
  • Spatial Mapping
  • Spatial  Sound
  • Utilities

.. Read More about HoloToolkit Here

Including HoloToolkit in to the Unity Solution

You can include the HoloToolkitUnity, easily in to your Unity Project.

Adding HoloToolkit in your Unity Project

Adding HoloToolkit in your Unity Project


Adding a Gaze Input Cursor

To add the Gaze Input Cursor from HoloToolkit  we need follow two steps

  • Placing the Gaze Manager Script
  • Placing the Cursor in the Scene

Placing the Gaze Manager Script

The Gaze Manager script used to calculates the Raycast hit position and normal. If the Raycast hits a hologram, it set the position and normal to match the intersection point.

Add an Empty GameObject in the Scene, and named it as “GameRoot

Adding Game Script in Root

Adding Game Script in Root

Navigate to HoloToolkit –> Input and drag the GazeManager.cs script on to GameRoot object in the object Hierarchy.

Adding Gaze Manager

Adding Gaze Manager

Once the Script is added, you should be able to see this as a components in the Object Inspector.

Verify the Gaze Manager Script associations

Verify the Gaze Manager Script associations

Placing the Cursor in the Scene

Cursor is the Visual Indication for the Gaze. When we  Raycast hits the Hologram, we need to display this cursor to indicate the gazed object.

Adding the Cursor

Adding the Cursor

Navigate to HoloToolkit –> Prefabs –> Input, and add the cursor into the object hierarchy

Adding the Cursor

Adding the Cursor

Adding the Cursor

Adding the Cursor

At this point of time if you inspect the Cursor Object, you will find “Cursor Manager”  is already attached with it.

Cursor Manager Script to Control The Cursor

Cursor Manager Script to Control The Cursor

CursorManager class takes Cursor GameObjects and  shows the appropriate Cursor when a Hologram is hit.  This hit is calculated by the GazeManage Raycast.

Gaze Manager Interaction with Cursor Manager

Gaze Manager Interaction with Cursor Manager

That’s it for now ! We can run and test the cursor appearance when Gaze.


Run and Test the Gaze Input Cursor

Run the application and you should be able to view the cursor as soon as focused is on any of the Cube. When there is no focused object, you won’t see the cursor appearing.

Test Gaze Cursor

Test Gaze Cursor


Behind the scene… How it works ?

If you want to check what is happening in behind, the entire things is managed by the Gaze Manager and Cursor Manager scripts. On each frame updates, the Update() method for Gaze Manager is invoked, which calling the UpdateRaycast() method to find the Raycast hit information.  When Raycast hot the Hologram, LateUpdate() method from the Cursor Manager is invoked, that takes control for showing and hiding the cursor.

Update - Update Raycast - Last Update Interaction

Update – Update Raycast – Last Update Interaction


Gaze Handler –  Adding Visual Effects When Gaze Enter / Exit

Let’s take it a step forward and giving a Visual effects on the object when it is being gazed.

Add a new Script  and name  it as “’Gaze Handler”  and have it as a Singleton implementation.

image

Add Two method OnGazeEnter() and OnGazeExit() respectively for Gaze Enter and Gaze Exit.

image

Now, let’s try to change the color of the object as soon as Gaze enter, and revert back to the original color when Gaze off.  Following code snippet are just trying to get the material color used for objects and updating with a fixed (yellow) color and when Gaze Exit, the previously stored color is getting reverted back.

Gaze - Enter / Exit

Gaze – Enter / Exit

Here is how the entire script class looks like…

public class GazeHandler : Singleton<GazeHandler>
{
    private Color startColor;

    // Use this for initialization
    void Start()
    {
    }

    // Update is called once per frame
    void Update()
    {
    }

    void OnGazeEnter()
    {
        var com = gameObject.GetComponent<Renderer>();
        startColor = com.material.color;
        com.material.color = Color.yellow;
    }

    void OnGazeExit()
    {
        var com = gameObject.GetComponent<Renderer>();
        com.material.color = startColor;
    }
}

Now refer to the script over here Git repository / Or Explore it in Visual Studiohttps://github.com/Microsoft/HoloToolkit-Unity/blob/master/Assets/HoloToolkit/Input/Scripts/GazeManager.cs

You will find how the Gaze manager is checking the change of current Hit Object. Just update the method name that we write recently for Gaze Exit and Gaze Entry.

Invoking Gaze Enter / Exit

Invoking Gaze Enter / Exit

Add this script as components to all the cube object.

Adding Gaze Handler Script

Adding Gaze Handler Script

That’s it.

Now, as soon as a old object is changed and new object gets focus, the Gaze Enter and Exit will start invoking. Here is a complete set of code map that shows the entire flow starting from Raycast to On Gaze Enter.

Gaze Manager - Gaze Handler Interaction Flow

Gaze Manager – Gaze Handler Interaction Flow

Here is how it look like – Video

HoloToolkit is really very useful and can make your HoloLens App development very fast ! Start exploring it !

Hope this help !

11 thoughts on “Understanding the Gaze and Adding a Gaze Input Cursor into your Unity 3D Holographic App

  1. I followed your steps but its not working in my case. Few differences i found in Hololens kit and your script. For instance under gazeManager we SendMessageOptions.RequireReceiver in your sample but in Hololens kit its showing SendMessageOptions.DontRequireReceiver.
    I’m bit confuse here.
    Would you please help?

    Like

    • Hi Amit, What is not working ? Are you able to see the cursor ? did you tried with breakpoint and see if the script are getting called as expected.

      With Respect of SendMessageOptions, the Requirer Receiver is used when there is a listener. In the next article I have created my own handler to handle the gaze. you may look at that as well. Please do let me know if that resolve your issues. Or else, I can help you further. Thanks again for asking.

      Cheers !

      Like

  2. I also followed your steps. My Cube color is changing to yellow when I gaze over the object. But when I leave, the color still remains yellow. I debugged the code and I saw that the color in “starColor” ist always the same. Even after changing the the color of the cube in unity. So i guess its saving the yellow color. Do you know what I am missing?

    Thanks
    Kevin

    Like

    • Hello Kevin, at OnGazeEnter() method, could you please check what value are you receiving in the Start Color for first time ?
             var com = gameObject.GetComponent();
              startColor = com.material.color;

      Like

  3. hello,
    I’m very new in unity and hololens .
    There’s no cursor manager inside the holotoolkit assets, and when I want to just importing the “cursor manager”script from academy tutorial, an error 0246 occurs.
    where’s the cursor manager? And how to add gaze without this script?

    Like

    • With the new version of HoloToolkit, you don’t required Cursor manager. Have a Empty Root object, Add Input Manager, Gaze Manager and Gaze Stabilizer.
      Add your 3D Object
      Add Cursor Prefabs from the Input > Prefabs
      Run the application, You should start seeing the Cursor.

      For Gesture, Use below interface and implement them in the class.

      NewBehaviourScript : MonoBehaviour, IFocusable, IInputClickHandler

      Let me know if you still have problem.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s