This is the sixth post in the series of Holographic App Development Using Microsoft HoloLens. In one of my previous post, we have discussed How to build a Holographic Apps with Unity 3D and Microsoft HoloLens.  Now let’s try to interact with the same holographic app we made so far. To start with, we will focus on Gaze Input.  Like we use a mouse to move a cursor, it is now Gaze for the HoloLens. Gaze is one form of input on HoloLens. Gaze Input in HoloLens indicates where the user is currently focusing. A cursor is used to indicates the direction of the Gaze. The cursor follows the point of attention as users heads move in the scene.

 

To determine the point of user’s attention or user currently focusing on,  each frame does a raycast into the scene and place cursor on the object where the raycast is intersected with the objects. The same formula goes even when the user is moving along with HoloLens, and thus the focus is maintained.

Watch this Video to get a quick glance on what is Gaze Input and how it works.

Just to summarize, here is the list of  points that were highlighted in this video.

• Cursor Indicates the Gaze Direction.
• Raycast determines point of attention.
• Cursor can hug surfaces it intersects.
• Non-directional cursors often work best.
• Arrow can lead user to action.
• Cursor can be temporarily  hidden.

To start with Adding a Gaze cursor into our application, we will refer to the previous developed 3D Holographic application.. from a bit early stage and HoloToolkit – Unity

Developing Holographic Apps with Unity 3D and Microsoft HoloLens

---

Start Using HoloToolkit – Unity

The HoloToolkit – Unity is the repository that contains all Unity specific components and scripts to make development faster. The HoloToolkit is a collection of scripts and components intended to accelerate development of holographic applications targeting Windows Holographic…

HoloToolkit contains the following feature areas:

• Input
• Sharing
• Spatial Mapping
• Spatial  Sound
• Utilities

.. Read More about HoloToolkit Here …

Including HoloToolkit in to the Unity Solution

You can include the HoloToolkit – Unity, easily in to your Unity Project.

• Download the Toolkit from  https://github.com/microsoft/HoloToolkit-Unity
• Unpack and Place it inside the Asset Folder of the Unity Project.

---

Adding a Gaze Input Cursor

To add the Gaze Input Cursor from HoloToolkit  we need follow two steps

• Placing the Gaze Manager Script
• Placing the Cursor in the Scene

Placing the Gaze Manager Script

The Gaze Manager script used to calculates the Raycast hit position and normal. If the Raycast hits a hologram, it set the position and normal to match the intersection point.

Add an Empty GameObject in the Scene, and named it as “GameRoot”

Navigate to HoloToolkit –> Input and drag the GazeManager.cs script on to GameRoot object in the object Hierarchy.

Once the Script is added, you should be able to see this as a components in the Object Inspector.

Placing the Cursor in the Scene

Cursor is the Visual Indication for the Gaze. When we  Raycast hits the Hologram, we need to display this cursor to indicate the gazed object.

Navigate to HoloToolkit –> Prefabs –> Input, and add the cursor into the object hierarchy

At this point of time if you inspect the Cursor Object, you will find “Cursor Manager” is already attached with it.

CursorManager class takes Cursor GameObjects and  shows the appropriate Cursor when a Hologram is hit.  This hit is calculated by the GazeManage Raycast.

That’s it for now ! We can run and test the cursor appearance when Gaze.

---

Run and Test the Gaze Input Cursor

Run the application and you should be able to view the cursor as soon as focus is on any of the Cube. When there is no focused object, you won’t see the cursor appearing.

---

Behind the scene… How it works ?

If you want to check what is happening in behind, the entire things are managed by the Gaze Manager and Cursor Manager scripts. On each frame updates, the Update() method for Gaze Manager is invoked, which calling the UpdateRaycast() method to find the Raycast hit information.  When Raycast hot the Hologram, LateUpdate() method from the Cursor Manager is invoked, that takes control of showing and hiding the cursor.

---

Gaze Handler –  Adding Visual Effects When Gaze Enter / Exit

Let’s take it a step forward and giving a Visual effects on the object when it is being gazed.

Add a new Script  and name  it as “’Gaze Handler”  and have it as a Singleton implementation.

Add Two method OnGazeEnter() and OnGazeExit() respectively for Gaze Enter and Gaze Exit.

Now, let’s try to change the color of the object as soon as Gaze enter, and revert to the original color when Gaze off.  Following code snippet are just trying to get the material color used for objects and updating with a fixed (yellow) color and when Gaze Exit, the previously stored color is getting reverted.

Here is how the entire script class looks like…

public class GazeHandler : Singleton<GazeHandler>
{
    private Color startColor;

    // Use this for initialization
    void Start()
    {
    }

    // Update is called once per frame
    void Update()
    {
    }

    void OnGazeEnter()
    {
        var com = gameObject.GetComponent<Renderer>();
        startColor = com.material.color;
        com.material.color = Color.yellow;
    }

    void OnGazeExit()
    {
        var com = gameObject.GetComponent<Renderer>();
        com.material.color = startColor;
    }
}

Now refer to the script over here Git repository / Or Explore it in Visual Studio – https://github.com/Microsoft/HoloToolkit-Unity/blob/master/Assets/HoloToolkit/Input/Scripts/GazeManager.cs

You will find how the Gaze manager is checking the change of current Hit Object. Just update the method name that we write recently for Gaze Exit and Gaze Entry.

Add this script as components to all the cube object.

That’s it.

Now, as soon as an old object is changed and the new object gets focus, the Gaze Enter and Exit will start invoking. Here is a complete set of code map that shows the entire flow starting from Raycast to On Gaze Enter.

Here is how it look like – Video

HoloToolkit is very useful and can make your HoloLens App development very fast ! Start exploring it!

Hope this help !