Understanding the Gesture and Adding Air Tap Gesture into your Unity 3D Holographic App

Air Tap Gesture

Here is the seventh post in the series of Holographic App Development Using Microsoft HoloLens.  In this post we are going to learn how to integrate Gesture into your Holographic App. In the previous post we discussed about Understanding the Gaze and Adding a Gaze Input Cursor into your Unity 3D Holographic App. The gesture works in the conjunction with Gaze. You Gaze on some object, then take an action by using GestureGesture in HoloLens is similar to the mouse click on a desktop application or tapping on some item on the touch based app.

HoloLens view port are the few feet to either side where you are standing and looking at with wearing the device. The sensors inside HoloLens can recognized the gesture as long as we keep our hands inside the viewable range of the device .

HoloLens View Port

HoloLens View Port

Image Source : https://support.microsoft.com/en-us/help/12644/hololens-use-gestures

Here is a quick video that gives an overall inside on Gesture inputs for Holographic App.

There  are several types of gesture including following

  • Bloom
  • Tap and Hold
  • Air Tap

Let’s try of focus on what is Air Tap Gesture, and how to add the air tap gesture in your application.  Using the air tip, you can select holograms and take action for the same. Air Tap is similar to the mouse click where we can take an action on taping our finger down and take them up back, .

Air Tap Gesture

Air Tap Gesture

Image Source : https://support.microsoft.com/en-us/help/12644/hololens-use-gestures

Let’s see how to implement it in our Holographic application.

If this is the first of post of this series that you are reading, before you go for any further steps, I would recommend you must read following two posts as they are interrelated.

Let’s open the same solution (Holo Gaze) in unit we built as part of the previous post.

Existing HoloGaze Solution

Existing Holo Gaze Solution

Placing the Gesture Manager Script

Navigate to HoloToolkit –> Input and drag the GazeManager.cs script on to GameRoot object in the object Hierarchy.

Once the Script is added, you should be able to see this as a components in the Object Inspector . This time you must see there are two script, one is Gaze Manager, which was added as part of previous exercise, and now we have added Gesture Manager.

Gesture Manager Script

Gesture Manager Script

Gesture Handler

Gesture Handler should handle the messages sent by the Gesture Manager and trigger the required actions.  In this case we will rotate the cube object when tapped, and stop when they  tapped again.

Add a new Script  and name  it as “Gesture Handler”  and have it as a Singleton implementation

Gesture Handler

Gesture Handler

Add a new private variable to check if the rotation is already on, which mean if the air tap is already performed to the object.

Gesture Handler

Gesture Handler

Write a new method called OnAirTapped() where just change the value of isActive, and use the same value in the update() method to start or stop the rotation. Yes ! very simple to understand .

Gesture Handler

Gesture Handler

Let’s put the Gesture Handler Code together..

public class GestureHandler : Singleton<GestureHandler>
{
    private bool isActive = false;

    void Update()
    {

        if (isActive)
        {
            this.transform.Rotate(0, 1, 0);
        }

    }

    void OnAirTapped()
    {
        isActive = !isActive;
    }

}

Adding Gesture Handler To the Cube Objects

Drag & Drop or add the Gesture Handler script as components to all the cube objects in the hierarchy.

Adding Gesture Handler To the Cube Objects

Adding Gesture Handler To the Cube Objects

Now along with Gaze Handler,  all the Cube object will have Gesture Handler associated with it.

Invoking the Gesture Handler

Open the Gesture Manager, and Update the GestureRecognizer_TappedEvent with following code. This is just sending message the the Gazed Object and eventually invoking the OnAirTapped method

  private void GestureRecognizer_TappedEvent(InteractionSourceKind source, int tapCount, Ray headRay)
        {
            if (focusedObject != null)
            {
                focusedObject.SendMessage("OnAirTapped", SendMessageOptions.RequireReceiver);
            }
        }

How it works ?

GestureManager intialize a gesture recognizer and register for a tap event. When the tap gesture is detected,  GestureManager uses GazeManager to find the Gazed game object.  Then we can use the  GestureManager then sends a message to that game object and tell what’s needs to be done.

How it works ?

How it works ?

That’s it !

Build the solution in Unity, Run it from Visual Studio.  When the object is gaze, if you perform the Air Tip ( Press Space bar), you should see the cube is started rotating.  Air Tap it once again, the rotation will stop.

Gesture in Holographic App

Gesture in Holographic App

Gesture in Holographic App

Gesture in Holographic App

Applying Lighting effects

Let’s do some fun ! Apply some lightings effects. We already had one spotlight added in the solution. Add two additional spotlight, and position them  as show below image.

Gesture in Holographic App - Adding Lighting Effects

Gesture in Holographic App – Adding Lighting Effects

Of course you can select the color as per your preferences.  But make sure you have Shadow Type selected as “Soft Shadows” to get the shadow effects from the reflection of lights.

Gesture in Holographic App - Adding Lighting Effects

Gesture in Holographic App – Adding Lighting Effects

Let’s run it once again !!

Here is how your application is looks like now !

References

Hope this helps ! Thanks for reading out the series of posts on Holographic App Development Using Microsoft HoloLens.  Feel free to share your feedback, and let me know incase you have any questions.

5 thoughts on “Understanding the Gesture and Adding Air Tap Gesture into your Unity 3D Holographic App

  1. Hello,

    At first thanks for the Tutorial!

    I have a Question about this Codeline:

    void OnAirTapped()
    {
    isActive = !isActive;
    }

    This is not working in my project. I don´t know why. I tried to replace “isActive = !isActive” with “isActive = true”. Then the cube ist rotating. But when I take your line or for example this:


    if(isActive == true)
    isActive = false
    else
    isActive = true

    I hope I could explain my Problem. It´s not difficult I guess but I don´t know why its not working either.

    Thanks for your help

    Regards
    kevin

    Like

  2. I think when developing for a platform like Hololens it is very easy to overcomplicate simple things like rotation or translation. Thank you for simplifying the process of understanding. Great article that explains the concepts.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s