Air Tap Gesture

Understanding the Gesture and Adding Air Tap Gesture into your Unity 3D Holographic App

Here is the seventh post in the series of Holographic App Development Using Microsoft HoloLens.  In this post, we are going to learn how to integrate Gesture into your Holographic App. In the previous post, we discussed Understanding the Gaze and Adding a Gaze Input Cursor into your Unity 3D Holographic App. The gesture works in conjunction with Gaze. You Gaze on some object, then take action by using GestureGesture in HoloLens is similar to the mouse click on a desktop application or tapping on some item on the touch based app.

HoloLens view port is the few feet to either side where you are standing and looking at with wearing the device. The sensors inside HoloLens can recognize the gesture as long as we keep our hands inside the viewable range of the device.

HoloLens View Port
HoloLens View Port

Image Source : https://support.microsoft.com/en-us/help/12644/hololens-use-gestures

Here is a quick video that gives an overall inside on Gesture inputs for Holographic App.

There  are several types of gesture including following

  • Bloom
  • Tap and Hold
  • Air Tap

Let’s try of focus on what is Air Tap Gesture, and how to add the air tap gesture in your application.  Using the air tip, you can select holograms and take action for the same. Air Tap is similar to the mouse click where we can take action on taping our finger down and take them up back.

Air Tap Gesture
Air Tap Gesture

Image Source : https://support.microsoft.com/en-us/help/12644/hololens-use-gestures

Let’s see how to implement it in our Holographic application.

If this is the first of post of this series that you are reading, before you go for any further steps, I would recommend you must read following two posts as they are interrelated.

Let’s open the same solution (Holo Gaze) in unit we built as part of the previous post.

Existing HoloGaze Solution
Existing Holo Gaze Solution

Placing the Gesture Manager Script

Navigate to HoloToolkit –> Input and drag the GazeManager.cs script on to GameRoot object in the object Hierarchy.

Once the Script is added, you should be able to see this as components in the Object Inspector. This time you must see there are two scripts, one is Gaze Manager, which was added as part of the previous exercise, and now we have added Gesture Manager.

Gesture Manager Script
Gesture Manager Script

Gesture Handler

Gesture Handler should handle the messages sent by the Gesture Manager and trigger the required actions.  In this case, we will rotate the cube object when tapped, and stop when they tapped again.

Add a new Script  and name  it as “Gesture Handler”  and have it as a Singleton implementation

Gesture Handler
Gesture Handler

Add a new private variable to check if the rotation is already on, which mean if the air tap is already performed to the object.

Gesture Handler
Gesture Handler

Write a new method called OnAirTapped() where just change the value of isActive, and use the same value in the update() method to start or stop the rotation. Yes! very simple to understand.

Gesture Handler
Gesture Handler

Let’s put the Gesture Handler Code together..

public class GestureHandler : Singleton<GestureHandler>
{
    private bool isActive = false;

    void Update()
    {

        if (isActive)
        {
            this.transform.Rotate(0, 1, 0);
        }

    }

    void OnAirTapped()
    {
        isActive = !isActive;
    }

}

Adding Gesture Handler To the Cube Objects

Drag & Drop or add the Gesture Handler script as components to all the cube objects in the hierarchy.

Adding Gesture Handler To the Cube Objects
Adding Gesture Handler To the Cube Objects

Now along with Gaze Handler,  all the Cube object will have Gesture Handler associated with it.

Invoking the Gesture Handler

Open the Gesture Manager, and Update the GestureRecognizer_TappedEvent with following code. This is just sending message the the Gazed Object and eventually invoking the OnAirTapped method

  private void GestureRecognizer_TappedEvent(InteractionSourceKind source, int tapCount, Ray headRay)
        {
            if (focusedObject != null)
            {
                focusedObject.SendMessage("OnAirTapped", SendMessageOptions.RequireReceiver);
            }
        }

How it works ?

GestureManager intializes a gesture recognizer and registers for a tap event. When the tap gesture is detected,  GestureManager uses GazeManager to find the Gazed game object.  Then we can use the  GestureManager then sends a message to that game object and tell what’s needs to be done.

How it works ?
How it works ?

That’s it !

Build the solution in Unity, Run it from Visual Studio.  When the object is gaze, if you perform the Air Tip ( Press Space bar), you should see the cube is started rotating.  Air Tap it once again, the rotation will stop.

Gesture in Holographic App
Gesture in Holographic App
Gesture in Holographic App
Gesture in Holographic App

Applying Lighting effects

Let’s do some fun! Apply some lightings effects. We already had one spotlight added to the solution. Add two additional spotlight, and position them as shown below image.

Gesture in Holographic App - Adding Lighting Effects
Gesture in Holographic App – Adding Lighting Effects

Of course, you can select the color as per your preferences.  But make sure you have Shadow Type chosen as “Soft Shadows” to get the shadow effects from the reflection of lights.

Gesture in Holographic App - Adding Lighting Effects
Gesture in Holographic App – Adding Lighting Effects

Let’s run it once again !!

Here is how your application is looks like now !

References

Hope this helps! Thanks for reading out the series of posts on Holographic App Development Using Microsoft HoloLens.  Feel free to share your feedback, and let me know in case you have any questions.

Advertisements

Written by Abhijit Jana

I am a Modern and Cloud App Development Consultant working at Microsoft. Presently I am focusing on adopting and delivering world class solution on N+1 Technologies, where I mainly drive engagements on Universal Windows Platforms, Microsoft Azure, IoT and Cross Platform and Device related application Development. I am a former Microsoft MVP, Author of Kinect for Windows SDK and HoloLens BluePrints book and Run Daily .NET Tips . To know more about me click here.

16 comments

  1. Hello,

    At first thanks for the Tutorial!

    I have a Question about this Codeline:

    void OnAirTapped()
    {
    isActive = !isActive;
    }

    This is not working in my project. I don´t know why. I tried to replace “isActive = !isActive” with “isActive = true”. Then the cube ist rotating. But when I take your line or for example this:


    if(isActive == true)
    isActive = false
    else
    isActive = true

    I hope I could explain my Problem. It´s not difficult I guess but I don´t know why its not working either.

    Thanks for your help

    Regards
    kevin

    Like

  2. I think when developing for a platform like Hololens it is very easy to overcomplicate simple things like rotation or translation. Thank you for simplifying the process of understanding. Great article that explains the concepts.

    Like

  3. Hey Abhijit,

    Thank you so much for your articles on HoloLens.
    I was trying to add the AirTap Gesture as described in this article. But I did not find any script file by the name GestureManager. On checking the GitHub repository, I found that the file has been removed. Now, I’m not really clear on how to proceed here.
    Could you please let me know how to proceed in this scenario? Would be really appreciate it.

    Like

    1. Dear Wade,
      With the current version of HoloToolkit, you can achieve the Gesture by following

      Attached this with the object for which you intended to do gesture action.

      public class SomeObjectGestureHandler : MonoBehaviour, IInputClickHandler
      {
      public void OnInputClicked(InputEventData eventData)
      {
      // Write your gesture code here
      }

      // Use this for initialization
      void Start()
      {

      }

      // Update is called once per frame
      void Update()
      {

      }
      }

      You can also Implement, IFocusable interface, which has two method definition OnFocusEnter() — for Gaze In and OnFocusExit() — for Gaze Out.

      Hope this helps !

      Let me know if you need any additional help.

      Thanks

      Like

      1. Hey Abhijit,

        Thank you so much for the prompt reply. I tried implementing the Gesture Action as you suggested by implementing IInputClickHandler interface. Unfortunately, none of the gestures were detected. And none of the GestureHandler methods were called. What am I missing here? FYI, I was just trying to replicate this exact tutorial.
        Also, how can I get console logs to show up when debugging?

        Like

        1. Hello Wade, Did you attached the script with object from which you intent a gesture ? For an example – You have a Cube, and you want some action on Cube, then you have to attached that SomeObjectGestureHandler script with your Cube Object ?
          Were you able to get cursor ./ Gaze ? Please refer to my previous articles “comments” section where I explained about changes for Gaze with new HoloToolkit.

          Let me know if this works.

          Cheers!!

          Like

  4. Hey Abhijit,

    I did get the Gaze tutorial working. Also, I did attach the GazeHandler script to the cubes. Following is my code for the GestureManager.
    I have a feeling it might be because I left the start & update methods empty. But I’m not sure.
    Also, are we supposed to used the selectedObkject property of the InputEventData?
    Your help is highly appreciated.

    public class GestureManager : MonoBehaviour, IInputClickHandler {

    public void OnInputClicked(InputEventData eventData)
    {
    eventData.selectedObject.SendMessage(“OnAirTapped”, SendMessageOptions.RequireReceiver);
    }

    // Use this for initialization
    void Start () {

    }

    // Update is called once per frame
    void Update()
    {
    }
    }

    Like

      1. No. The cubes didn’t rotate as shown in the tutorial video.
        Also, as mentioned in the tutorial, I have attached the GestureManager script to the GameRoot object and the GestureHandler script to the cubes.
        Are we supposed to used the selectedObject property of the InputEventData?

        Like

  5. Hello Abhijit,

    I think we are missing something here.
    Found this about capturing gestures on the Github community for HoloLens:
    “To handle gestures you need to have one InputManager (singleton) and at least one GestureInput in your scene. To handle the events, the MonoBehaviours that handle them has to implement any of the following (one or more), depending on what you are interested: IInputClickHandler, IHoldHandler, IManipulationHandler, INavigationHandler.
    The InputManager calls the handler that is at the GameObject with the gaze focus. If there is none, it goes up the scene hierarchy until it finds one. If none is found, it calls the registered fallback. If you want the handler to be always called, add SetGlobalListener.cs to the GameObject.”

    On trying this suggested technique, I did see the method OnTappedEvent in GestureInput and the InputSource_SourceClicked method in the InputManager being called. The issue was that the OnInputClickedEventHandler in the InputManager is not calling the correct handler. (In our case, the GestureHandler.cs file).

    Will you be able to look into & say what I maybe missing. Once again, your help is highly appreciated.

    Like

  6. Abhijit,

    I want to perform a script whenever the user airtaps anywhere and want it to affect an object. So I don’t want the tap to be attached to any object in particular but want the script to run whenever there’s an airtap. How would you suggest achieving this?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s