Here is the seventh post in the series of Holographic App Development Using Microsoft HoloLens. In this post, we are going to learn how to integrate Gesture into your Holographic App. In the previous post, we discussed Understanding the Gaze and Adding a Gaze Input Cursor into your Unity 3D Holographic App. The gesture works in conjunction with Gaze. You Gaze on some object, then take action by using Gesture. Gesture in HoloLens is similar to the mouse click on a desktop application or tapping on some item on the touch based app.
HoloLens view port is the few feet to either side where you are standing and looking at with wearing the device. The sensors inside HoloLens can recognize the gesture as long as we keep our hands inside the viewable range of the device.
Image Source : https://support.microsoft.com/en-us/help/12644/hololens-use-gestures
Here is a quick video that gives an overall inside on Gesture inputs for Holographic App.
There are several types of gesture including following
- Bloom
- Tap and Hold
- Air Tap
Let’s try of focus on what is Air Tap Gesture, and how to add the air tap gesture in your application. Using the air tip, you can select holograms and take action for the same. Air Tap is similar to the mouse click where we can take action on taping our finger down and take them up back.
Image Source : https://support.microsoft.com/en-us/help/12644/hololens-use-gestures
Let’s see how to implement it in our Holographic application.
If this is the first of post of this series that you are reading, before you go for any further steps, I would recommend you must read following two posts as they are interrelated.
Let’s open the same solution (Holo Gaze) in unit we built as part of the previous post.
Placing the Gesture Manager Script
Navigate to HoloToolkit –> Input and drag the GazeManager.cs script on to GameRoot object in the object Hierarchy.
Once the Script is added, you should be able to see this as components in the Object Inspector. This time you must see there are two scripts, one is Gaze Manager, which was added as part of the previous exercise, and now we have added Gesture Manager.
Gesture Handler
Gesture Handler should handle the messages sent by the Gesture Manager and trigger the required actions. In this case, we will rotate the cube object when tapped, and stop when they tapped again.
Add a new Script and name it as “Gesture Handler” and have it as a Singleton implementation
Add a new private variable to check if the rotation is already on, which mean if the air tap is already performed to the object.
Write a new method called OnAirTapped() where just change the value of isActive, and use the same value in the update() method to start or stop the rotation. Yes! very simple to understand.
Let’s put the Gesture Handler Code together..
public class GestureHandler : Singleton<GestureHandler>
{
private bool isActive = false;
void Update()
{
if (isActive)
{
this.transform.Rotate(0, 1, 0);
}
}
void OnAirTapped()
{
isActive = !isActive;
}
}
Adding Gesture Handler To the Cube Objects
Drag & Drop or add the Gesture Handler script as components to all the cube objects in the hierarchy.
Now along with Gaze Handler, all the Cube object will have Gesture Handler associated with it.
Invoking the Gesture Handler
Open the Gesture Manager, and Update the GestureRecognizer_TappedEvent with following code. This is just sending message the the Gazed Object and eventually invoking the OnAirTapped method
private void GestureRecognizer_TappedEvent(InteractionSourceKind source, int tapCount, Ray headRay)
{
if (focusedObject != null)
{
focusedObject.SendMessage("OnAirTapped", SendMessageOptions.RequireReceiver);
}
}
How it works ?
GestureManager intializes a gesture recognizer and registers for a tap event. When the tap gesture is detected, GestureManager uses GazeManager to find the Gazed game object. Then we can use the GestureManager then sends a message to that game object and tell what’s needs to be done.
That’s it !
Build the solution in Unity, Run it from Visual Studio. When the object is gaze, if you perform the Air Tip ( Press Space bar), you should see the cube is started rotating. Air Tap it once again, the rotation will stop.
Applying Lighting effects
Let’s do some fun! Apply some lightings effects. We already had one spotlight added to the solution. Add two additional spotlight, and position them as shown below image.
Of course, you can select the color as per your preferences. But make sure you have Shadow Type chosen as “Soft Shadows” to get the shadow effects from the reflection of lights.
Let’s run it once again !!
Here is how your application is looks like now !
References
Hope this helps! Thanks for reading out the series of posts on Holographic App Development Using Microsoft HoloLens. Feel free to share your feedback, and let me know in case you have any questions.
Abhijit's Blog 
Hello,
At first thanks for the Tutorial!
I have a Question about this Codeline:
void OnAirTapped()
{
isActive = !isActive;
}
This is not working in my project. I don´t know why. I tried to replace “isActive = !isActive” with “isActive = true”. Then the cube ist rotating. But when I take your line or for example this:
…
if(isActive == true)
isActive = false
else
isActive = true
I hope I could explain my Problem. It´s not difficult I guess but I don´t know why its not working either.
Thanks for your help
Regards
kevin
LikeLike
I think when developing for a platform like Hololens it is very easy to overcomplicate simple things like rotation or translation. Thank you for simplifying the process of understanding. Great article that explains the concepts.
LikeLike
Thanks Baishali !! Glad that it helped !
LikeLike
Hey Abhijit,
Thank you so much for your articles on HoloLens.
I was trying to add the AirTap Gesture as described in this article. But I did not find any script file by the name GestureManager. On checking the GitHub repository, I found that the file has been removed. Now, I’m not really clear on how to proceed here.
Could you please let me know how to proceed in this scenario? Would be really appreciate it.
LikeLike
Dear Wade,
With the current version of HoloToolkit, you can achieve the Gesture by following
Attached this with the object for which you intended to do gesture action.
public class SomeObjectGestureHandler : MonoBehaviour, IInputClickHandler
{
public void OnInputClicked(InputEventData eventData)
{
// Write your gesture code here
}
// Use this for initialization
void Start()
{
}
// Update is called once per frame
void Update()
{
}
}
You can also Implement, IFocusable interface, which has two method definition OnFocusEnter() — for Gaze In and OnFocusExit() — for Gaze Out.
Hope this helps !
Let me know if you need any additional help.
Thanks
LikeLike
Hey Abhijit,
Thank you so much for the prompt reply. I tried implementing the Gesture Action as you suggested by implementing IInputClickHandler interface. Unfortunately, none of the gestures were detected. And none of the GestureHandler methods were called. What am I missing here? FYI, I was just trying to replicate this exact tutorial.
Also, how can I get console logs to show up when debugging?
LikeLike
Hello Wade, Did you attached the script with object from which you intent a gesture ? For an example – You have a Cube, and you want some action on Cube, then you have to attached that SomeObjectGestureHandler script with your Cube Object ?
Were you able to get cursor ./ Gaze ? Please refer to my previous articles “comments” section where I explained about changes for Gaze with new HoloToolkit.
Let me know if this works.
Cheers!!
LikeLike
Hey Abhijit,
I did get the Gaze tutorial working. Also, I did attach the GazeHandler script to the cubes. Following is my code for the GestureManager.
I have a feeling it might be because I left the start & update methods empty. But I’m not sure.
Also, are we supposed to used the selectedObkject property of the InputEventData?
Your help is highly appreciated.
public class GestureManager : MonoBehaviour, IInputClickHandler {
public void OnInputClicked(InputEventData eventData)
{
eventData.selectedObject.SendMessage(“OnAirTapped”, SendMessageOptions.RequireReceiver);
}
// Use this for initialization
void Start () {
}
// Update is called once per frame
void Update()
{
}
}
LikeLike
So if you attached the GestureManager Script with Cube isn’t hit the OnInputClicked() method ?
LikeLike
No. The cubes didn’t rotate as shown in the tutorial video.
Also, as mentioned in the tutorial, I have attached the GestureManager script to the GameRoot object and the GestureHandler script to the cubes.
Are we supposed to used the selectedObject property of the InputEventData?
LikeLike
Hello Abhijit,
I think we are missing something here.
Found this about capturing gestures on the Github community for HoloLens:
“To handle gestures you need to have one InputManager (singleton) and at least one GestureInput in your scene. To handle the events, the MonoBehaviours that handle them has to implement any of the following (one or more), depending on what you are interested: IInputClickHandler, IHoldHandler, IManipulationHandler, INavigationHandler.
The InputManager calls the handler that is at the GameObject with the gaze focus. If there is none, it goes up the scene hierarchy until it finds one. If none is found, it calls the registered fallback. If you want the handler to be always called, add SetGlobalListener.cs to the GameObject.”
On trying this suggested technique, I did see the method OnTappedEvent in GestureInput and the InputSource_SourceClicked method in the InputManager being called. The issue was that the OnInputClickedEventHandler in the InputManager is not calling the correct handler. (In our case, the GestureHandler.cs file).
Will you be able to look into & say what I maybe missing. Once again, your help is highly appreciated.
LikeLike
Abhijit,
I want to perform a script whenever the user airtaps anywhere and want it to affect an object. So I don’t want the tap to be attached to any object in particular but want the script to run whenever there’s an airtap. How would you suggest achieving this?
LikeLike
Hello Abhijit,
Any update on this?
LikeLike