Development With Kinect .NET SDK (Part II) – Using NUI APIs with Camera

In my previous post Development With Kinect .NET SDK (Part I) – Installation and Development Environment Setup I discussed about  fundamentals of  Kinect sensors and developing application using Kinect .NET SDK. I also explained different types of API that mainly used for interacting with Kinect Device. Natural User interfaces (NUI) and Audio are the core APIs stack for Kinect sensors and  and in the previous article we starts with NUI APIs. By this time you must know how to get start with of NUI API’s by initializing the runtime. Let’s take a inner look of NUI API. In this post I will discuss about NUI APIs for Kinect .NET SDK and will see how we can interact with Kinect Camera Sensors using it.

As we are going to explore the NUI APIs, first of all I will try to explain about fundamentals of Kinect SDK Library and type of APIs, then we will look inside NUI APIs and in the end we will be developing some application as shown in below.

Yeah. Super cool and lots of learning from this article !!

NUI APIs are the set of APIs that retrieves data from Kinect Sensors, camera and control the devices. This enables you to access images and stream data from Kinect Sensors.  So, what it does? For any kind of application your application access the NUI APIs, which interacts with Kinect Sensors and transfer back Image Stream, Depth Stream data to applications and this APIs also allow user to take control of Kinect Motor.

Why this is called as Natural User Interface (NUI ) ? , because this APIs help developer to use the Kinect for Natural Inputs like skeleton tracking, sensing, depth measuring etc.   As said, NUI APIs interacts with Camera , Sensors and Motors where as you  need a separate set of APIs to take control over Audio Mic Array ( We are not covering Audio array part in this article) . Below picture illustrate overall API Classification for Kinect SDK.

To start with any Kinect application, you have to Add Microsoft.Research.Kinect.dll as reference to interact with Kinect and first of all we need to  define a Runtime of Kinect Sensors.

This will create an instance of Kinect which represent the current device. ( Well, you can interact with multiple Kinect , will talk about this later ) .  Below diagram representing the overall APIs available for  Kinect SDK NUI . Let’s focus on the methods inside Runtime Class. This contain Initialize() and Uninitialize() method along with other methods. This two methods user for initializing and Uninitializing the Kinect Runtime.

In my previous post I talked about initializing and uninitializing of Kinect runtime.  Initialize() method Initializes the NUI subsystems that the application requires.  we have initialized the Kinect Runtime with RuntimeOptions as shown in below snippet.

RuntimeOptions specifies the runtime options for a Kinect sensor.  Kinect process data using Multi-stage channel. So, specifying runtime tell the what are the channel need to start.

RuntimeOption is a public Enumeration inside the NUI library.

UseColor streams the color image from the Kinect and this is used for color Video. UseDepth option stream the depth image data and UseSkeleatal provides the skeleton positions.

When you are done with all operation, call Runtime.Uninitialize() to close the instance.

Well, we learned lot about NUI API. Let’s used some of these API

Demo : NUI API’s Video and Depth Image

Start a new Visual Studio Instance and Create a New WPF Project Template with Name “KinectCameraNUIDemo”.

From Solution Explorer, right click on the project and Add “Microsoft.Research.Kinect.dll” as reference assembly

you already knows about the initializing the Kinect devices, so I am not discussing on that. Below code snippets shows the basic code blocks which instantiate a Kinect Runtime for the create WPF Application.

 /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        /// <summary>
        /// Runtime Instance for Device
        /// </summary>
        Runtime nuiRuntime = new Runtime();

        /// <summary>
        /// Initializes a new instance of the <see cref="MainWindow"/> class.
        /// </summary>
        public MainWindow()
        {
            InitializeComponent();
            Loaded += new RoutedEventHandler(MainWindow_Loaded);
            Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
        }

        /// <summary>
        /// Handles the Unloaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Unloaded(object sender, RoutedEventArgs e)
        {
            // Uninitialize the Kinect
            nuiRuntime.Uninitialize();
        }

        /// <summary>
        /// Handles the Loaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
           // Initialize the Runtime with Color Channel
            nuiRuntime.Initialize(RuntimeOptions.UseColor);
        }
    }

We are going to display some video capture from Kinect. So, lets first create the UI. The UI is very simple, put one Image and one button control.

Below is the XAML markup for the above design.

 <Grid Height="365">
        <Image Height="277" HorizontalAlignment="Left"
               Margin="64,12,0,0"
               Name="videoControl"
               Stretch="Fill"
               VerticalAlignment="Top" Width="385" />
        <Button Content="Capture"
                Height="41" HorizontalAlignment="Left"
                Margin="191,295,0,0" Name="buttonCapture"
                VerticalAlignment="Top" Width="143" />
    </Grid>

You have already initialized the Kinect runtime with UseColor RuntimeOptions. Which means, Kinect already initialized a channel for the color image streaming.  So in the next step you need to write an event handler for video frame. This Occurs when a video frame is ready. Code snippet for the VideoFrameReady event handler look like this :

Once you have done with signing the video frame, in next step you have to Open a video Stream channel for the image stream.

ImageStreamType gets the image stream type. It can be Video or Depth depends the type of event handler we have attached with it. Along with the ImageStreamType you have specify the pool size , resolution and ImageType which is also an enumeration that specifies an image type .

Let’s have handle the event for VideoFrameReady. Kinect  Video returns PlanarImage  and we just need to create new Bitmap source and display on the Image control.

        /// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            PlanarImage planarImage = e.ImageFrame.Image;
            videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
                PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

      }

That’s all, let run the application. and Click on the “Capture Button”, you will find output like below,

of course this should be a video display, not still image !!!

Very easy to capture Video. Want to make more easy , yes you can do it using Coding for Fun Kinect Toolkit . This Toolkit contains some useful extension methods for both WinForm and WPF version which will make things much easier.

As for example, using this Coding for Fun Toolkit, you need to write below code block for VideoStream event handler.

  /// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            //PlanarImage planarImage = e.ImageFrame.Image;
            //videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
            //    PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoControl.Source = e.ImageFrame.ToBitmapSource();
      }

but before using Coding for Fun APIs, you have add the “Coding4Fun.Kinect.Wpf.dll” ( As this a WPF Application ) as reference and put using Coding4Fun.Kinect.Wpf; as namespaces.

Let’s make this demo more interesting, while we are creating a bitmap using BitmapSource.Create() , we are passing the PixelFormats as one of the argument. Let’s change it do something different (Cmyk32) and bind the image to a different Image control.

/// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            PlanarImage planarImage = e.ImageFrame.Image;
            videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
                PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoSource2.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
              PixelFormats.Cmyk32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);
         }

Below is the resulting output :

Till now we covered only about the Video images, let’s have a look what happened for Depth Image.  Handling depth stream image is pretty similar to handling the Video steam, the only change required is the type of images and streams.

The first thing you need to change is initialization of runtime. This time you have to use UseDepth as well.

Once you are done with initialization, you need to raise an event handler for Depth Stream and need to open a new channel for Depth image where you have to specify the ImageStreamType as Depth and ImageType as Depth.

That’s all you really do not need to worry about other stuff. You can put multiple image control and bind different types of images to display it. Yes, its all about fun !!

Here is the complete code snippet :

namespace KinectCameraNUIDemo
{
    using System;
    using System.Windows;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;
    using Coding4Fun.Kinect.Wpf;
    using Microsoft.Research.Kinect.Nui;

    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        /// <summary>
        /// Runtime Instance for Device
        /// </summary>
        Runtime nuiRuntime = new Runtime(0);

        /// <summary>
        /// Initializes a new instance of the <see cref="MainWindow"/> class.
        /// </summary>
        public MainWindow()
        {

            InitializeComponent();
            Loaded += new RoutedEventHandler(MainWindow_Loaded);
            Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
        }

        /// <summary>
        /// Handles the Unloaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Unloaded(object sender, RoutedEventArgs e)
        {
            // Uninitialize the Kinect
            nuiRuntime.Uninitialize();
        }

        /// <summary>
        /// Handles the Loaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            // Initialize the Runtime with Color Channel
            nuiRuntime.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);

        }

        /// <summary>
        /// Handles the Click event of the buttonCapture control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonCapture_Click(object sender, RoutedEventArgs e)
        {

            // Signing the Event for Image frame ready
            nuiRuntime.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nuiRuntime_VideoFrameReady);
            nuiRuntime.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nuiRuntime_DepthFrameReady);
            nuiRuntime.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
            nuiRuntime.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);
        }

        void nuiRuntime_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            depthSourceControl.Source = e.ImageFrame.ToBitmapSource();

        }

        /// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            PlanarImage planarImage = e.ImageFrame.Image;
            videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
                PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoSource2.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
              PixelFormats.Cmyk32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoSource3.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
              PixelFormats.Bgr24, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);
        }
    }
}

Below is the XAML Markup Snippet, I just drag and drop few control .

  <Grid Height="451" Width="607">
         <Image Height="343" HorizontalAlignment="Left"
               Margin="12,12,0,0"
               Name="videoControl"
               Stretch="Fill"
               VerticalAlignment="Top" Width="385" />
        <Button Content="Capture"
                Height="41" HorizontalAlignment="Left"
                Margin="299,376,0,0" Name="buttonCapture"
                VerticalAlignment="Top" Width="143" Click="buttonCapture_Click" />
        <Image Height="150" HorizontalAlignment="Left" Margin="360,23,0,0"
               Name="videoSource2" Stretch="Fill" VerticalAlignment="Top" Width="200" />
        <Image Height="150" HorizontalAlignment="Left" Margin="360,183,0,0"
               Name="depthSourceControl" Stretch="Fill" VerticalAlignment="Top" Width="200" />
        <Image Height="113" HorizontalAlignment="Left" Margin="481,129,0,0"
               Name="videoSource3" Stretch="Fill" VerticalAlignment="Top" Width="114" />
    </Grid>

References :

Coding for Fun
Kinect Documentation

Download Complete Project

In this post we have learned about NUI API’s and learned how we can use those APIs to interact with Kinect Camera and sensors. In the next post I will talk about something more on camera and then we will start with skeleton tracking followed by Audio.

Read Out First Part over here Development With Kinect .NET SDK (Part I) – Installation and Development Environment Setup
Hope this helps !

Cheers !!

AJ