Development With Kinect .NET SDK (Part II) – Using NUI APIs with Camera

In my previous post Development With Kinect .NET SDK (Part I) – Installation and Development Environment Setup I discussed about  fundamentals of  Kinect sensors and developing application using Kinect .NET SDK. I also explained different types of API that mainly used for interacting with Kinect Device. Natural User interfaces (NUI) and Audio are the core APIs stack for Kinect sensors and  and in the previous article we starts with NUI APIs. By this time you must know how to get start with of NUI API’s by initializing the runtime. Let’s take a inner look of NUI API. In this post I will discuss about NUI APIs for Kinect .NET SDK and will see how we can interact with Kinect Camera Sensors using it.

As we are going to explore the NUI APIs, first of all I will try to explain about fundamentals of Kinect SDK Library and type of APIs, then we will look inside NUI APIs and in the end we will be developing some application as shown in below.

image

Yeah. Super cool and lots of learning from this article !!

NUI APIs are the set of APIs that retrieves data from Kinect Sensors, camera and control the devices. This enables you to access images and stream data from Kinect Sensors.  So, what it does? For any kind of application your application access the NUI APIs, which interacts with Kinect Sensors and transfer back Image Stream, Depth Stream data to applications and this APIs also allow user to take control of Kinect Motor.

image

Why this is called as Natural User Interface (NUI ) ? , because this APIs help developer to use the Kinect for Natural Inputs like skeleton tracking, sensing, depth measuring etc.   As said, NUI APIs interacts with Camera , Sensors and Motors where as you  need a separate set of APIs to take control over Audio Mic Array ( We are not covering Audio array part in this article) . Below picture illustrate overall API Classification for Kinect SDK. image

To start with any Kinect application, you have to Add Microsoft.Research.Kinect.dll as reference to interact with Kinect and first of all we need to  define a Runtime of Kinect Sensors.

image

This will create an instance of Kinect which represent the current device. ( Well, you can interact with multiple Kinect , will talk about this later ) .  Below diagram representing the overall APIs available for  Kinect SDK NUI . Let’s focus on the methods inside Runtime Class. This contain Initialize() and Uninitialize() method along with other methods. This two methods user for initializing and Uninitializing the Kinect Runtime.

image

In my previous post I talked about initializing and uninitializing of Kinect runtime.  Initialize() method Initializes the NUI subsystems that the application requires.  we have initialized the Kinect Runtime with RuntimeOptions as shown in below snippet.

image

RuntimeOptions specifies the runtime options for a Kinect sensor.  Kinect process data using Multi-stage channel. So, specifying runtime tell the what are the channel need to start.

RuntimeOption is a public Enumeration inside the NUI library.

image

UseColor streams the color image from the Kinect and this is used for color Video. UseDepth option stream the depth image data and UseSkeleatal provides the skeleton positions.

When you are done with all operation, call Runtime.Uninitialize() to close the instance.

Well, we learned lot about NUI API. Let’s used some of these API

Demo : NUI API’s Video and Depth Image

Start a new Visual Studio Instance and Create a New WPF Project Template with Name “KinectCameraNUIDemo”.

image

From Solution Explorer, right click on the project and Add “Microsoft.Research.Kinect.dll” as reference assembly

image

you already knows about the initializing the Kinect devices, so I am not discussing on that. Below code snippets shows the basic code blocks which instantiate a Kinect Runtime for the create WPF Application.

 /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        /// <summary>
        /// Runtime Instance for Device
        /// </summary>
        Runtime nuiRuntime = new Runtime();

        /// <summary>
        /// Initializes a new instance of the <see cref="MainWindow"/> class.
        /// </summary>
        public MainWindow()
        {
            InitializeComponent();
            Loaded += new RoutedEventHandler(MainWindow_Loaded);
            Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
        }

        /// <summary>
        /// Handles the Unloaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Unloaded(object sender, RoutedEventArgs e)
        {
            // Uninitialize the Kinect
            nuiRuntime.Uninitialize();
        }

        /// <summary>
        /// Handles the Loaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
           // Initialize the Runtime with Color Channel
            nuiRuntime.Initialize(RuntimeOptions.UseColor);
        }
    }

We are going to display some video capture from Kinect. So, lets first create the UI. The UI is very simple, put one Image and one button control.

image

Below is the XAML markup for the above design.

 <Grid Height="365">
        <Image Height="277" HorizontalAlignment="Left"
               Margin="64,12,0,0"
               Name="videoControl"
               Stretch="Fill"
               VerticalAlignment="Top" Width="385" />
        <Button Content="Capture"
                Height="41" HorizontalAlignment="Left"
                Margin="191,295,0,0" Name="buttonCapture"
                VerticalAlignment="Top" Width="143" />
    </Grid>

You have already initialized the Kinect runtime with UseColor RuntimeOptions. Which means, Kinect already initialized a channel for the color image streaming.  So in the next step you need to write an event handler for video frame. This Occurs when a video frame is ready. Code snippet for the VideoFrameReady event handler look like this :image

Once you have done with signing the video frame, in next step you have to Open a video Stream channel for the image stream.

image

ImageStreamType gets the image stream type. It can be Video or Depth depends the type of event handler we have attached with it. Along with the ImageStreamType you have specify the pool size , resolution and ImageType which is also an enumeration that specifies an image type .

Let’s have handle the event for VideoFrameReady. Kinect  Video returns PlanarImage  and we just need to create new Bitmap source and display on the Image control.

        /// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            PlanarImage planarImage = e.ImageFrame.Image;
            videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
                PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

      }

That’s all, let run the application. and Click on the “Capture Button”, you will find output like below,

image

of course this should be a video display, not still image !!!

Very easy to capture Video. Want to make more easy , yes you can do it using Coding for Fun Kinect Toolkit . This Toolkit contains some useful extension methods for both WinForm and WPF version which will make things much easier.

As for example, using this Coding for Fun Toolkit, you need to write below code block for VideoStream event handler.

  /// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            //PlanarImage planarImage = e.ImageFrame.Image;
            //videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
            //    PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoControl.Source = e.ImageFrame.ToBitmapSource();
      }

but before using Coding for Fun APIs, you have add the “Coding4Fun.Kinect.Wpf.dll ( As this a WPF Application ) as reference and put using Coding4Fun.Kinect.Wpf; as namespaces.

image

Let’s make this demo more interesting, while we are creating a bitmap using BitmapSource.Create() , we are passing the PixelFormats as one of the argument. Let’s change it do something different (Cmyk32) and bind the image to a different Image control.

/// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            PlanarImage planarImage = e.ImageFrame.Image;
            videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
                PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoSource2.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
              PixelFormats.Cmyk32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);
         }

Below is the resulting output :

image

Till now we covered only about the Video images, let’s have a look what happened for Depth Image.  Handling depth stream image is pretty similar to handling the Video steam, the only change required is the type of images and streams.

The first thing you need to change is initialization of runtime. This time you have to use UseDepth as well.

image

Once you are done with initialization, you need to raise an event handler for Depth Stream and need to open a new channel for Depth image where you have to specify the ImageStreamType as Depth and ImageType as Depth.

image

That’s all you really do not need to worry about other stuff. You can put multiple image control and bind different types of images to display it. Yes, its all about fun !!

image

Here is the complete code snippet :


namespace KinectCameraNUIDemo
{
    using System;
    using System.Windows;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;
    using Coding4Fun.Kinect.Wpf;
    using Microsoft.Research.Kinect.Nui;

    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        /// <summary>
        /// Runtime Instance for Device
        /// </summary>
        Runtime nuiRuntime = new Runtime(0);

        /// <summary>
        /// Initializes a new instance of the <see cref="MainWindow"/> class.
        /// </summary>
        public MainWindow()
        {

            InitializeComponent();
            Loaded += new RoutedEventHandler(MainWindow_Loaded);
            Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
        }

        /// <summary>
        /// Handles the Unloaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Unloaded(object sender, RoutedEventArgs e)
        {
            // Uninitialize the Kinect
            nuiRuntime.Uninitialize();
        }

        /// <summary>
        /// Handles the Loaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            // Initialize the Runtime with Color Channel
            nuiRuntime.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);

        }

        /// <summary>
        /// Handles the Click event of the buttonCapture control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonCapture_Click(object sender, RoutedEventArgs e)
        {

            // Signing the Event for Image frame ready
            nuiRuntime.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nuiRuntime_VideoFrameReady);
            nuiRuntime.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nuiRuntime_DepthFrameReady);
            nuiRuntime.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
            nuiRuntime.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);
        }

        void nuiRuntime_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            depthSourceControl.Source = e.ImageFrame.ToBitmapSource();

        }

        /// <summary>
        /// Handles the VideoFrameReady event of the nuiRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void nuiRuntime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            PlanarImage planarImage = e.ImageFrame.Image;
            videoControl.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
                PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoSource2.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
              PixelFormats.Cmyk32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);

            videoSource3.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96,
              PixelFormats.Bgr24, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);
        }
    }
}

Below is the XAML Markup Snippet, I just drag and drop few control .

  <Grid Height="451" Width="607">
         <Image Height="343" HorizontalAlignment="Left"
               Margin="12,12,0,0"
               Name="videoControl"
               Stretch="Fill"
               VerticalAlignment="Top" Width="385" />
        <Button Content="Capture"
                Height="41" HorizontalAlignment="Left"
                Margin="299,376,0,0" Name="buttonCapture"
                VerticalAlignment="Top" Width="143" Click="buttonCapture_Click" />
        <Image Height="150" HorizontalAlignment="Left" Margin="360,23,0,0"
               Name="videoSource2" Stretch="Fill" VerticalAlignment="Top" Width="200" />
        <Image Height="150" HorizontalAlignment="Left" Margin="360,183,0,0"
               Name="depthSourceControl" Stretch="Fill" VerticalAlignment="Top" Width="200" />
        <Image Height="113" HorizontalAlignment="Left" Margin="481,129,0,0"
               Name="videoSource3" Stretch="Fill" VerticalAlignment="Top" Width="114" />
    </Grid>

References :

Coding for Fun
Kinect Documentation

Download Complete Project

In this post we have learned about NUI API’s and learned how we can use those APIs to interact with Kinect Camera and sensors. In the next post I will talk about something more on camera and then we will start with skeleton tracking followed by Audio.

Read Out First Part over here Development With Kinect .NET SDK (Part I) – Installation and Development Environment Setup
Hope this helps !

Cheers !!

AJ

13 thoughts on “Development With Kinect .NET SDK (Part II) – Using NUI APIs with Camera

  1. I tried this example and i get a NullReferenceException for the line nuiRuntime.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);

    Am i missing something ???

    • Hi Ashish,
      Have you define the Runtime Option for DepthStream ( RuntimeOptions.UseDepth) ?

      Check out the initialize option

      nuiRuntime.Initialize( RuntimeOptions.UseColor | RuntimeOptions.UseDepth )

      This mentioned in the section “The first thing you need to change is initialization of runtime. This time you have to use UseDepth as well.”

      Let me know if you need further help !

  2. Hi Great tutorial,
    Ive followed along and i understand how it is everything is working but when i run the program i get an odd issue concerning COM -> the problem itself says
    “Error HRESULT E_FAIL has been returned from a call to a COM component.”

    Now i am using the BETA 2 SDK for the kinect and the only difference between your code in mine is that in BETA 2 the Runtime has changed to allow for 1 or more kinects to be plugged in so my runtime call now looks like
    “Runtime kinectRuntime = Runtime.Kinects[0];” rather than “Runtime kinectRuntime = new Runtime();”
    Have you any idea what may cause this issue?
    Thank you

    • Hi,
      When you are receiving this error ? Please make sure your device is plugged in with external power supply and power is turn on. Kinect camera and sensors needs external power .

      Let me know .

      • Yeah that was the issue, Hadnt it plugged in, When you have it plugged in at USB the green light flashs and i assumed that i had it plugged in at the wall, but alas i did not :P

      • Yeah i had it plugged in via usb and the green light was flashing so i assumed that i had the power adapter plugged in also. So yeah just make sure both usb and power adapter are plugged in. Thank you.

  3. I am working with kinect sdk I opened the kinect sensor in wpf using c# and i designed a button for video capture plz let me know how to capture a video through kinect sensor.

  4. Hello again,
    I’ve contiuned working along your old tutorials and the crux of my problem is still the same. I do not exactely know, what to change in lines 63-64 at the end in the brackets (nuiRuntime_VideoFrameReady) / (nuiRuntime_DepthFrameReady). I’ve tried something from the latest patch-notes here, but it seems not to be the correct thing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s