Development With Kinect .NET SDK (Part V) – Developing Application using Multiple Kinect Devices

This is the fifth post in the series of Development With Kinect .NET SDK.  In this post I am going to discuss about interacting with multiple Kinect devices with in a single system using Kinect .NET SDK. Before going forward, I will strongly recommend you to read my previous post where I have discussed about Connecting Multiple Kinect Devices with System  which will help you setup your system with multiple devices.

Once both devices configured and setup properly,  you can run the below code snippet to check if Kinect SDK detects both of  the  devices.

image

Once you have the correct device count you can start with application development. During this exercise we will be applying all the learning that we have gained over the past few articles.

Let’s start with the a New “WPF” Project and named it as “MultipleKinectDemo

image

Navigate to solution explorer, Right Click on the Project and Select “Add Reference” and add “Microsoft.Research.Kinect.dll” as reference assembly.

image

Add a new class named “KinectDevice”  which will be  a container entity for multiple Kinect devices.   Below is the code snippet for the KinectDevice.cs.

using Microsoft.Research.Kinect.Nui;
/// <summary>
/// Kinect Device
/// </summary>
public sealed class KinectDevice
{
    /// <summary>
    /// Gets or sets the name of the device.
    /// </summary>
    /// <value>The name of the device.</value>
    public string DeviceName { get; set; }

    /// <summary>
    /// Gets or sets the kinect runtime.
    /// </summary>
    /// <value>The kinect runtime.</value>
    public Runtime KinectRuntime { get; set; }
}

In the above class “KinectRuntime” is type of  Kinect Runtime and  We will be creating a List<KinectDevice> to contain the multiple object of devices.

image

Capturing video, Depth images are same as we did earlier for a single kinect, but the new steps involves with initializing of multiple Kinect devices.   The Kinect .NET SDK does provide support for multiple Kinect devices.   Runtime class has a overloaded constructed where it take index as argument.

image

You can create a new Runtime object and pass the index of sensors  as   runtimeNui = new Runtime(index).

In our application we will create a List<KinectDevice> and store the  each runtime object  .

Before that, create a basic US as shown in below,

image

Very straight forward UI, you can check out the XAML markup which in given in then end.

Let’s do it now Step by Step

Detect Kinect Devices 

First of all we need to detect the devices. Kinect SDK APIs provides a class “Device” which represents a system’s Kinect sensors. It’s has a property “Count” which holds the number of Kinect been detected.  Write the blow code snippet in the Detect Button click event.

 /// <summary>
        /// Handles the Click event of the buttonDetect control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonDetect_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                Device device = new Device();
                this.DeviceCount = device.Count;
                this.buttonDetect.Content = string.Format("Kinect Device Detected {0}", this.DeviceCount.ToString());
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Error Occured in Device Detection : ", exp.Message));
            }
        }

this.DeviceCount is a private properties which defined locally to the current class.

        /// <summary>
        /// Gets or sets the device count.
        /// </summary>
        /// <value>The device count.</value>
        private int DeviceCount { get; set; }

After successfully detection of the devices, we changed the content of the Button with number of detected device.

image

Multiple Kinect Runtime

As discussed earlier, Kinect .SDK support detection of multiple SDK and Runtime class has a overloaded constructor that takes index as arguments.  Below code snippets shows how we can initialize multiple Kinect Runtime

 /// <summary>
        /// Handles the Click event of the buttonInitializeRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonInitializeRuntime_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                for (int i = 0; i < this.DeviceCount; i++)
                {
                    kinectDevices.Add(new KinectDevice
                                        {
                                            KinectRuntime = new Runtime(i),
                                            DeviceName = string.Format("{0} {1}",
                                            DeviceName, i.ToString())
                                        });
                }

                this.buttonKinectOne.IsEnabled = true;
                this.buttonKinectTwo.IsEnabled = true;
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Runtime Initialization Failed : ", exp.Message));
            }
        }

kinectDevices is a List of KinectDevice which I have discussed earlier.

        /// <summary>
        /// Kinect Device Place Holder
        /// </summary>
        List<KinectDevice> kinectDevices = new List<KinectDevice>();

Taking Control over Individual Kinect  

Once we have the runtime initialized, we can easily take control over the individual Kinect. kinectDevices[0] and kinectDevices[1] are the runtime container for individual Kinect. Below code snippets show how we can detect the first Kinect devices.

  /// <summary>
        /// Handles the Click event of the buttonKinectOne control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonKinectOne_Click(object sender, RoutedEventArgs e)
        {
            KinectDevice deviceOne = kinectDevices[0];
            Runtime deviceRuntimeOne = deviceOne.KinectRuntime;
            deviceRuntimeOne.Initialize(RuntimeOptions.UseColor);
            deviceRuntimeOne.NuiCamera.ElevationAngle = 0;
            deviceName1.Content = deviceOne.DeviceName;
            deviceRuntimeOne.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(deviceRuntimeOne_VideoFrameReady);
            deviceRuntimeOne.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
        }

As we have the specific Kinect runtime, we can easily take care of VideoFrameReady event and  VideoStream.Open() method.

 /// <summary>
        /// Handles the VideoFrameReady event of the deviceRuntimeOnce control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void deviceRuntimeOne_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            image1.Source = e.ImageFrame.ToBitmapSource();
            PlanarImage imageData = e.ImageFrame.Image;
            camera1Small.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Cmyk32, null, imageData.Bits, imageData.Width * imageData.BytesPerPixel);
        }

Similar we can write the code for kinectDevices[1]

Well, this is the base, rest things are very similar to work with single Kinect device.

Here is the complete code snippet of the application

/// Developing Application using Multiple Kinect
/// Author : Abhijit Jana
/// Date : 17 - 09 2011
/// ---------------------------------------------------------------------------

namespace MultipleKinectDemo
{
    using System;
    using System.Collections.Generic;
    using System.Windows;
    using Coding4Fun.Kinect.Wpf;
    using Microsoft.Research.Kinect.Nui;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;

    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        /// <summary>
        /// Gets or sets the device count.
        /// </summary>
        /// <value>The device count.</value>
        private int DeviceCount { get; set; }

        /// <summary>
        /// Kinect Device Place Holder
        /// </summary>
        List<KinectDevice> kinectDevices = new List<KinectDevice>();

        private const string DeviceName = "Kinect Sensors Device";

        /// <summary>
        /// Initializes a new instance of the <see cref="MainWindow"/> class.
        /// </summary>
        public MainWindow()
        {
            InitializeComponent();
            Loaded += new RoutedEventHandler(MainWindow_Loaded);
            Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
        }

        /// <summary>
        /// Handles the Unloaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Unloaded(object sender, RoutedEventArgs e)
        {
            foreach (KinectDevice device in kinectDevices)
            {
                device.KinectRuntime.Uninitialize();
            }
        }

        /// <summary>
        /// Handles the Loaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            this.buttonKinectOne.IsEnabled = false;
            this.buttonKinectTwo.IsEnabled = false;
        }

        /// <summary>
        /// Handles the Click event of the buttonDetect control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonDetect_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                Device device = new Device();
                this.DeviceCount = device.Count;
                this.buttonDetect.Content = string.Format("Kinect Device Detected {0}", this.DeviceCount.ToString());
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Error Occured in Device Detection : ", exp.Message));
            }
        }

        /// <summary>
        /// Handles the Click event of the buttonInitializeRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonInitializeRuntime_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                for (int i = 0; i < this.DeviceCount; i++)
                {
                    kinectDevices.Add(new KinectDevice
                                        {
                                            KinectRuntime = new Runtime(i),
                                            DeviceName = string.Format("{0} {1}",
                                            DeviceName, i.ToString())
                                        });
                }

                this.buttonKinectOne.IsEnabled = true;
                this.buttonKinectTwo.IsEnabled = true;
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Runtime Initialization Failed : ", exp.Message));
            }
        }

        /// <summary>
        /// Handles the Click event of the buttonKinectOne control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonKinectOne_Click(object sender, RoutedEventArgs e)
        {
            KinectDevice deviceOne = kinectDevices[0];
            Runtime deviceRuntimeOne = deviceOne.KinectRuntime;
            deviceRuntimeOne.Initialize(RuntimeOptions.UseColor);
            deviceRuntimeOne.NuiCamera.ElevationAngle = 0;
            deviceName1.Content = deviceOne.DeviceName;
            deviceRuntimeOne.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(deviceRuntimeOne_VideoFrameReady);
            deviceRuntimeOne.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
        }

        /// <summary>
        /// Handles the VideoFrameReady event of the deviceRuntimeOnce control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void deviceRuntimeOne_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            image1.Source = e.ImageFrame.ToBitmapSource();
            PlanarImage imageData = e.ImageFrame.Image;
            camera1Small.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Cmyk32, null, imageData.Bits, imageData.Width * imageData.BytesPerPixel);
        }

        /// <summary>
        /// Handles the Click event of the buttonKinectTwo control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonKinectTwo_Click(object sender, RoutedEventArgs e)
        {
            KinectDevice deviceTwo = kinectDevices[1];
            Runtime deviceRuntimeTwo = deviceTwo.KinectRuntime;
            deviceRuntimeTwo.Initialize(RuntimeOptions.UseColor);
            deviceRuntimeTwo.NuiCamera.ElevationAngle = 0;
            deviceName2.Content = deviceTwo.DeviceName;
            deviceRuntimeTwo.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(deviceRuntimeTwo_VideoFrameReady);
            deviceRuntimeTwo.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
        }

        /// <summary>
        /// Handles the VideoFrameReady event of the deviceRuntimeTwo control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void deviceRuntimeTwo_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            image2.Source = e.ImageFrame.ToBitmapSource();
            PlanarImage imageData = e.ImageFrame.Image;
            camera2Small.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Cmyk32, null, imageData.Bits, imageData.Width * imageData.BytesPerPixel);
        }

        /// <summary>
        /// Handles the Click event of the camera1Up control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera1Up_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle + 5;
            }
            catch (ArgumentOutOfRangeException aore)
            {

                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }

            catch (Exception)
            {

            }

        }

        /// <summary>
        /// Handles the Click event of the camera1Down control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera1Down_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle - 5;
            }
            catch (ArgumentOutOfRangeException argumentExcpetion)
            {
                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }
        }

        /// <summary>
        /// Handles the Click event of the camera2Up control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera2Up_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle + 5;
            }
            catch (ArgumentOutOfRangeException argumentExcpetion)
            {

                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }
        }

        /// <summary>
        /// Handles the Click event of the camera2Down control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera2Down_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle - 5;
            }
            catch (ArgumentOutOfRangeException aore)
            {
                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }
        }
    }
}

XAML Markup. just drag and drop control :D

<Grid>
        <Border BorderBrush="Silver" BorderThickness="5" Height="257" HorizontalAlignment="Left"
                Margin="32,207,0,0" Name="border1" VerticalAlignment="Top" Width="269"></Border>
        <Border BorderBrush="Silver" BorderThickness="5" Height="257" Name="border2" Width="269" Margin="488,206,46,90">
            <Image Height="246" Name="image2" Stretch="Fill" Width="259" />
        </Border>
        <Button Content="Kinect 1" Height="42" HorizontalAlignment="Left" Margin="35,480,0,0" Name="buttonKinectOne" VerticalAlignment="Top" Width="266" Click="buttonKinectOne_Click" />
        <Button Content="Kinect 2" Height="42" HorizontalAlignment="Right" Margin="0,480,51,0" Name="buttonKinectTwo" VerticalAlignment="Top" Width="259" Click="buttonKinectTwo_Click" />
        <Button Content="Detect Kinects" Height="40" HorizontalAlignment="Left" Margin="243,12,0,0" Name="buttonDetect" VerticalAlignment="Top" Width="280" Click="buttonDetect_Click" />
        <Label Content="Camera Device Name" Height="26" HorizontalAlignment="Left" Margin="32,180,0,0" Name="deviceName1" VerticalAlignment="Top" Width="271" />
        <Label Content="Camera Device Name" Height="26" HorizontalAlignment="Left" Margin="486,180,0,0" Name="deviceName2" VerticalAlignment="Top" Width="271" />
        <Image Height="249" Name="image1" Stretch="Fill" Margin="35,211,502,93" />
        <Button Content="Initialize Runtime" Height="38" HorizontalAlignment="Left" Margin="243,67,0,0" Name="buttonInitializeRuntime" VerticalAlignment="Top" Width="280" Click="buttonInitializeRuntime_Click" />
        <Button Content="UP" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="307,356,0,0" Name="camera1Up" VerticalAlignment="Top" Width="41" Click="camera1Up_Click" />
        <Button Content="Down" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="307,413,0,0" Name="camera1Down" VerticalAlignment="Top" Width="41" Click="camera1Down_Click" />
        <Image Height="109" HorizontalAlignment="Left" Margin="254,228,0,0" Name="camera1Small" Stretch="Fill" VerticalAlignment="Top" Width="127" />
        <Border BorderBrush="#FFB10E00" BorderThickness="1" Height="121" HorizontalAlignment="Left" Margin="248,222,0,0" Name="border3" VerticalAlignment="Top" Width="140"></Border>
        <Button Content="UP" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="441,356,0,0" Name="camera2Up" VerticalAlignment="Top" Width="41" Click="camera2Up_Click" />
        <Button Content="Down" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="441,413,0,0" Name="camera2Down" VerticalAlignment="Top" Width="41" Click="camera2Down_Click" />
        <Border BorderBrush="#FFB10E00" BorderThickness="1" Height="121" Name="border4" Width="140" Margin="402,222,261,210">
            <Image Height="118" Name="camera2Small" Stretch="Fill" Width="133" />
        </Border>
    </Grid>

Here is the output of the above application.

image

Download Complete Project

Hope this helps !
Cheers !
Aj

My Twitter Handler : @abhijitjana
Email Me : abhijitjana@outlook.com
My Kinect Book : Kinect for Windows SDK Programming Guide

KinectBookLogo

24 thoughts on “Development With Kinect .NET SDK (Part V) – Developing Application using Multiple Kinect Devices

  1. Nice post! But i have a question. How come those two kinect are working in the same time? I’m trying to use 2 Kinect’s but one of this camera doesn’t starts? (device manager cod 10). do you know a solution?
    thx

  2. thank you for your good posts.

    excuse me ! you said in previous posts you discuss about skeleton tracking but don’t do that! please help us on it too.

    tanx a lot mr.

    • Thanks for the posts! They have been very helpful. I am having trouble with the “Device” class though. I installed the KinectSDK-v1.0-beta2-x64 bits and it seems the “Nui.Device” calss is missing! I am wondering what SDK version you have installed.

      Thanks a ton!

      Matt

      • I have figured it out. I am using the new Kinect SDK version and it does not have the Device class anymore. You need to use Runtime.Kinects.Count instead. Also, you need to download the newest Coding4Fun Kinect toolkit and reference that for it to work again…

        Thanks again for these great articles!
        Matt

      • Hi Mat,
        Yes, you are working on SDK Beta 2. Where as this post was written on SDK Beta 1. and there are some changes in APIs between Beta 1 and Beta 2.

        I am planning to update all the article in coming months.

        Cheers !
        Abhijit

  3. I appreciate your help for these five posts. Thank you very much.I hava some problems about using Multiple Kinect Devices, I wish you can give me the answers.

    1.In your MultipleKinectDemo,I found you just use color channel of each kinect,I want to know if i can use two depth senors can two color senors at the same time.
    2.When I use two kincets on the same system,can I get the two image at the same time.That means is there any sync signal I can get.

    Because my laptop just hava one USB controller, I can not try my best to figured these out.Thanks again for these great posts!

    zhangyuting

    • Hi.. you can use the multiple devices for both color and depth sensor at same time. But, only one Kinect sensor can track the skeleton. But, you can change which sensor to track skeleton using code, but at a time one will track. Let me know if you need further help !!

    • No John. It’s not updated for V1. In V1, it’s very simple, you can just use KinectSensors.KinectSensor[index] to get the device and there is also an KinetSensorCollection Class.

  4. Hi Abhijit, may i know what type of docking station you used. I already bought Kensington docking station but cannot work with my panasonic and ienovo laptop.Thanks!

    • Hi Kun,
      Configuration and Set up will be same. You computer must have 3 different USB controller and each of the Kinect should have connected with individual.

      Now, using KinectSensorCollection class or KinectSensor.KinectSensors[index] / [id] you can take your development further.

      Let me know if you need further help.

  5. Thanks for this information. You are very awesome guy. My team try to use 2 Kinects for the first time. Hope to keep in touch with you later.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s