Development With Kinect .NET SDK (Part V) – Developing Application using Multiple Kinect Devices

This is the fifth post in the series of Development With Kinect .NET SDK.  In this post I am going to discuss about interacting with multiple Kinect devices with in a single system using Kinect .NET SDK. Before going forward, I will strongly recommend you to read my previous post where I have discussed about Connecting Multiple Kinect Devices with System  which will help you setup your system with multiple devices.

Once both devices configured and setup properly,  you can run the below code snippet to check if Kinect SDK detects both of  the  devices.

Once you have the correct device count you can start with application development. During this exercise we will be applying all the learning that we have gained over the past few articles.

Let’s start with the a New “WPF” Project and named it as “MultipleKinectDemo”

Navigate to solution explorer, Right Click on the Project and Select “Add Reference” and add “Microsoft.Research.Kinect.dll” as reference assembly.

Add a new class named “KinectDevice”  which will be  a container entity for multiple Kinect devices.   Below is the code snippet for the KinectDevice.cs.

using Microsoft.Research.Kinect.Nui;
/// <summary>
/// Kinect Device
/// </summary>
public sealed class KinectDevice
{
    /// <summary>
    /// Gets or sets the name of the device.
    /// </summary>
    /// <value>The name of the device.</value>
    public string DeviceName { get; set; }

    /// <summary>
    /// Gets or sets the kinect runtime.
    /// </summary>
    /// <value>The kinect runtime.</value>
    public Runtime KinectRuntime { get; set; }
}

In the above class “KinectRuntime” is type of  Kinect Runtime and  We will be creating a List<KinectDevice> to contain the multiple object of devices.

Capturing video, Depth images are same as we did earlier for a single kinect, but the new steps involves with initializing of multiple Kinect devices.   The Kinect .NET SDK does provide support for multiple Kinect devices.   Runtime class has a overloaded constructed where it take index as argument.

You can create a new Runtime object and pass the index of sensors  as   runtimeNui = new Runtime(index).

In our application we will create a List<KinectDevice> and store the  each runtime object  .

Before that, create a basic US as shown in below,

Very straight forward UI, you can check out the XAML markup which in given in then end.

Let’s do it now Step by Step

Detect Kinect Devices 

First of all we need to detect the devices. Kinect SDK APIs provides a class “Device” which represents a system’s Kinect sensors. It’s has a property “Count” which holds the number of Kinect been detected.  Write the blow code snippet in the Detect Button click event.

 /// <summary>
        /// Handles the Click event of the buttonDetect control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonDetect_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                Device device = new Device();
                this.DeviceCount = device.Count;
                this.buttonDetect.Content = string.Format("Kinect Device Detected {0}", this.DeviceCount.ToString());
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Error Occured in Device Detection : ", exp.Message));
            }
        }

this.DeviceCount is a private properties which defined locally to the current class.

        /// <summary>
        /// Gets or sets the device count.
        /// </summary>
        /// <value>The device count.</value>
        private int DeviceCount { get; set; }

After successfully detection of the devices, we changed the content of the Button with number of detected device.

Multiple Kinect Runtime

As discussed earlier, Kinect .SDK support detection of multiple SDK and Runtime class has a overloaded constructor that takes index as arguments.  Below code snippets shows how we can initialize multiple Kinect Runtime

 /// <summary>
        /// Handles the Click event of the buttonInitializeRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonInitializeRuntime_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                for (int i = 0; i < this.DeviceCount; i++)
                {
                    kinectDevices.Add(new KinectDevice
                                        {
                                            KinectRuntime = new Runtime(i),
                                            DeviceName = string.Format("{0} {1}",
                                            DeviceName, i.ToString())
                                        });
                }

                this.buttonKinectOne.IsEnabled = true;
                this.buttonKinectTwo.IsEnabled = true;
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Runtime Initialization Failed : ", exp.Message));
            }
        }

kinectDevices is a List of KinectDevice which I have discussed earlier.

        /// <summary>
        /// Kinect Device Place Holder
        /// </summary>
        List<KinectDevice> kinectDevices = new List<KinectDevice>();

Taking Control over Individual Kinect  

Once we have the runtime initialized, we can easily take control over the individual Kinect. kinectDevices[0] and kinectDevices[1] are the runtime container for individual Kinect. Below code snippets show how we can detect the first Kinect devices.

  /// <summary>
        /// Handles the Click event of the buttonKinectOne control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonKinectOne_Click(object sender, RoutedEventArgs e)
        {
            KinectDevice deviceOne = kinectDevices[0];
            Runtime deviceRuntimeOne = deviceOne.KinectRuntime;
            deviceRuntimeOne.Initialize(RuntimeOptions.UseColor);
            deviceRuntimeOne.NuiCamera.ElevationAngle = 0;
            deviceName1.Content = deviceOne.DeviceName;
            deviceRuntimeOne.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(deviceRuntimeOne_VideoFrameReady);
            deviceRuntimeOne.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
        }

As we have the specific Kinect runtime, we can easily take care of VideoFrameReady event and  VideoStream.Open() method.

 /// <summary>
        /// Handles the VideoFrameReady event of the deviceRuntimeOnce control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void deviceRuntimeOne_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            image1.Source = e.ImageFrame.ToBitmapSource();
            PlanarImage imageData = e.ImageFrame.Image;
            camera1Small.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Cmyk32, null, imageData.Bits, imageData.Width * imageData.BytesPerPixel);
        }

Similar we can write the code for kinectDevices[1]

Well, this is the base, rest things are very similar to work with single Kinect device.

Here is the complete code snippet of the application

/// Developing Application using Multiple Kinect
/// Author : Abhijit Jana
/// Date : 17 - 09 2011
/// ---------------------------------------------------------------------------

namespace MultipleKinectDemo
{
    using System;
    using System.Collections.Generic;
    using System.Windows;
    using Coding4Fun.Kinect.Wpf;
    using Microsoft.Research.Kinect.Nui;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;

    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        /// <summary>
        /// Gets or sets the device count.
        /// </summary>
        /// <value>The device count.</value>
        private int DeviceCount { get; set; }

        /// <summary>
        /// Kinect Device Place Holder
        /// </summary>
        List<KinectDevice> kinectDevices = new List<KinectDevice>();

        private const string DeviceName = "Kinect Sensors Device";

        /// <summary>
        /// Initializes a new instance of the <see cref="MainWindow"/> class.
        /// </summary>
        public MainWindow()
        {
            InitializeComponent();
            Loaded += new RoutedEventHandler(MainWindow_Loaded);
            Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
        }

        /// <summary>
        /// Handles the Unloaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Unloaded(object sender, RoutedEventArgs e)
        {
            foreach (KinectDevice device in kinectDevices)
            {
                device.KinectRuntime.Uninitialize();
            }
        }

        /// <summary>
        /// Handles the Loaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            this.buttonKinectOne.IsEnabled = false;
            this.buttonKinectTwo.IsEnabled = false;
        }

        /// <summary>
        /// Handles the Click event of the buttonDetect control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonDetect_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                Device device = new Device();
                this.DeviceCount = device.Count;
                this.buttonDetect.Content = string.Format("Kinect Device Detected {0}", this.DeviceCount.ToString());
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Error Occured in Device Detection : ", exp.Message));
            }
        }

        /// <summary>
        /// Handles the Click event of the buttonInitializeRuntime control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonInitializeRuntime_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                for (int i = 0; i < this.DeviceCount; i++)
                {
                    kinectDevices.Add(new KinectDevice
                                        {
                                            KinectRuntime = new Runtime(i),
                                            DeviceName = string.Format("{0} {1}",
                                            DeviceName, i.ToString())
                                        });
                }

                this.buttonKinectOne.IsEnabled = true;
                this.buttonKinectTwo.IsEnabled = true;
            }
            catch (Exception exp)
            {
                MessageBox.Show(string.Format("Runtime Initialization Failed : ", exp.Message));
            }
        }

        /// <summary>
        /// Handles the Click event of the buttonKinectOne control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonKinectOne_Click(object sender, RoutedEventArgs e)
        {
            KinectDevice deviceOne = kinectDevices[0];
            Runtime deviceRuntimeOne = deviceOne.KinectRuntime;
            deviceRuntimeOne.Initialize(RuntimeOptions.UseColor);
            deviceRuntimeOne.NuiCamera.ElevationAngle = 0;
            deviceName1.Content = deviceOne.DeviceName;
            deviceRuntimeOne.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(deviceRuntimeOne_VideoFrameReady);
            deviceRuntimeOne.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
        }

        /// <summary>
        /// Handles the VideoFrameReady event of the deviceRuntimeOnce control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void deviceRuntimeOne_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            image1.Source = e.ImageFrame.ToBitmapSource();
            PlanarImage imageData = e.ImageFrame.Image;
            camera1Small.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Cmyk32, null, imageData.Bits, imageData.Width * imageData.BytesPerPixel);
        }

        /// <summary>
        /// Handles the Click event of the buttonKinectTwo control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void buttonKinectTwo_Click(object sender, RoutedEventArgs e)
        {
            KinectDevice deviceTwo = kinectDevices[1];
            Runtime deviceRuntimeTwo = deviceTwo.KinectRuntime;
            deviceRuntimeTwo.Initialize(RuntimeOptions.UseColor);
            deviceRuntimeTwo.NuiCamera.ElevationAngle = 0;
            deviceName2.Content = deviceTwo.DeviceName;
            deviceRuntimeTwo.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(deviceRuntimeTwo_VideoFrameReady);
            deviceRuntimeTwo.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
        }

        /// <summary>
        /// Handles the VideoFrameReady event of the deviceRuntimeTwo control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="Microsoft.Research.Kinect.Nui.ImageFrameReadyEventArgs"/> instance containing the event data.</param>
        void deviceRuntimeTwo_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            image2.Source = e.ImageFrame.ToBitmapSource();
            PlanarImage imageData = e.ImageFrame.Image;
            camera2Small.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Cmyk32, null, imageData.Bits, imageData.Width * imageData.BytesPerPixel);
        }

        /// <summary>
        /// Handles the Click event of the camera1Up control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera1Up_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle + 5;
            }
            catch (ArgumentOutOfRangeException aore)
            {

                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }

            catch (Exception)
            {

            }

        }

        /// <summary>
        /// Handles the Click event of the camera1Down control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera1Down_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[0].KinectRuntime.NuiCamera.ElevationAngle - 5;
            }
            catch (ArgumentOutOfRangeException argumentExcpetion)
            {
                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }
        }

        /// <summary>
        /// Handles the Click event of the camera2Up control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera2Up_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle + 5;
            }
            catch (ArgumentOutOfRangeException argumentExcpetion)
            {

                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }
        }

        /// <summary>
        /// Handles the Click event of the camera2Down control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.Windows.RoutedEventArgs"/> instance containing the event data.</param>
        private void camera2Down_Click(object sender, RoutedEventArgs e)
        {
            try
            {
                kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle = kinectDevices[1].KinectRuntime.NuiCamera.ElevationAngle - 5;
            }
            catch (ArgumentOutOfRangeException aore)
            {
                MessageBox.Show("Elevation angle must be between Elevation Minimum/Maximum");
            }
        }
    }
}

XAML Markup. just drag and drop control 😀

<Grid>
        <Border BorderBrush="Silver" BorderThickness="5" Height="257" HorizontalAlignment="Left"
                Margin="32,207,0,0" Name="border1" VerticalAlignment="Top" Width="269"></Border>
        <Border BorderBrush="Silver" BorderThickness="5" Height="257" Name="border2" Width="269" Margin="488,206,46,90">
            <Image Height="246" Name="image2" Stretch="Fill" Width="259" />
        </Border>
        <Button Content="Kinect 1" Height="42" HorizontalAlignment="Left" Margin="35,480,0,0" Name="buttonKinectOne" VerticalAlignment="Top" Width="266" Click="buttonKinectOne_Click" />
        <Button Content="Kinect 2" Height="42" HorizontalAlignment="Right" Margin="0,480,51,0" Name="buttonKinectTwo" VerticalAlignment="Top" Width="259" Click="buttonKinectTwo_Click" />
        <Button Content="Detect Kinects" Height="40" HorizontalAlignment="Left" Margin="243,12,0,0" Name="buttonDetect" VerticalAlignment="Top" Width="280" Click="buttonDetect_Click" />
        <Label Content="Camera Device Name" Height="26" HorizontalAlignment="Left" Margin="32,180,0,0" Name="deviceName1" VerticalAlignment="Top" Width="271" />
        <Label Content="Camera Device Name" Height="26" HorizontalAlignment="Left" Margin="486,180,0,0" Name="deviceName2" VerticalAlignment="Top" Width="271" />
        <Image Height="249" Name="image1" Stretch="Fill" Margin="35,211,502,93" />
        <Button Content="Initialize Runtime" Height="38" HorizontalAlignment="Left" Margin="243,67,0,0" Name="buttonInitializeRuntime" VerticalAlignment="Top" Width="280" Click="buttonInitializeRuntime_Click" />
        <Button Content="UP" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="307,356,0,0" Name="camera1Up" VerticalAlignment="Top" Width="41" Click="camera1Up_Click" />
        <Button Content="Down" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="307,413,0,0" Name="camera1Down" VerticalAlignment="Top" Width="41" Click="camera1Down_Click" />
        <Image Height="109" HorizontalAlignment="Left" Margin="254,228,0,0" Name="camera1Small" Stretch="Fill" VerticalAlignment="Top" Width="127" />
        <Border BorderBrush="#FFB10E00" BorderThickness="1" Height="121" HorizontalAlignment="Left" Margin="248,222,0,0" Name="border3" VerticalAlignment="Top" Width="140"></Border>
        <Button Content="UP" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="441,356,0,0" Name="camera2Up" VerticalAlignment="Top" Width="41" Click="camera2Up_Click" />
        <Button Content="Down" FontWeight="ExtraBold" Height="43" HorizontalAlignment="Left" Margin="441,413,0,0" Name="camera2Down" VerticalAlignment="Top" Width="41" Click="camera2Down_Click" />
        <Border BorderBrush="#FFB10E00" BorderThickness="1" Height="121" Name="border4" Width="140" Margin="402,222,261,210">
            <Image Height="118" Name="camera2Small" Stretch="Fill" Width="133" />
        </Border>
    </Grid>

Here is the output of the above application.

Download Complete Project

Hope this helps !
Cheers !
Aj

My Twitter Handler : @abhijitjana
Email Me : abhijitjana@outlook.com
My Kinect Book : Kinect for Windows SDK Programming Guide