Tell a lie, I actually have done some work with Kinect since 1.5. I came to the conclusion while doing bounding box area changes that actually grabbing the frame or UI element in a general Win7 desktop isn't a good idea (probably why Win8 touts Metro and touch/gesture so much).
Using skeleton wrist positions, a custom winform, some buttons and position notification objects I threw together a sample for testing. It runs off a main hand and a secondary hand for manipulation. Some words follow:
1: Either hand mouse over UI elements causes them to change visuals as an indication.
2: Main hand hover over elements for X seconds causes them to be "picked up" and centered to the main hand, they can then be dragged about the usable area.
3: Sharp movement of the main hand breaks the pick up calculation and releases the UI element.
4: With a picked up element, moving the second hand into the area causes it to enter a resize mode. Main hand left/right changes width, secondary up/down changes height.
It's just a test setup but it has already pointed me to areas that need thought i.e. you end up crossing hands too much during re-sizing, "dropping" controls is just a side effect of framerates and WinForms aren't really a great way to go here for the UI. Something perhaps in OpenTK might be the better target, but that lends a fun issue of, if this is to be a presentation system, so I end up writing my own readers for document formats etc. Maybe I'll have to look at WPF eventually.
And I'm still kicking around ideas to manage Z axis (where a "proper" 3D environment for the UI would make it excellent).
Showing posts with label kinect. Show all posts
Showing posts with label kinect. Show all posts
Sunday, June 3, 2012
Microsoft.Speech, getting voices in your head
A quick glance through the voice samples of the Kinect shows the Microsoft.Speech namespace. Great, get it working and test a few things out. Fire up some documentation on how to use the Synthesizer, set to default audio and crash. Quick further check, apparently you should be dumping the stream to a SoundPlayer and playing that way. Zero size stream, hrmm.
Turns out all you really need is one of these http://www.microsoft.com/en-us/download/details.aspx?id=27224
Using System.Speech you'll get Microsoft Anna when running GetInstalledVoices(), but nothing when using Microsoft.Speech. Yup, you're missing a voice.
Some of them are a bit hit and miss. Anna isn't very fluid at all, en-GB_Hazel can't say my wifes name yet en-US_ZiraPro can. Oh well. I now know how to do both direct audio output and to a stream for sending somewhere, such as a notification via a custom Lync client.
Speech recognition itself with the grammar builder file is quite nice. I can see myself spending some time building a query/response/action library for automating small activities such as queuing up the latest episode of something (why yes, I mean on youtube, of course), checking for patches for DayZ and download/extract to location, look at my RSS lists and bring new items to my attention and read them out. Having a new small human means this type of thing would really help (and can leverage my non-iDevice environment and actual computing power with ease of customisation).
No further work done on any Kinect projects though. The Lync project I have is almost at a decent beta phase, all the questions from my testing earlier this week have answers now, thanks to a few chats with two very smart chaps at work. The main part left is a decent configuration system and wiring the client to look them all up.
Turns out all you really need is one of these http://www.microsoft.com/en-us/download/details.aspx?id=27224
Using System.Speech you'll get Microsoft Anna when running GetInstalledVoices(), but nothing when using Microsoft.Speech. Yup, you're missing a voice.
Some of them are a bit hit and miss. Anna isn't very fluid at all, en-GB_Hazel can't say my wifes name yet en-US_ZiraPro can. Oh well. I now know how to do both direct audio output and to a stream for sending somewhere, such as a notification via a custom Lync client.
Speech recognition itself with the grammar builder file is quite nice. I can see myself spending some time building a query/response/action library for automating small activities such as queuing up the latest episode of something (why yes, I mean on youtube, of course), checking for patches for DayZ and download/extract to location, look at my RSS lists and bring new items to my attention and read them out. Having a new small human means this type of thing would really help (and can leverage my non-iDevice environment and actual computing power with ease of customisation).
No further work done on any Kinect projects though. The Lync project I have is almost at a decent beta phase, all the questions from my testing earlier this week have answers now, thanks to a few chats with two very smart chaps at work. The main part left is a decent configuration system and wiring the client to look them all up.
Sunday, May 27, 2012
Kinect SDK 1.5
Well it's out, been out for a day or two now with some great new additions such as the face recognition and near mode skeletal tracking. I've not had any time to really play with it and my ideas but I did discover something daft that I should have realised earlier.
Looking at the green screen sample my brain went "hold on, how is it showing the parts that make up the person instead of just the skeleton?" and dove into the code.
Yes, depth data has all the parts of a person. The original SDK had this as well in the colouring of the players depth data. Why didn't I realise this!
So my trials of bounding boxes, while not a total loss, were based on the assumption I'd have to find all the depth points that make up a person myself.
MS also included an excellent PDF link called Human_Interface_Guidelines_v1.5.0. It covers a lot of points from hardware spec, UI design, usability and essentially says "Hey, we did a whole bunch of research with a whole bunch of people, these are the things you should know" which is really nice of MSR (one of my favourite groups out there).
On the Lync front, I've completed the code for my latest project barring some EWS calls. The ideas surrounding it are simple enough but should hopefully revolutionise how we collaborate and "use space" once I get some Kinect integrated. Small steps (and lots of testing).
Looking at the green screen sample my brain went "hold on, how is it showing the parts that make up the person instead of just the skeleton?" and dove into the code.
Yes, depth data has all the parts of a person. The original SDK had this as well in the colouring of the players depth data. Why didn't I realise this!
So my trials of bounding boxes, while not a total loss, were based on the assumption I'd have to find all the depth points that make up a person myself.
MS also included an excellent PDF link called Human_Interface_Guidelines_v1.5.0. It covers a lot of points from hardware spec, UI design, usability and essentially says "Hey, we did a whole bunch of research with a whole bunch of people, these are the things you should know" which is really nice of MSR (one of my favourite groups out there).
On the Lync front, I've completed the code for my latest project barring some EWS calls. The ideas surrounding it are simple enough but should hopefully revolutionise how we collaborate and "use space" once I get some Kinect integrated. Small steps (and lots of testing).
Wednesday, May 16, 2012
Kinect for Windows
Well; I have run through a number of trials and tests using OpenCV and a webcam looking at my own implementation of the sixthsense style of interaction. While the ways of implementing it are there, I sat down with a co-worker one day and realised, why not buy a Kinect? It has less of the over the shoulder portability of a sixthsense system but more of a presentation/presence potential and MS is about to release 1.5 of the SDK.
The ideas I've come up with are interesting, especially since Kinect for Windows is just a bunch of smarts in a box with a camera, IR sensor and IR laser emitter.
Milestone 1 became:
"While the skeleton points are provided easily, I need to locate the object at the point"
After a nice side track off onto depth camera smoothing (a limited implementation of this project which I kind of deduced myself but it's a nice link regardless) and change only depth frame display, I ended up with a quick and dirty depth and skeleton point based bounding box.
Walking points up, left and right using a quick guess at the range of depths to include (to allow for precision problems, angles etc) it gave me a nice framed area for further depth and image mapping.
A happy coincidence was that, when tracking hands, the open palm has a larger area automatically than a closed or half closed hand due to those parts falling outside the allowed +- Z axis. (Here I wrote "Handy!" then slapped myself for the bad pun).
So, Milestone 1 is completed. It's not great but I can make it better and it's required for two of the ideas I'm going to attempt to implement.
Next up, Milestone 2, depth map storing and comparison along with some image recognition.
I'll probably throw my "fun with Lync 2010" things on here as well since that's been driving me up the wall recently.
The ideas I've come up with are interesting, especially since Kinect for Windows is just a bunch of smarts in a box with a camera, IR sensor and IR laser emitter.
Milestone 1 became:
"While the skeleton points are provided easily, I need to locate the object at the point"
After a nice side track off onto depth camera smoothing (a limited implementation of this project which I kind of deduced myself but it's a nice link regardless) and change only depth frame display, I ended up with a quick and dirty depth and skeleton point based bounding box.
Walking points up, left and right using a quick guess at the range of depths to include (to allow for precision problems, angles etc) it gave me a nice framed area for further depth and image mapping.
A happy coincidence was that, when tracking hands, the open palm has a larger area automatically than a closed or half closed hand due to those parts falling outside the allowed +- Z axis. (Here I wrote "Handy!" then slapped myself for the bad pun).
So, Milestone 1 is completed. It's not great but I can make it better and it's required for two of the ideas I'm going to attempt to implement.
Next up, Milestone 2, depth map storing and comparison along with some image recognition.
I'll probably throw my "fun with Lync 2010" things on here as well since that's been driving me up the wall recently.
Subscribe to:
Posts (Atom)