Whoa nelly are there some awesome things to post about today. Kinda/sorta. Actually I'm not allowed to talk about most of them... Which means I'm potentially on the right track for game development if I am being included in hush-hush things. Suffice to say, I really want to spill the beans about awesome stuff, but it's mind over matter, sorry everybody, I can't do that.
In vague terms, though, I can say that Kinect is making leaps and bounds within Zombie Yoga, and as of a few minutes ago I got the new pose system that David and I have dreamed up working full throttle in our C# version of the code, and now we finally have a way to get this code ported over into the game engine so Kinect plays nice with the rest of the game.
And, as I mentioned in my other shorter post, David and I received affirmation that we are not, how you say, doin' it wrong when it comes to the Kinect. Lots of people are hacking away at this thing just to get it to work, and currently the best and easiest way to do this is... however you see fit, however you can fit it, however you can get it. Anyway you want it, that's the way you need it... fill in the rest! And our new system, though not thoroughly tested yet (It's just been me our little (yet huge... it's a funny room) studio apt. trying some whacky poses. So far it's more forgiving (which, when it comes to the Kinect can be a good thing) and easier, which is all that we ever really wanted.
How does our new system work as opposed to our old? Well, for starters our old system was based off of joint proximity, how close one joint was to the other. That's all that's needed, right? Well several problems arise from this, stop me if you've heard this before. Or... I guess skip to the next section unless you can communicate telepathically with me RIGHT NOW. Oops, you are reading this, I posted it, too late.
But seriously, it wasn't a bad system... if you didn't have a lot of poses to check and everyone in the whole wide world was the same size.
We solved a lot of our problems pretty ingeniously, we had a complex yet effective means of eliminating "bad" poses when checking a player's pose, using outlying points to quickly scan through a list of poses and almost recursively loop through down these outlying points into the center positions (hip location and shoulder location, etc), each pass eliminating more and more poses until just one pose remained.
We also had a trick in terms of scaling the poses to be adjusted to the player's individual size. We got a calibration before the player began the game, and compared these limb lengths with that stored in our pose data. Every posed we had queued up then had it's limbs scaled appropriately, according to the calibration we just took and the calibration of the original yogi.
This system, however, only worked in practice. Kinect kept stepping in and kicking our butts, either by having a bad value during calibration (Left Hip was once (-infinity, 5, 5)...) or by it's lovely method of "inferring" joints. Kinect doesn't always know exactly where your joint's are, it gets a rough guesstimate (Better than nothing!) but at times this could result in limb shortening or lengthening in the 3D skeleton data.. which is fine, if everyone's body types got this limb shortening at the same time for the same poses, but they don't. So when the scaling was introduced with certain poses it would screw the rest of the data up by this limb shortening/lengthening putting the Yogi's original joints in strange, wild places.
There's ALSO the issue of player offset. The Kinect returns joint locations as position in 3D Cartesian space (X,Y,Z, think graphing, remember High School math? Yeah, you DO end up using it for something) in terms of meters from the Kinect. Which means if I'm in pose A and I'm a foot to the left of where our Yogi originally had the pose captured, then you would not be in the correct position.
This was a relatively easy fix, if you know vector math or understand 3D coordinate space all we did was make each of the joints relative to the center hip, or took each joint position and subtracted the center hip to make everything in "local space" as opposed to Kinect space.
Add, on top of this, the fact that Kinect loses your joints. This is not a matter of if, but when. The Kinect does great considering the fact that it's pretty much still just reading one dimension. The problem is the minute things (read: limbs) start obscuring body parts, the Kinect beings to give it's best guesses as to where things are, which end up being really awful guesses. So half (more than half, at this point, actually) of our poses involved a very three-dimensional stance. Warrior One, and Warrior Two. Google them. Have safe search on, too. You shouldn't need to, but I found out the hard way the other day that even something as innocuous and/or nerdy as C-strings (an oldschool format for printing lines of text in C) can give you very, very, very uncomfortable results. Here's a hint, don't ever google c-strings.
Holy smokes that's a lot of workarounds Batman! Why in the world did you do that?
Well considering Kinect only gives joint position and not rotation, it appeared to be the most straightforward and logical solution. And in certain circumstances this would be the ideal solution. For a small number of poses, if we had all of the tech we wanted down (appropriate body scaling, effective elimination system and not Kinect giving us random bad data) it actually could have been a good and flexible system. It's highly accurate, pose definitions could be kept as tight;y or as loosely defined as we wanted to specify. It's static, but that's all we needed for poses. Before we got into the muck and mire of Kinect work, it seemed like a flawless plan.
Given what I know now, I'd go back in time and throw my past self out of a very tall building to save him pain, agony, and sleepless nights.
So what's the new system? The new system is a combination of several different things that we thought were "hacks" but tend to actually be industry standard. That is to say fake everything.
We combine a pose flag system, which is just a fancy programmer way of saying we store a lot of true/false (or binary) values in a single integer (4 bytes = 32 bits, each bit can be a 0 or 1, giving us 32 compact and easily comparable true/false values) that tell us information such as
"Are the player's hands above his head?"
"Are the player's hands togther?"
"Is the Right hand forward?"
and etc. to ad infinitum.
(Actually only to 32!)
These are our first test. The really sweet thing about bit flags is that comparison between multiple true/false values is really simple and really fast. So we use the pose flags as our first order of elimination. A pose has a specific set of these very general flags that must be true in order for the player to be in the given pose. Just because the player's left foot and hand are forward doesn't mean that they are in Warrior Two, but it certainly means they aren't in Tree Pose or Warrior One.
This is fast elimination and means we don't have to use that really complicated (and expensive) distance-check for each joint location (lota joints x lota poses = big slow down) To be fair, David had begun implementing the pose flags system in our old game too, but problems plagued that implementation, starting fresh really helped us out.
Then after that we ditch position. Position data means nothing to us, now, we use the positions of joints to get the angular distance between them, and nothing more.
This is achieved by getting the dot product between two normalized vectors. The result is the cosine of the angle between the vectors. We take these dot products at the main joints, the elbows and knees. The second phase is checking the these joint angles (anywhere between 0 degrees, the limb is straight, and 180 degrees, which technically is impossible for the human body, but that's the limb bent all the way back so that, say, your wrist is pointing in the exact opposite direction of your elbow) so that means Warrior Two is defined by the fact that, say, your right hand and leg are forward, your right elbow's angle is 0 degrees (or a dot product of 1) and your right knee is around 90 degrees or so (dot product of 0)
This is a terrible approximation, don't attempt those exact angles or you might suffer some joint strain, but you get the idea.
Now that's getting closer, but as you can imagine that still leaves a fair amount of wiggle room. Or far too much forgiveness. I could turn my torso so that my hands are pointing off at angles, but the angle between my bicep and forearm is still the same amount, and my right hand is still in front of my body, I'm just all twisted up. The last layer on the triple chocolate cake is another dot product, but this time it's in terms of the proximal extremities. Basically where your biceps and quads point. We take these and change them into normalized vectors, and then also store these in our pose file. When you the player are trying to get into a pose, we check the direction of your biceps/quads and see if they match, or are close enough too, the original pose file's directions. This ensures that players are keeping, generally, the correct form and are facing all the correct directions, etc.
There's still wiggle room, but at this point we've eliminated the major stuff. And the nice part about working with pose flags and angles is that they are all relative. Pose flags get set by comparing positions in the skeleton itself. Is the Z value of this greater than the Z value of that? It's relative, it scales with the person playing. Same can be said of the angles. Barring the few possible exceptions, people have the same degrees of motion, and, if you remember trigonometry, angles are independent of size. A 60-30-90 triangle can be infinitesimally small, or colossally large. Same applies to the person on the other end of the Kinect. As long as they fit in the Kinect's FOV (Field Of View) then the angles between their limbs and the directions of their limbs are independent of size and it makes no difference to the Kinect.
We eliminated our scaling and our too-hardcore accuracy problems by a base shift in how we work with the Kinect. Our new system still needs to be thoroughly tested, but it's already been easier to set up and work with than our previous iteration.
Professor Ed Keenan is right, don't be afraid to throw away code, and throw it away often. You build something, it kind of works, delete the code and start over with a better understanding and I guarantee that you will have a better and easier time dealing with the code than if you try and forge through with your old base code. Again, their are exceptions to every rule, but I used to never do that, thinking it was a cardinal sin. Well, I've only just started, but it's helped tremendously.
Delete old code. Re-work it. You get better, faster, and the end result makes more sense.
TL;DR, I know, sorry, no doodles for now, it's 0230, I need to sleep before class tomorrow.
And finish that math homework.
Fiddlesticks.
-Kev
