Using iPi DMC with Dual Kinect Sensors [dKS]

Greenlaw · **Posted:** Fri Oct 14, 2011 12:39 pm

Okay, I understand what I'm looking at now. After calibration is completed, in each frame you should see three 'dots' on the board, which define the angle of the plane to iPi Studio. What you want to do is make sure a majority of the frames contain only green dots. If you have any yellow dots, this means the tracker could not resolve the angle of the plane in that frame, and too many bad frames means your calibration video is poor and motion tracking will fail.

My calibration video is 1400 frames long, which is probably way more data than is required for good results, but luckily I'm only seeing green dots throughout the entire video. (The cats really behaved themselves this morning. Good boys!.)

Yay! Now, on to tracking. :)

Edit: I'm not sure there are yellow dots. I only ever see green dots.

mrbones · **Posted:** Fri Oct 14, 2011 1:00 pm

One note about the RGB visualixation.

It is fantastic and I would suggest making a a switch mode on it.

When you press the RGB it toggles to a material that wraps the depth data with the rgb material information, instead of being a 2D static background, It would become a living breathing animated depth character complete with muscle and cloth deformations...

That would be awesome....

Greenlaw · **Posted:** Fri Oct 14, 2011 1:09 pm

On more tip: Make sure the 'plane' object you use in calibration is perfectly flat. I used a large sheet of foam core board which was slightly warped from standing in the garage all year. I think my results are okay but if the board is warped or bent too much this will obviously affect the quality of the calibration results. (Time to get a new sheet of foam core.)

I have a question: Does the color of the board affect calibration quality? I'm certain a black board will not work since it may absorb the IR rays from the Kinect, but what about a pure white board? I'm asking because the foam core board I'm using it pure white. I think the results I'm getting are okay but I wonder if it would be better to use a board with a neutral color value.

G.

Edit: White board works fine.

Greenlaw · **Posted:** Fri Oct 14, 2011 2:10 pm

Right now I'm running my first tracking passes. Setting up the scene was easy; first I loaded my mocap video data, then the calibration scene settings I created earlier today. Next I loaded my actor from past sessions and clicked the Refit Pose button a few times. Placing the rig inside the T-Posed volume was was not necessary; the actor just snapped into place on its own. The left leg was slightly out of place on the first 'Refit Pose' click but by the third click the rig was perfectly in place.

Next, I clicked the Recenter Origin button to orient the scene and ground plane to the actor's origin. The ground plane appears to be a little high compared to the video reference but maybe that's okay. I proceeded anyway.

I'm seeing that dKS tracking is just slightly slower than with single Kinect tracking, but the time difference is negligible. With a single Kinect, I was getting about 0.67 sec per frame; with dKS I'm seeing about 0.7 sec per frame. I think that's very good for my graphics card. (My graphics card is a Nvidia GTX 460, which these days is probably considered a mid-range card.)

At a glance, the dKS tracking results appears to be more stable than with a single Kinect. I'll know for sure once the tracking process is complete. My video is more than 3700 frames long, so 'first pass' will take about 40 minutes unsupervised. The individual takes will be much, much shorter of course, and most of the 'first pass' data won't be used at all. I just find it easier to let the software do its own thing first before I start paying attention to the details.

More soon....

G.

Greenlaw · **Posted:** Fri Oct 14, 2011 2:15 pm

kristi_12 wrote:

Thank you Greenlaw for all your contributions.

Sure, glad you're finding this info helpful. I know that writing it all down helps me remember what I'm doing. :p

G.

Pat · **Posted:** Fri Oct 14, 2011 4:55 pm

Thanks G! I must be up to date already then.

Pat

Greenlaw · **Posted:** Fri Oct 14, 2011 9:41 pm

The results of my first attempts at tracking dKS data today were a little mixed. It looks like the system did its best to track my data but there were a few problems with this session.

Part of this is my fault: I was standing a little too close to the sofa behind me and I think the tracker had trouble distinguishing my left leg from the sofa, causing the leg to flip out backwards. I was aware of this potential problem from past single Kinect sessions but it was made a little worse today with dKS because this configuration requires a bit more room, which forced me to step back a little further than I did with the single Kinect. I think I can push the two Kinects back much farther if I change their orientation in the room and add another repeater cable.

Another issue is that the calibration result positions the floor a couple of inches higher than normal, which caused issues with the legs. Switching off feet tracking helps a little but this isn't really desirable of course. (In this case it doesn't matter because in this shot you never see my character's feet on the ground, so the data is technically good for me.) I'm not sure why this happened. Maybe the depth sensor is slightly confused between the shiny wooden floor and 'matte' carpet. May try calibration with and without the carpet and see if there's a difference in calibration accuracy.

I have a few other adjustments in mind so I'll post again when I have something new to say.

G.

Greenlaw · **Posted:** Sat Oct 15, 2011 12:36 am

I'm running my second dKS session data now with improved camera placement. Originally, I used the length of the room to space the cameras as wide as possible to increase coverage. The problem with this orientation is that I didn't have a lot of depth data, which affected calibration and tracking accuracy. This time I moved the cameras to the end of the room to maximize the range; the cameras are not as far apart now but I now see a broader range of colors in the depth data.

Calibration result is significantly better; ground plane looks correct now. This time I tracked about 500 frames and that seems plenty. A new feature is the ability to 'hide' the background. Basically, the room gets differenced out using 2 seconds of 'background only' data, which is especially cool to visualize in 3D by rotating the perspective view.

I've attached five screen caps below to illustrate the dKS calibration method:

Image 1. This is what the raw depth data from Kinect 'camera' 1 looks like.

Image 2. This is the same data with the background data 'hidden'.

Image 3. The same data seen from Kinect 'camera' 2.

Image 4. Looking from behind the 3D volume at the two Kinect Sensors.

Image 5. Even better, depth data from Kinect 1 and 2 merged together! This gives you a fairly good idea of the size of the capture space.

BTW, if you look to the left of the board you can see Sergeant Marshmallow sleeping on the couch. :)

I enabled 'Don't Record RGB Stream' for this session because I'm shooting at night and you really can't see much with the low-intensity living room lighting; the extra data would just be wasteful. If the RGB data was recorded, you could see a normal video image in the background or as zoomable PIP images in the upper right corner. I'll do a quick session in the morning with embedded RGB data when the living room lighting is much brighter.

Okay, getting prepared to track the new mocap data now.

G.

Greenlaw · **Posted:** Sat Oct 15, 2011 1:55 am

BTW, I rolled up the carpet that was on the floor and moved it aside. There is a lot of black in the pattern which I think created depth errors for the Kinect. Just one more thing to be aware of.

Greenlaw · **Posted:** Sat Oct 15, 2011 3:02 am

Okay running my 'first pass' tracking for the latest dKS data now. I have to say, I'm very pleased with what I'm seeing so far. Poses that were very problematic with the single Kinect from are being interpreted accurately now. Jitter appears to be less too. I'll know for certain once the first pass is completed.

One thing to pay attention to: earlier this evening I had trouble tracking just the T-Pose. It turns out my calibration scene data was not loaded even though I was sure I had loaded it. This was likely user error but the lesson learned is: if you have trouble tracking even the most basic poses, make sure the calibration data was actually loaded. A quick way to check is to rotate the perspective view and look at the position of the virtual Kinects. If the Kinects are both sharing the same location, then the calibration data obviously was not loaded. You may also notice that the volume data is not merged properly.

G.

Using iPi DMC with Dual Kinect Sensors [dKS]

Who is online