I am trying to capture a dance session with 6 PS move cameras.

In general the tracking works well, but at some points, especially when the dancer's arms are extended, the tracking algorithm will flip the arm rotations for no particular reason. For example, the first screenshot below is a correct pose: notice the dancer's left palm is facing down. Two frames later, the palm is now incorrectly facing up. This happens about 5 to 10 times in a 10 second part of the dance that includes extended arms.

Even if I manually change the position and refit it, then continue tracking from there, at some point the arm will flip randomly again.

I understand that an extended arm can be ambiguous, but this is not an acceptable result. Is there a way to tell the software not to flip the arm randomly?

Hi, thanks for report
We're aware of this issue. You're right that it is difficult to find the twist accurately when arm is nearly straight. We'll try to decrease the negative effect of this inaccuracy in future versions.
Also it would be great if you could share the project displaying this issue with us. We'll use it for testing purposes only and keep confidential. We'll need both .iPiVideo and .iPiMotion files. You can use our tech support e-mail to send files or file-sharing links.

...

Yes, it happens, but the use of a light material colored glove, I use black, will help... I see she has a hand color that may get confused with the background, or hidden within the lighting shown in the images.

Just a suggestion, but white/yellow/and natural hand color of really any natural skin tone doesn't work best for hands IMO, but will work ok for the arm color, just being a bigger area of color for the tracker to follow I imagine, she also has black sleeves with a lot of black behind her, usually not a big issue, but could have some effect.

The use of hand Move controllers are highly recommended for use, when you use them then add the data after tracking, during the Refine process the wrists will maintain the correct rotations and the arms will better adjust themselves, then re run the apply Move data after also, Moves data should always be the very last process run before exporting the animation also.

Her arm skeleton bone is a bit short to her actual arm length also, make wrist joint better match her wrist joint and looks like something is wrong with the height match, as the head bone is bending weirdly when her head is upright, all this stuff matters for better tracking.

Dropping leg length to where hip bone is where her leg bend crease is will also help and adjust other bone according to that, I see you have it set a bit too high to me.

You may never get zero issues with the arms, but gloves will help, when it does happen, you will have to go to the frame and pull the IK handle a bit off the video position, (maybe more than once), before hitting refit, or the arm won't change position to refit, same with a leg, arrow key next to the ? to frame forward and repeat when necessary, arm should refit to proper positioning when done right, but if use a Move later in hands, you will need to re run the data before using Refine again.

Another thing is when you set the cameras parameters, you can turn down the gain a bit in a well lit room to get a bit more contrast of performer to background.

Hope any of this helps.

Good Luck!

...

Thanks for your reply. I am sending you the affected files via email.

I have some additional questions:

1. In principle having inertial sensors in the hands should fix the disambiguation, but I understand that this will only affect the wrist bone, and no inverse kinematics are applied to the arm to get to a realistic arm twist. Is this correct? In my case, the 3D model for this capture has long sleeves, and it is important for the sleeve to correctly point down, so if I were to use an inertial sensor, I would need for it to correctly solve the arm so it points in the correct direction, not just the hand.

2. I made a pair of gloves with the inertial sensors in the back of the hands, but since the gloves are black and fingerless, and the long sleeves are also black, wouldn't this interfere with the feature detection of iPi? I understand the hands need to be of one color, and a different color from the sleeves.

3. I was trying to use the inertial sensors (PS move), but they start to drift very quickly. I understand new controllers don't have a magnetometer. I got them second hand in mid 2016, but the way in which I fixed the controllers to the gloves doesn't let me read the serial number, and disassembling it will probably damage the gloves at this point. Is there another way to know if my inertial sensors have a magnetometer?

4. For some reason, the head is always kinda pointing up. Is there a way to get a better head tracking?

...

What 3D editing program are you using?

In order of questions:

1) The model you create will control the sleeves by the weighting to the particular bone, since iPi doesn't use twist bones in their retargeting, the Moves will only control the rotations for the hands, that is correct.
If your model skeleton has twist bones for the wrist, you can then manually adjust the wrist rotations when needed, iPi tracking is just not as accurate for slight motion as marker systems in that regard, but it is close enough for most actions.

2) The black gloves would be best, I haven't used a black sleeve with black gloves, easy to change long shirt color if used, yes? (I stopped using long sleeve shirt at all, just use arm skin color now), but I do use solid black pants with solid black shoes, or only black socks without issue, I guess a test if black on black arms/hands would work similarly would be the only way for you to personally find out.
Many times you don't need to worry so much about the arm twist if just slightly off while tracking, with the Moves after you finish tracking the video, apply their data, then run the Refine process, don't worry if you see the hand flip backwards while running it, when it finishes re-apply the Move data again and it should correct many of the issues you saw prior, or limit them much better, but you may still need to do a slow-mo scrub of the timeline and fix anything that needs it, many times just running running Refine again back and forth a few times over those areas will correct it, but don't run the jitter removal at all until you are happy with the tracking and refine.

3) Without being able to see the MPN long number on the sticker, there is no real way to tell, but drifting is a sure sign they are newer ones without the magnetometer, and usually I hear those start to drift after 30 secs. and no way to really stop it... This drifting I don't experience, but I am sure anything I state would be less effective with drift.
The MPN that has the mag. is lower MPN number starting with 200 or 220, possibly up to 300, but anything starting with 400 and up is new with no mag, older models which are hard to find now are out there on eBay, price varies, but be sure to ask what MPN from seller before purchase. It is the really long number at the bottom of the rear sticker.

4) Head looking up a bit is common without a head Move, you can try being sure the performer always starts off looking at something lower, human nature is to hold head tilted back a bit when looking directly forward, or try a bit taller Actor size, 1-2 cm, or pulling the head wrap used a bit lower on the brow, but remember to always have the performers rear of neck exposed fully, or neck tracking will suffer and again, in my opinion head tracking should always be turned on while tracking the PS Eye video for better results, you can always use a layer in post to adjust the head from frame 1 throughout, so shouldn't be that big of an issue to fix later.

5) When using the hand Moves, it is also very important to set the Move on the Actor in iPi and the hand of the Actor very close to what the video shows before applying the data to get more accurate rotations, (you can adjust the hand, or the Moves positioning at anytime for better results, doing this is non destructive to the already refined animations, but the arm must be the right length of the performer also, then always run Moves data as very last step before any exporting.

If you are new to using iPi, there are certain things you will learn through trial and error, there are just too many varying ways users run their own processes, I am just stating some of the ways I found helpful for me in my studio set up, yours may vary as well, but iPi is not a one tracking run and done type system, you will have to get your own best practices worked out, but most can be solved right within iPi studio itself before exporting, just takes a bit of time to get it working faster for yourself.

I can get in and out of an iPi tracking of a 1 minute single dancer video in an hour or less from opening a new video, all refinements and export within an hour now for the most part, but I run my tracking on low resolution speed using a GTX 970 FTW, would be faster yet if I upgraded to a 1070, or better, but not going to yet, waiting for the next version from EVGA/Nvidia to be released, 20 series, or whatever it will be named, the 1070 model has been stated to track twice as fast as the 970 though, but dual cards don't have any better results with iPi usually.

Also, in tough areas, you can switch back to the old tracking algorithm, this doesn't affect any of the tracking you have done in other areas, but to be safer you can adjust the ROI just to the areas to be re-run, what I do anyway, just use the bracket icons in the lower tool ribbon to quickly set this.

You can also use the search function at the top of the page also, using my user name to find many posts that may explain more for you.

I record and track high energy dancers frequently, some videos in the Videos Index, with great results just using the iPi integral tools and hand Moves, then in post I clean up more, which you will have to do with any mocap system anyway and depends on how well motions were performed to match the character used on, mocap and characters are not always symetrical.

Hope this helps.

...

...

For my own information, I just recorded a video using all black clothing with black long sleeve, under blue t shirt, (I use a very short sleeved T also, only wraps top 4 inches of shoulder), but it worked fine throughout the actions performed, crossing arms, touching arms, touching legs, running arm down leg to touch feet, bringing forearm to bicep, many motions where if there would be an issue, it would have shown.

To me there were no what I would consider serious issues, no real arm flipping issues either, but I have found that occurs more on thinner female performers than with a male, maybe due to thicker arms.

In my opinion the tracking worked well with my set up, but you can determine that with your own tests also.

...

Hello,

1. Head tracking issue in your project can be mostly solved if we decrease actor height a bit. Height set in you project is bigger than it is expected, taking into account actual video. This may be caused by imprecise scale setting after calibration.

2. As to gloves without fingers for motion controllers, I think this should work. Palm is small body part, so this shouldn't spoil arm tracking.

_________________
iPi Soft

...

Gloves without fingers are fine, as long as it auto detects the hand color as black, (turn off hand color same as face checkbox in Actor appearance set up first), if not just use the color picker to choose off the glove.

I think they may have re scaled the actor since first post, wrong Actor and skeleton scale was evident just from the images, as they stated now they have the actor looking up a bit, which is common if the performer uses a straight forward gaze, just have any performer look at something a bit lower to bring head down at start and anytime they look forward will help that when not using a Move on a headband.

Complete skeleton sizing should be pretty accurate and Actor scale should be very close to the actual performers height, with Actor heals on, or very near the floor grid, if calibration and Actor scaling was set right with accurate height off floor set in calibration phase of cam one for best results, all other cameras should then fall to be very close to real world cam heights within 1 cm or less, you can adjust floor height a bit in studio for alignment too, as well with actor mesh sizing, with PS Eyes using a bit thinner overall mesh and better match clothing outline works fine.

Personally, if you are going to continue to use the area shown in the images, I would switch the T shirt to a more saturated color, not off white, yellow or gray, some color that throws less self shadowing and try to have less black in the background of all cameras when possible.

...