Does anyone have experiences with capturing rock climbing moves in regard to camera configurations and setup? I have both kinect2 and 6xps3 eye setup and right now both cause serious problems each.

Kinect2: obvious occlusion, since lots of rock climbing moves involves hand movement in inbetween wall and body plus arms and legs cross over lots of the time.

Ps3eye: shadow issues constantly due to the body being close to the wall and keeps messing up tracking.

I have thought of purchasing two kinect1 to try if that could solve occlussion issues, since i now dont have the funds to buy both another kinect2 or access to another pc...

Suggestions are most welcome...

Odd i would have sworn i just created a post and now its gone?

I want to motion capture rock climbing movements. However the nature of these movements obviously has some challenges, like overlapping arms/hands and crossing of legs.

Now i've tried with two setups. Both cause problems in each their way, so i would like to hear if someone else can come with suggestions to this.

With kinect2: i have the obvious occlussion problems. I have the kinect2 behind and since the hands are lots of the time inbetween wall and body i loose track of them. Having the camera on the side still cause problems with the arms the furtherst away. And since i dont have access to a second pc or funds for a second kinect2 i havent tried with two.

With 6xps3: this setup has lots of problems with shadows from body and arms/legs and was totally impossible for me to get any results out of. I tried with two cameras behind and two on each side. One low at about 1 meter and one at around 2.5 meters. The two back cameras was about 4-5 meters away. The calibration was perfect, but tracking failed terribly at finding most of my limb movements...

Would it make sense to try two kinect1? Or can i try something else?

...

PS Eyes would be the best solution for this scenario to me, as for one thing even 2 Kinects 2s can't be positioned correctly to capture enough of the movements accurately and with only one Kinect, no way you could get this to work properly.

You didn't state the size of the area this set up was in, but the minimum needed area would be I would suspect around 4m L from wall x 7m W x 4m H to top camera.
(I am assuming this is an indoor climbing wall?) If it's outdoors, there are other factors to consider when using PS Eyes, of course you will never be able to capture the full height scaling of a real rock face.

When positioning the 4-PS Eyes, I would put one directly behind actor maybe 1.50m-1.80m high approx. with a buffer of at least 3m, maybe more, off visible capture volume area, (Tape mark on floor helps positioning), and 2 directly to the sides approx, same position heights as rear one, then one high directly looking down the face of the wall above the actor as high as possible, (this one would control the depth to wall of the actor), as I don't know how high this wall is for how high that one can be positioned, but should try to keep the 3m buffer distance from highest point climbed to. (Adjust to where you feel would work best and make sure camera view is set correctly reflected in iPi Recorder).

You may only need these 4 cams, but you can position 2 a bit closer to the wall and performer to record the hands/feet placement better, still with minimum view covering full height to be climbed and positioned in between the side and rear looking cams at say a 30-45 degree angle a bit higher, or one higher, one lower (1.20 low-2m+ high), as long as they can always see the floor touch of the light probe, there is no need to raise the light high off the floor for calibration, lower trajectories work fine also.

Calibration will work fine as long as all cameras can track the light through-out, you should have no issues with that set up.

Use soft white lighting directly up and down the wall face, (Filtered through cloth or paper if too bright, bright lighting isn't needed, just consistent lighting), this will minimize shadows of protruding obstacles and using low ambient light on back of actor.

The use of dark gloves will help in the hand tracking, with richer colors for other clothing, (Blue, black, green, red solid colors), work well, with solid dark shoes, or socks instead of multi-colored shoes, if possible. (Darker colors limit shadowing on the materials).

Facial hair and long hair should be minimized, or pulled up to expose the sides/back of the neck, (I don't know if this pertains to you).

Stand a little bit off the wall to get your refit on the actor for tracking, then move onto the wall.

I don't know your means of setting up this way, but if you attempt this configuration, please post with the results.

Hope this helps.

...

Thanks for the detailed feedback Snapz, i talked to the guy responsible for the climbing gym i usally climb at, and was allowed to use a smaller wall outside actually. It's only about 3-4m tall (bouldering) and is mounted on a industry container, so i should be able to position cameras on the sides as well. About placing the camera above, i might have to work out some sort of intermediate pole solution. However, right now the weather conditions in denmark does not really allow me to do the recordings outside, so i guess i have to wait a bit til it gets warmer.

In the meantime i will make tests in our backyard just to get the calibration fixed. The first two tries i had failed to create any real tracking because i didn't have the prober clothing. So once i've gotten a hang of the basics of calibration and know how to produce consistent good results with a normal circular/half circle and produce some working results i'll also try to make some tests with movements close to a wall (outside). I'll see if i can post my findings on youtube on my channel and keep this thread updated.

I agree that PS Eye may be the more cost effective option. However, depending on how you shoot the rock climbing actions, there may be a few non-device related issues working against you.

Assuming you're recording actual rock climbing actions, the first problem is shadows. Being close to the wall, it's likely that the performer will cast hard dark shadows which can confuse the PS Eye capture. You'll want to use soft boxes or some other soft lighting. (When I'm using PS3 Eye cameras, I like to use at least two soft boxes to light the environment.) If you're recording the climbing outdoors, you'll want to do this on an overcast day, in the shade, or very early in the morning before the sun has fully risen.

The second issue you're facing is confusion caused from being too close to the wall. I know from experience that Mocap Studio can get confused if the performer is too close to large furniture or a wall. The problem is that the software may partially merge the actor's data with the large object, making the human form less clear for tracking. iPi Soft generally recommends giving a bit of space between large objects and the performer, which obviously poses a challenge when climbing a wall.

Lighting and shadows is far less of a concern with Kinect v1 or v2, than it is with PS3 Eye and soft boxes will not be necessary. However, the second issue is probably a bigger problem for Kinect recording. If you're going to use Kinect, you'll very likely want to use dual Kinect v2 for its higher definition--this feature should help define the human figure and minimize the 'merging' problem.

All things considered, recording actual rock climbing may be a bit challenging with either device.

An alternative approach is to simulate the rock climbing by 'climbing' the floor. By performing on the floor, and aiming the devices (PS3 Eye or multi-Kinect) appropriately you maybe be able to minimize the occlusion issue since you can position the devices around the performer for greater coverage (i.e., top, bottom back and sides as opposed to only the back and sides of the actor). You'll have to pantomime the gravity factor but if your performer does a lot of rock climbing, they should be able to fake that convincingly. A lot of motion capture is pretty much 'acting' after fall. :)

Also, if you're recording motions on the floor, try lowering the ground plane and disabling feet tracking. This should solve collision issues with the floor.

Based on my past experiences with iPi motion capture, I believe it's very possible to record a convincing rock climbing performance with either PS3 Eye or dual Kinect Sensors. However if you're expecting the rock climbing motions to be 100% accurate and realistic, you may need to go with a higher end system (i.e., expensive and more complex setup). On the other hand, if you're not too concerned about capturing perfect realism and are willing to accept some compromise to get a good performance, you should be able to pull it off with Mocap Studio.

Good luck! Let us know how it goes--this sounds very interesting.

G.

_________________
Greenlaw
Artist/Partner - Little Green Dog | My Demo Reels (2013,) (2015,) (2017,) and (2019)

Watch a one minute excerpt on Vimeo now!

Hi Greenlaw, and thanks also for your input.

Im pretty sure i will need some kind of soft light source to compensate for the shadows. I made some test recordings just outside in our backyard, where the first 2 tests was complete failures due to a number of noob mistakes on my end. However this was my first tries with ps3 eyes and i didn't realize how important lightconditions was and also clothing.

I will post some pictures and sample videos of the experiences i gather as i get more time. But right now im still in the process of learning how to setup the cameras and get the calibration to work in a normal circular setup without any "close-to-wall" movements.

One thing i noticed yesterday when i was calibrating outside was that my light source which was first just a normal bike-light, kept getting missed due to the fact that the bike-light is directional, so several of the cameras was constantly missing it. Also since it was quite bright (it was 13:00 in denmark, but partly cloudy), my hands kept getting tracked for a light source, so i ended up covering them up with my hoodie which is dark blue, and that helped a lot. Also the pole i attached the light source to, i painted black to further ease up the recognition process.

Final thing that helped out with the calibration was to take my mini-maglight, which initially proved totally useless, and pop on a table-tenisball on top of which caused it to light up very bright and thus enable the calibration to be perfect in the end.

All this gave me the idea of wearing colored gloves once i will make the test recordings for "close-to-wall" movements. Just like Snapz suggested with wearing "socks".

Since i belive this learning process is going to be very valuable to anyone doing any kind of tracking with ps3 eyes, i think i'll post a set of videos initially to address this and then move on to the issues i find with recording close to walls so i finally can figure out a way to record my climbing moves.

If all fails i might try the approach you mention with recording movements on floor. It does totally make sense, but wanna try out doing the moves in a real setup first before i abandon it :-) after all, some of the swings and dyno moves that rock climbing involves are gonna be really hard to mimic on floor... anyway, it might be worth trying yes. Thanks for that suggestion too..

So i actually went ahead today and made a test setup near our backentrance. The weather was clear without any rain, so i ceased the moment and grabbed my gear and made the setup to test how it would perform with a camera directly above.

I've posted a video showing the raw results here.

https://youtu.be/gAqzhJCrZQA

I'll continue trying to improve this setup until i have it ready to transport out to the bouldergym where i can borrow a small outdoor wall to make some real recordings. Let me know what you think.

Clever setup :)
A couple more cams between side and back ones should improve tracking.
Also, add more color contrast between clothing and the wall. I suppose issues with legs are caused by the low contrast, and possibly by inaccurate settings of model colors and/or scene lighting.

That's super cool! The results are looking pretty good for a first run. The occasional pops should be fixable with some manual corrections and tracking backward through these sections. Thanks for sharing that.

A few years ago I made a video on how to tap the PS3 Eye base for mounting on a standard 1/4" screw, like on a tripod. This mod is very to do using a few inexpensive tools and it makes the cameras more convenient to set up. Naturally, it's much steadier than taping down the devices too.

I've busy re-designing my website and one of the new features is a tutorial section. When it's ready, I'll dig up the video and post the video there.

G.

_________________
Greenlaw
Artist/Partner - Little Green Dog | My Demo Reels (2013,) (2015,) (2017,) and (2019)

Watch a one minute excerpt on Vimeo now!

Thanks for the feedback. I will look forward to that video Greenlaw.

I plan on going bouldering tomorrow and if i remember it, i'll record a small video of the climbing wall i intend to use for the recordings. I just remembered that all the crashpads we have out there are dark green, so i think i need to purchase some big pieces of cloth of some sort to cover it up with, to get better results.

About the trousers i wear, i just noticed that i was wearing some slightly lighter ones for this recording and it is probably the cause of some of the glitches, but even with the darker ones i wore yesterday there was some popping going on. Anyway as mentioned, it should be pretty easy to fix by reposing and re-tracking..