I found it surprisingly difficult replicating result i'm seeing in ipistudio in 3ds max.
Exporting scene includes only skeleton. How can i export actor and camera?
In Ipi 3d actors proportions matched accurately to those of the subject all joints are positioned and rigged correctly
As a result contour of the video matched well with 3d actor in the camera view.
I have to do a lot of manual guess work to try to replicate it in max and it's still not as clean.

Also there is a feature in IPi studio "align color video to depth" it somehow distort video to compensate for the difference between RGB and depth . How can i recreate such distortion outside the ipi-studio?

...

You can not export whole iPi scene (cameras, lights, actor) there is no need for all that be exported anyway.

You can import your own FBX model into iPi, characterize it, then export it as FBX to import into Max.

You can also export from iPi as the default 3ds max skeleton as a BVH and import it onto a biped, re-save it as a BIP file then re-open it on a character with either a biped base skeleton, or link a biped to a boned skeleton and apply the BIP file motion.

What is your experience level in 3ds max? If you aren't sure of the processes, there are many tutorials on youtube for importing BVH files and linking a biped to a pre rigged character.

If using a CAT rig to import to, that's a whole other process and have to map the bones first.

There are several ways to do this, just depends on what works best for you.

So you know, a biped skeleton will need some tweeking in a layer to correct some import offsets of the collar bones and probably the head, if you tracked head rotations with a Move controller when recorded the action, do all adjustments on an animation layer.

...

How do i import/export FBX actor model to/from ipistudio i don't see that anywhere in the UI?

There is a need for exporting the whole setup to achieve what i'm trying to do. which is to create 3d animation that would overlay RGB video as close as possible. I already got this looking pretty close in ipi Studio with very little effort and now having really hard time trying ti recreate it in max. I need exact position of camera for that relative to the mocap. I see camera coordinates on "scene" tab so i learn to do it manually (after guessing axes differences )
I tried both bvh - to biped as well as ipi's default skeleton. as you mentioned it required adjusting the model and rigging it to skeleton to have all joints at the right places. And that is exactly what i was trying to avoid.
If i see exact result i need in ipi environment it shouldn't be that hard of a task to export it to another 3d platform including rigging setup and the model.

...

Under the Export tab in iPi Studio, you click the drop down arrow to reveal the skeleton choices, OR under the scene EDIT tab select add target character, then click select from file and import your FBX character file from the location on your computer you saved it to, from wherever you got it from, or if you created it in a program yourself and exported it as an FBX, has to be in common format extension to use in iPi Studio.

Don't worry if it imports into iPi Studio with the textures messed up, it may or may not, after it is characterized in iPi, it will then take on the animation exactly as the iPi does, or should if properly done, but know that sometimes characters that have ROOT nodes and pelvis bones sometimes need extra work to import/export properly, when exported back to your editor as an FBX, the textures will be correct in the model again.

NOTE: DO NOT export out of 3ds max with any controllers, just the properly weighted mesh and its bones, and only the IK bone set if it has multiple skeletons attached, (i.e. FK and IK bones).

If you are just trying to add a character to a real life image, there are several ways to achieve that, but to me 3ds max wouldn't be the best choice for that type of scene set up, but if that's what you intend on using, you can export the actual video background of any camera from iPi as an AVI, then import it to 3ds max as a background image, or maybe onto a video plane (I would have to look into the latter to be sure), but I am sure there are youtube videos on the process as well, but would most likely need to be converted to image sequence video.

Without using an FBX model in 3ds max, you will have no choice but to link a biped to your characters bones, that's how 3ds max handles its MoCap impoting as BVH and it isn't all that hard to do, then you bake the animation to your FBX character and delete the biped skeleton.

Also if you save several animations as BIP files, you can use the motion sequence editor to create linked animations of separate animations, so you don't need full animation takes of a single mocap to achieve a self created animation sequence for a scene.

iPi will not explain how to exactly do what you want, they are not really versed in 3ds max, I think they have some answers in the iPi Wiki Docs pertaining to 3ds max exporting, but not much, so better to research on youtube or other forums that deal with 3ds max more.

Re-creating the light position inside Max shouldn't be that hard, nor should placing cameras in a close position to where they were in iPi, but you can't export that whole scene from iPi Studio at this time anyway, but you can export just the Empty Background Image with the room lighting already shining where it would hit in scene, OR the whole video with the iPi actor, or your imported actor as a reference video that should be able to contain all the visible components, cameras, light sphere, floor grid, iPi Actor, skeleton, but can only be exported per camera as a single file.

Adobe After Effects would be a better choice for this type of scene set up and adding your components to the scene in it after animation edited in another editor (3ds max).

I am not fully sure of what you are trying to achieve, maybe a youtube upload describing it and showing what you would like to achieve and put that link in here.

This may take some research on your part to get what you want, but I am pretty sure it is possible to achieve, although your route and mine may probably be very different.

Here is a video exported right out of iPi Studio 3 using their export video option and holding down the show background button for full render time with my studio background showing floor grid, but not needed, can hide all scene components, and an FBX character only, performing the 3D animation, just a sample, iPi doesn't import characters with high detail or texture shaders that I know of anyway, just a basic textured character and you can see she is missing the eyes texture that iPi din't import correctly, but this is the same effect you can get with many external sources and video editors with better appearance.

Animation here was just edited inside iPi, but you could export the animation, clean it up and re-import the character and do the same steps above, hiding all but the character and the background, to test it, but of course not the optimal way for use of a final rendered project.

Link: https://youtu.be/_-dEVIUk63M view in HD

Hope some of this helps.

...

...

Another cam view without floor grid, but also note you can't add character floor shadows in iPi, music overdubbed after export as sample, not the song that was danced to, YouTube blocked original song uploaded, but this one works ok, as many dances go with many different songs as long as the beat tempo is close.

If this is basically what you trying to achieve, you can use any video recorded background and apply an animated character overlay on it, or even use an added green screen background in your 3D editor then using other software editors to remove it and just have a video of the character animation only and apply it to a scene to get the same effect, without ever even using a 3D character model for final render, unless you are camera tracking the video and need to see the full character from different angles, but can be done also.

These are recorded with 6 PS Eye cams, so if using Kinects, I can't guarantee same appearance achieved.

Link: https://youtu.be/B2NMn4L0Rdw View in HD

...

...

I did a quick test render in 3ds max 2014, just a straight render, didn't worry with too much set up.

Exported the FBX character out of iPi.

Imported the AVI video directly to a video plane and adjusted things to match better.

I don't use 3ds max for rendering, just wanted to see how it would look.

Render size scaled down to speed it up.

Link: https://youtu.be/GAEZxw2c_B0 View in HD

I can see with more time in scene set up and using a higher quality background video it could look pretty good.

...

Snapz,

I don't think you understand what i'm trying to do. here are couple of examples:
https://www.dropbox.com/s/x1108b07xlagb ... 1.mp4?dl=0
https://www.dropbox.com/s/v6ldd3m4fghrg ... 1.mp4?dl=0

I'm trying to use actual video of the actor (not the background) to match the mesh.
Those two samples i obviously did using markers based tracking. But i feel with some fine-tuning i can get something close to this using ipi without markers.

Question -what do you use for head tracking?

...

Let me try to understand this...

You want to wrap the mesh with an actual image texture of a persons skin, hair, clothing, etc? Sorry, I am just not following your idea. Maybe you want an RGB point cloud image to wrap a mesh? You can feel free to contact through the email link in the forum and possibly explain in more detail easier there.

Those samples really didn't explain much to me, except a mesh was created to fit the video actor and tracked motion by markers.

When you said overlay an FBX character on an RGB image, that's what I took it as, character on an RGB background video.

iPi is for motion tracking, the mesh textures would most likely have to done with Photoshop, or similar program and taking 4 angles shots of a person, front back and both sides, cutting them out and blending the textures at the seams to create a wrapped texture on the UV mapping you create in 3ds max, or you could just use any character mesh and do the same thing using the texture files that already exist and re-create a real skin image to fit those, couldn't you? Seems that would be a lot easier and quicker.

Am I even in the ball park of what you are trying to convey, or achieve? Are you trying to create some kind of new procedural texturing?

I am not sure iPiSoft would do anything different to satisfy any other type of exporting at this time, but they would have to answer that.

I only use the internal head tracking option in iPi, but Kinects require a head mounted Move controller, or the head tracking is very limited. (Head tracking box should be checked even when using a head Move, or the spine reacts badly with PS Eyes, or Kinects).

I could use a Move and the head tracking option checked, but I choose not to and just fix any needed offsets manually in post, which usually isn't all that much.

I only use 6 PS Eye cams, I don't even mess with the Kinects.

The animation in the above video is only tracked and processed using the integral tools and smoothing inside iPi, there has been no external post editing to it, that's just what iPi tracking puts out for me, then I do whatever clean up needed in an external 3D editor and use layer offsets where necessary.

...

...

Can you tell me if I am getting warmer to what you are trying to achieve?

I have been watching the video references you posted trying to decipher what you are wanting to do.

You may be trying to avoid playing so much with your characters mesh proportions to match a video performers proportions and saying if the iPi Actor would export out of iPi it's proportions are already very close to what you want?

Are you wanting to use an externally recorded video, or the Kinects video exported?

If external video, the scene camera in 3ds max may be all you really need and square the video plane to it at grid level, since you are using the Kinect cams RGB video pointing squarely at the performers front, you may need to tilt it forward or backward a bit to match the angle of the plane that you saw in iPi Studio and leave the 3ds max view port in perspective view and slide the plane back or forth to match the character to the video plane and I would think could probably be very close.

These are the only scenarios I can gather from what you stated for what you are trying to achieve without a much more detailed description.

I guess I am not understanding why you would need the proportions of an animation on a wire frame character mesh to match a video performers proportions behind it, but again, I guess you have your reasons.

...

It easy for me to answer why someone needs to have a 3d mesh to match video of an actor.
This is pretty much holly grail of film VFX. As it allows to simulate contact interaction between CGI elements and the actor.
And that is most common as well as most difficult task in VFX. ether you need to split someone's face open and have bugs croll from the opening or you simply need to put a cgi light emitting object in actors hand and have it realistically reflect light on the actor.
You will always need a 3d mesh in virtual camera view to match the actor as it captured on real world camera, accurately on every frame
couple of examples (that i did merely as a test of the tracking) :


https://www.dropbox.com/s/wut1fmw9extve ... 1.mp4?dl=0
https://www.dropbox.com/s/oc9gcmgj91ci7 ... y.mp4?dl=0