user10619: What program does this kind of tracking and how hard is it to achieve good results? I've seen this done in a lot of vfx breakdowns.


user13882: I come from a Hollywood visual effects background, and I was a matchmove lead for a couple of shows. For the above process mentioned, Vfx facilities do it all the time. 
Like Poor mentioned, it is very labour, skill and time intensive. Unfortunately, in almost all cases if you see the actor in the shot without a tracking suit, and if we need to track / match animate him, we will ALWAYS have to manually match-animate his motions by hand (and by eye). 
When we think of matching character motion we tend to immediately associate that process with the imagery of behind the scenes publicity shots for Ironman, Pirates of the Carribean, the Golem in Lord of the Rings,  The Avengers, Avatar, etc. In these behind the scenes shots we see men in motion capture suits, and face cameras. 
You need to know that in these cases, these men in suits are meant to be entirely replaced with computer generated characters in the final shot. Golem, Ironman, The Hulk in Avengers, blue aliens in Avatar, were all replaced with CGI. 
The fully replaced CG characters always get the limelight. For the tracking team, a huge amount of effort also needs to be put into manually tracking and matching live actors that do not get replaced by CG at the end of the day. 
Why do we need to track live actors with a painfully and accurately matched up model in 3D? It is for interactions with the CGI character that never existed on the set when the footage was shot. For example, when Frodo Baggins was holding a dagger wrestling the Golem, at the time of shooting, the dagger (and all metallic parts of his costume) never have the Golem reflected in them. Similarly, there was no Golem to cast an accurate shadow on Frodo too, in the areas where they contact. 
Thus in all the instances where you see any CGI character interact with footage elements that is shot live (physical location, physical props and physical actors), we have to create a digital counterpart in 3D and match them exactly, place them correctly in relation to the other objects in our 3D workspace, in order to capture shadows, reflections, colour diffusion, of the digital character. 
Besides having to match and body-track digital counterparts of live actors for reflection and shadow-catching, we also need to have them match-animated for dynamics simulation purposes. 
In John Carter I was in the digital costumes department, we have humans interacting with alien characters which are fully CGI. The characters all have flowing, draping costumes and accessories. We had to match-animate shots so that CG clothing items, hair and fur can collide, slide across and rest correctly on live human actors. If we do not create a digital counterpart to those actors and match them, those costumes, hair and fur will go straight through the live actors in the footage because those things never existed in the same 3D space where the simulation took place, for them to interact with. 
Finally to answer your question, lots of effort need to go into tracking a character. In the beginning we have the modelling stage, where we create a human model that need to have the same head, body and limb proportions as the actor. If these are wrong, you will fail to match up due to arms being too short, head being too large, and so on. Sexy industry methods like scanning of the actual actors are only reserved for films of very high budget, and where the effort and hours put in are justified. Most of the time, 3D artist will model them by eye, taking lots of references from the Web for proportions and such. Takes up much labour and time either way. 
Then comes the rigging stage where the otherwise rigid model is put through the technically involved process of adding complicated controls that will later allow match-move animators to articulate the digital model to match the live actor. 
Finally the match animation or body tracking artist will have to use their human understanding of perspective and how human actors move, to re-create the exact movement of the actor in the shot, from a 2d image to an accurate representation of movement in a 3D environment. Trust me, this is no easy feat and takes years of experience before an artist can become good at it.
Body tracking is several orders higher in difficulty than tracking inanimate, non deforming objects like a moving knife or a pole that was going to become a light-saber. Matching these rigid objects only require an animator to handle a set of translation (in X, Y, Z position), and a set of rotation (in X, Y, Z axes). 
In body tracking, each and every joint of the human body has many different possible positions, many possible degrees of rotation. To control and manipulate one digital character is more difficult than tracking multiple non deforming objects. 
Thus in a smaller budget or indie production, such accuracy is usually not required, and not viable given the time and resources needed. We would usually use cheats, hacks and shortcuts to get by. 
Finally finally, here's the difference between the motion capture process that are featured behind the scenes, versus matchmoving a character (which never gets featured anywhere). 
The purpose of capturing the motion for characters, is that they are ultimately going to be completely replaced by full CG characters. The reason for capturing motion of people in Mocap suits is so animators can acquire the mocap actirs' performance. Thereafter, the Mocap actor's performance is always modified. Due to jitter and inaccuracy in the captured data, Mocap data almost always needs cleaning up by artists. In most cases the captured movement will be modified, exaggerated more, acting and poses are pushed for greater impact. Some poses are changed because they do not work when viewed through the camera in the shot, or when composited in with real actors' acting, timing and reactions doesn't match or pace very well. Sometimes mocap data is used only for timing reference to the characters acting, as a guideline for the animator to start animating the digital character by hand from scratch. All these things are done so that we have a more engaging shot. Thus motion capture has no intent to match the action of an actor in shot. It serves a more artistic purpose - acting. 
The purpose of matchmove tracking of an actor serves a more technical purpose, to get CG elements to "gel" with the live elements in the shot, to ground them firmly, getting them to interact and react with each other, therefore they need to match the live action as close as possible, if not exactly. 
I hope this explanation is clear enough, and does not further confuse you in the end. 

10619: <p>What program does this kind of tracking and how hard is it to achieve good results? I've seen this done in a lot of vfx breakdowns.</p>

<p><a href="https://i.stack.imgur.com/2enlO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2enlO.png" alt="enter image description here"></a></p>
 13882: <p>I come from a Hollywood visual effects background, and I was a matchmove lead for a couple of shows. For the above process mentioned, Vfx facilities do it all the time. </p>

<p>Like Poor mentioned, it is very labour, skill and time intensive. Unfortunately, in almost all cases if you see the actor in the shot without a tracking suit, and if we need to track / match animate him, we will ALWAYS have to manually match-animate his motions by hand (and by eye). </p>

<p>When we think of matching character motion we tend to immediately associate that process with the imagery of behind the scenes publicity shots for Ironman, Pirates of the Carribean, the Golem in Lord of the Rings,  The Avengers, Avatar, etc. In these behind the scenes shots we see men in motion capture suits, and face cameras. </p>

<p>You need to know that in these cases, these men in suits are meant to be entirely replaced with computer generated characters in the final shot. Golem, Ironman, The Hulk in Avengers, blue aliens in Avatar, were all replaced with CGI. </p>

<p>The fully replaced CG characters always get the limelight. For the tracking team, a huge amount of effort also needs to be put into manually tracking and matching live actors that do not get replaced by CG at the end of the day. </p>

<p>Why do we need to track live actors with a painfully and accurately matched up model in 3D? It is for interactions with the CGI character that never existed on the set when the footage was shot. For example, when Frodo Baggins was holding a dagger wrestling the Golem, at the time of shooting, the dagger (and all metallic parts of his costume) never have the Golem reflected in them. Similarly, there was no Golem to cast an accurate shadow on Frodo too, in the areas where they contact. </p>

<p>Thus in all the instances where you see any CGI character interact with footage elements that is shot live (physical location, physical props and physical actors), we have to create a digital counterpart in 3D and match them exactly, place them correctly in relation to the other objects in our 3D workspace, in order to capture shadows, reflections, colour diffusion, of the digital character. </p>

<p>Besides having to match and body-track digital counterparts of live actors for reflection and shadow-catching, we also need to have them match-animated for dynamics simulation purposes. </p>

<p>In John Carter I was in the digital costumes department, we have humans interacting with alien characters which are fully CGI. The characters all have flowing, draping costumes and accessories. We had to match-animate shots so that CG clothing items, hair and fur can collide, slide across and rest correctly on live human actors. If we do not create a digital counterpart to those actors and match them, those costumes, hair and fur will go straight through the live actors in the footage because those things never existed in the same 3D space where the simulation took place, for them to interact with. 
Finally to answer your question, lots of effort need to go into tracking a character. In the beginning we have the modelling stage, where we create a human model that need to have the same head, body and limb proportions as the actor. If these are wrong, you will fail to match up due to arms being too short, head being too large, and so on. Sexy industry methods like scanning of the actual actors are only reserved for films of very high budget, and where the effort and hours put in are justified. Most of the time, 3D artist will model them by eye, taking lots of references from the Web for proportions and such. Takes up much labour and time either way. </p>

<p>Then comes the rigging stage where the otherwise rigid model is put through the technically involved process of adding complicated controls that will later allow match-move animators to articulate the digital model to match the live actor. </p>

<p>Finally the match animation or body tracking artist will have to use their human understanding of perspective and how human actors move, to re-create the exact movement of the actor in the shot, from a 2d image to an accurate representation of movement in a 3D environment. Trust me, this is no easy feat and takes years of experience before an artist can become good at it.</p>

<p>Body tracking is several orders higher in difficulty than tracking inanimate, non deforming objects like a moving knife or a pole that was going to become a light-saber. Matching these rigid objects only require an animator to handle a set of translation (in X, Y, Z position), and a set of rotation (in X, Y, Z axes). </p>

<p>In body tracking, each and every joint of the human body has many different possible positions, many possible degrees of rotation. To control and manipulate one digital character is more difficult than tracking multiple non deforming objects. </p>

<p>Thus in a smaller budget or indie production, such accuracy is usually not required, and not viable given the time and resources needed. We would usually use cheats, hacks and shortcuts to get by. </p>

<p>Finally finally, here's the difference between the motion capture process that are featured behind the scenes, versus matchmoving a character (which never gets featured anywhere). </p>

<p>The purpose of capturing the motion for characters, is that they are ultimately going to be completely replaced by full CG characters. The reason for capturing motion of people in Mocap suits is so animators can acquire the mocap actirs' performance. Thereafter, the Mocap actor's performance is always modified. Due to jitter and inaccuracy in the captured data, Mocap data almost always needs cleaning up by artists. In most cases the captured movement will be modified, exaggerated more, acting and poses are pushed for greater impact. Some poses are changed because they do not work when viewed through the camera in the shot, or when composited in with real actors' acting, timing and reactions doesn't match or pace very well. Sometimes mocap data is used only for timing reference to the characters acting, as a guideline for the animator to start animating the digital character by hand from scratch. All these things are done so that we have a more engaging shot. Thus motion capture has no intent to match the action of an actor in shot. It serves a more artistic purpose - acting. </p>

<p>The purpose of matchmove tracking of an actor serves a more technical purpose, to get CG elements to "gel" with the live elements in the shot, to ground them firmly, getting them to interact and react with each other, therefore they need to match the live action as close as possible, if not exactly. </p>

<p>I hope this explanation is clear enough, and does not further confuse you in the end. </p>
