We’ve had requests from users to have a written breakdown of how to use Mocha AE
for screen insert tracking and finish the shot in Adobe After Effects. What you may not realize, is that a lot of this information is already covered in our documentation, but I will walk you through the complete process here, using a combination of our documentation and personal experience. Let’s get started! - Mary Poplin, Product Specialist
For your reference, you can check out the final project below, before and after.
If you would like to follow along, please download our source file here: http://vimeo.com/mochapro/screeninsertsource
|Download our Screen Insert Source File Here
Workflow inside mocha
mocha AE workflow is designed around a project structure. It is good practice to only work on one shot per project file to minimize layer management and to keep the work streamlined.
The basic tracking workflow for mocha is:
- Import your footage
- Draw a loose spline around the shape you want to track
- Track the spline
- Set the ‘Surface’, or corner pin where you want the inserted image
- Adjust your track if necessary
- Export the completed track
Creating a New Project
|Click on Images to See High Resolution
In Adobe After Effects CC and CS6, you send clips to mocha AE. Select a layer in AE and go to “animation window>track in mocha AE“. (For After Effects CS4-CS5 you will need to launch mocha AE from the Adobe AE plugins folder.)
If you opened up your file from a layer in After Effects using the “track in mocha AE” command, you will get a new project dialogue automatically. If you are running stand alone version (mocha AE v3 or mocha Pro) you will either start a new project, or open a previous saved project file.
If starting a new project. Select the clip to import by clicking on the Choose… button to the right of the top line. This will bring up a file browser, where you can select almost any industry standard file formats. Image sequences will show up as individual frames. You can select any one of the frames and the application will automatically sequence the frames as a clip when importing.
- Name: A project name will automatically be generated based on the filename of the imported footage, but you can change it by editing the Name field.
- Location: Your project file and cache files will output to a directory called “Results” by default. This is created in the same folder your clip is imported from. You can change this using the Change… button or using the dropdown box to set a different relative or absolute path.
- Frame Range: The range of frames to import. We recommend to only work with the frames you need, rather than importing very large clips or multiple shots edited together.
- Frame Rate: Normally this is automatically detected, but you have options to adjust if necessary. Make sure you check the frame rate before you close the New Project dialog.
- Separate Fields: If you are using interlaced footage, set your field separation here to Upper or Lower. Make sure you check your fields match your footage before you close the New Project dialog. If you don’t set them correctly, you cannot modify them and will have to restart the project. I do not personally recommend interlaced footage because it can be a pain to work with.
- For more information on advanced project settings, click here.
Setting the In and Out Points
If you will only be working on a section of the shot you can use the In and Out points to set the range on the timeline. (Note: that the In and Out points also affect the range of the Überkey button.) You can zoom the timeline to only show you the part between you In and Out points by clicking the Zoom Timeline button.
IMPORTANT CONCEPT: Interlacing, Frame Rate and Pixel Aspect are especially important to get correct if you intend to export tracking or shape data to another application. If footage settings do not match between your Mocha project and your compositing/editing application (like After Effects), the track or shape data will be offset when imported. This is the single most common new user error, so don’t be intimidated, just double check your settings.
Tips for New Projects
- Only import as much as you need: Working with very long files can be time consuming for the artist and can slow down the tracking as it searches for more frames. Try to only use what you need, and work on individual shots, rather than multiple shots in one piece of footage.
- Frame rate, dimensions and pixel aspect ratio are important: Make sure these values match the settings in your compositor or editor, otherwise tracking and shape data will not match when you export it.
- If you are unsure which field your interlaced footage is in, import it and check: If you quickly start your project with a guessed field order, you can check to make sure it is correct by using the right arrow key to step through the footage. If you footage stutters or steps back a frame while you’re stepping through, it is probably in the wrong field order, or you may have to set pulldown.
- Try to avoid interlaced footage where possible: Interlaced footage can be difficult to work with. For your own sanity, try not to use it unless you have to!
Tracking with The Planar Tracker
Imagineer’s Planar Tracker technology provides 2D tracking data by tracking planes rather than points. The key to getting the most out of the Planar Tracker is to learn to find planes of movement in your shot which coincide with the object that you want to track or roto. Sometimes it will be obvious – other times you may have to break your object into different planes of movement. For instance if you were tracking a tabletop, you would want to draw the spline to avoid the flower arrangement in the center of the table — it is not on the same plane and will make your track less accurate. Think in terms of your objects as low poly models.
To select a plane you simply draw a spline around it. You can be fairly loose with your spline — the Planar Tracker is intelligent enough to discard the pixel movement that does not move with the majority of the pixels on the plane your are tracking.
mocha features two spline types, X splines and Bézier splines:
An X spline
A Bezier spline
In general X splines work better for tracking, especially with perspective motion. However, we also provide Bezier Splines as they can be versatile and is the industry spline standard.
The Relationship Between Splines and Tracking Data
One of the most important concepts to understand with the mocha planar tracking system is that the spline movement is not the tracking data.
It’s best to think of the splines you draw around objects as search areas. Here’s a breakdown of how the tracking works:
By default, any spline you draw is linked to the tracking data of the layer it is currently in. In hierarchical terms, the spline is the child of the track, even if there is no tracking data.
When you begin to track a layer, the area of detail contained within the spline(s) you have drawn will be searched for in the next frame.
- If the planar tracker finds the same area in a following frame, it will tell the tracker to move to that point. Because the spline is linked to the track by default, it will also move along with it and the search begins again for the next frame.
To see this relationship, turn on your surface and/or grid in the viewer after you have tracked something. Scrub the timeline and you will see that the grid and surface move with the spline.
Now select all the points of your spline and move it around the viewer. You will notice that the surface/grid will stay in the same place.
This is because the spline is linked to the track, but the track is not linked to the spline. The spline is merely a search area to tell the track where to go next. It is a common misconception that moving the spline while tracking is affecting the movement of the tracking data. It is not. Moving the spline is only telling the tracker to look in a different place and will not directly affect the motion of the tracking.
This makes the tracker very powerful, as you can move and manipulate your spline area around while tracking to avoid problem areas or add more detail for the search.
You can even unlink the spline from the track entirely so that any planar surface passing under the stationary spline area is tracked and you don’t have to move the spline if tracking starts to go off screen.
Selecting an Area to Track
With the Planar Tracker you simply draw a spline around something, as shown with the screen below.
Select one of the spline tools to create a shape around the outside edge of the area you wish to track.
Start creating your shape by clicking onto the screen.
After the third point, the shape will auto-close, but you can continue to add points.
When drawing splines it is best to keep the shape not tight on the edge, but actually give a little space to allow for the high contrast edges to show through, as these provide good tracking data.
Right-click to finish drawing.
If you are using the X-Spline tool you can adjust the handles at each point by pulling them out to create a straight cornered edge, or pull them in to make them more curved. Right clicking a handle will adjust all the handles in the spline at once.
Dealing With Obstructions or Reflective Surfaces
In some cases there are parts of an image that can interfere with the effectiveness of the Planar Tracker. To handle this, you can create an exclusion zone in the area you are tracking. This is very common on handheld device screens or on televisions.
For instance, in the phone example we are using, there are frames where there are strong reflections on the screen. These reflections can make the track jump. So we need to isolate that area so the tracker ignores it. Here’s how this is done:
1. Select the initial layer you created.
2. Select the add shape tool to add an additional shape to the current layer, which selects the area you want the tracker to ignore.
3. Draw this second shape inside the original shape. Note that both splines have the same color, which is an indication that they belong to the same layer. Also you will notice in the Layer Controls panel that you only have a single layer.
4. By turning on the Mattes button under View Controls you can see the area that will be tracked.
You can do this very same thing with any obstructions! So all you need to do to hold out a person or object from a screen is to draw and x spline around it and track in in a layer above your screen track layer. I suggest shooting a screen with a medium green image or movie playing on it, so that you can key both reflections and obstructions from the screen in your composite.
Various tracking parameters can be accessed by selecting the Track tab. On the left hand side of the Track tab, you will see two sections: Motion and Search Area.
Understanding the parameters section of the Track tab is vitally important for obtaining good tracks. Here we provide a breakdown of each parameter and how to use it effectively.
- Input Clip: This is the clip you are going to track. By default it is the one currently in the viewer.
- Input Channel: When tracking, mocha looks at contrast for detail. The input channel determines where to look for that contrast. Luminance looks for contrast in the light and dark of the image.Auto Channel looks for contrast in one of the colour channels. By default, Luminance does a good job. If you have low-luminance footage or you are not getting a good track, try Auto Channel.
- Min % Pixels Used: One of the most important parameters to look at for tracking. By default, the minimum percentage of pixels used is dynamic. When you draw a shape, mocha tries to determine the optimal amount of pixels to look for in order to speed up tracking. If you draw a very large shape, the percentage will be low. If you draw a small shape, the percentage will be high. In many cases, the cause of a drifting or slipping track is a low percentage of pixels. If you want a more solid and accurate track, try setting the Min % Pixels Used value to a higher amount. Keep in mind however that a larger percentage of pixels can mean a slower track.
- Smoothing Level: This value blurs the input clip before it is tracked. This can be useful when there is a lot of severe noise in the clip. It is left at zero by default.
These parameters control what motion you are looking for when you track:
Translation: The position of the object
Scale: Whether the object gets larger or smaller
Rotation: The angle of rotation of the object
Shear: How the object is skewing relative to the camera
Perspective: How the object is moving in perspective relative to the camera
The main difference between shear and perspective is the relative motion. Shear is defined as the object warping in only two corners, whereas perspective is most often needed where the object is rotating away from the viewer significantly in space.
As an example, if someone is walking towards you, their torso would be showing shear as it rotates slightly back and forth from your point of view.
The front of a truck turning a corner in front of you would be showing significant perspective change.
What kind of motion should I track?
- Large Motion: This is the default. It searches for motion and optimizes the track as it goes. Small Motion is also applied when you choose Large Motion. Stick with large motion tracking for the bulk of your projects, it is the default for a reason.
- Small Motion: This only optimizes. You would use Small Motion if there were very subtle changes in the movement of the object you are tracking.
- Manual Tracking: This is only necessary to use when the object you are tracking is completely obscured or becomes untrackable. Usually used when you need to make some adjustments to complete the rest of the automated tracking successfully.
This defines ranges for the tracker to search within, you rarely will need to mess with the default settings here unless you are consistently getting a bad track in your particular shot.
Horizontal/Vertical: The distance of pixels in the footage to search for the next object position. This is set to Auto by default.
Angle: If you have a fast rotating object, like a wheel, you can set an angle of rotation to help the tracker to lock onto the detail correctly. The tracker will handle a small amount of rotation, less than 10º per frame, with Angle set to zero.
Zoom: If you have a fast zoom, you can add a percentage value here to help the tracker. Again, the tracker will still handle a small amount of zoom with this set to zero.
Tracking the Spline
Before performing the actual track, adjust the settings depending on the movement in the clip.
Track the plane selected by pressing the Track Forwards button on the right- hand side of the transport controls section.
Stop the track and adjust the shape if it doesn’t seem to be tracking properly. You may keyframe the spline shape so that it tracks only the planar region of a shape by adjusting the shape and hitting Add Key in the keyframe controls menu. Keep in mind that no initial keyframe is set until you first hit Add Key or move a point with Auto-Key turned on.
Checking Your Track
The spline should be tracked in addition to the clip being cached to RAM. You can play it back and get an idea as to how the track went. Feel free to change the playback mode in the transport controls to loop or ping-pong your track.
Another trick you can do to check your track is hit the Stabilize button in the View Controls.
Turning on Stabilize will lock the tracked item in place, moving the image to compensate. In the track module, stabilize view is a preview mode to check your track. Actual stabilization output is handled by the Stabilize Module, explained in the Stabilize Overview chapter.
You can check the accuracy of your planar track by turning on the Surface and Grid overlay in the View Control panel. Drag the corners of the Surface overlay (the dark blue rectangle) to match the perspective of your tracked plane. If you play the clip, you should see the surface or grid line up perfectly with the plane you tracked.
The Surface and Grid have no keyframes; they are simply guides that let you check the accuracy of your track. Note that the position of the Surface WILL affect the exported tracking data, so you MUST position the corners of the Surface before exporting tracking data.
Next click on the Surface button under View Controls.
When you turn on the surface you will see the blue box that represents the 4 points of the corner-pin. Right now you will see that it is not lined up with the screen.
By selecting each corner one at a time you can adjust the surface area to cover the area of the screen.
The Grid overlay should line up with the plane you’re tracking and move with it as you cycle through the clip. You can change the density of the grid by adjusting the X and Y grid values in View | Viewer Preferences:
The grid overlay can give you a quick representation of the accuracy of the track.
The Trace feature allows you to see the position of the planar corners over time. Skip allows you to work with only every nth frame, useful on particularly long roto shots where the movement is predictable.
When you track a layer, the mattes of any active layers above the layer itself are subtracted from the matte of the layer and hence influence the area being tracked. To keep your tracking predictable, it is recommended that you keep your tracking layers on the top of the stack unless you specifically wish to use other layers to subtract from the tracking area of layers beneath it.
To monitor what the tracker “sees” as a tracking area, select the Track Matte button in the view control.
Tips for Tracking
- Scrub your timeline: When starting a new project, go through your footage a few times to see what your best options are for tracking. You will save yourself a lot of time by making note of obstructions and possible problem areas in advance.
- Use edges: When tracking surfaces you will usually get a much better track if you include the edges and not just the interior of an object. This is because mocha can define the difference between the background and the foreground and lock on better. For example, if you are tracking a greenscreen, it is better to draw your shape around the entire screen rather than just the internal tracking markers. In some cases this means you can avoid tracking markers altogether and save time on cleanup later.
- When in doubt, ramp up your pixels: You can quite often get a great result with default settings, but if you’re getting a lot of drift, try setting the Min % Pixels Used value higher. The processing can be slower, but you will usually get a much more solid track.
- Draw more shapes: Remember you are not limited to one shape in a layer. Use a combination of shapes to add further areas or cut holes in existing areas to maximize your search. If necessary, make an additional layer to track and mask out foreground obstructions before tracking the object you need.
- Use the grid while tracking: It’s common to use the surface and the grid to line up your corners after you track, but it can be much more advantageous to set up your surface before you track and leave the grid on to watch for any subtle drift while you are tracking. This way you can stop your track early to fix any issues and spend less time trying to find them later.
- Track from the largest, clearest point: In order for mocha to keep the best possible track, it is usually best to scrub through the timeline and find the largest and clearest area to begin tracking from, draw your shape there, then use backwards and forward tracking from that point. For example, if you have a shot of sign coming toward you down a freeway, it is usually better to start at the end of the clip where the sign is largest, draw your shape and track backwards, rather than start from the beginning of the clip.
- A planar surface does not necessarily have to be flat: We have a Planar Tracker which specifically tracks planes of motion, but this is not limited to tables, walls and other flat objects. Distant background is considered flat by the camera where there is no parallax. Faces can be tracked very successfully around the eyes and bridge of the nose. Rocky ground, rumpled cushions, clumps of bushes, human torsos and curved car bodies are all good candidates. The key is low parallax or no obvious moving depth. When in doubt, try quickly tracking an area to see if it will work, as you can quite often trick the planar tracker into thinking something is planar.
- In the end, there is no magic bullet: Mocha is a very flexible tracker and will save a lot of time, but you will eventually run into a piece of footage that just will not track. Large or continuous obstructions, extreme blur, low contrast details and sudden flashes can all cause drift or untrackable situations. If something just isn’t tracking no matter what you try, consider using mocha to track as much as possible then move to manual work. You can often get a lot more done fixing shots by hand or using Adjust Track in mocha rather than trying to tweak your shapes and parameters over and over again to get everything done automatically.
Exporting Tracks to Adobe After Effects
It is important that the clip length, frame rate, frame size, interlace mode, pulldown mode and pixel aspect ratio in the project match the corresponding settings in the After Effects project where you plan to use the data. You can change the frame rate and pixel aspect ratio settings in the Film and Time sections of the Clip tab.
To export tracking data to After Effects, press the Export Tracking Data button on either the Track or AdjustTrack tabs.
Choose either the After Effects Corner Pin Data*.
After Effects Corner Pin Data:
The corner pin data records and exports the 4 point x, y information from either the adjusted track or the raw track. There are three exports – two for recent After Effects versions and one for CS3 and older versions of After Effects.
In After Effects, load the footage you tracked and the footage/image/composition you wish to apply the transform or corner pin data to.
Select the item on the timeline that is the insert object. Paste the data to the selected layer. You can do this by selecting the ‘paste’ option in the edit menu or by typing “command- v” (Mac) or “ctrl-v” (Windows).
With the layer’s information expanded you can see either the 4-point tracking data for the corner pin, or the postion, rotation and scale information from the tracking is now applied to the insert layer.
If you are pasting transform data rather than corner pin data then you will need to delete the anchor point keyframes to see a result. We export both position and anchor point keyframes so that stabilisation or tracking can be achieved. See below*.
Applying Corner Pin for Layers with different dimensions
If your insert is not the same size as the dimensions of the composition in After Effects, you will need to take a few further steps to make sure your corner pin data fits correctly. The reason for this is that tracking data is basing itself on the relative size and aspect ratio of the footage, whereas After Effects treats the corner pin data relative to the size of the layer you are applying it to.
To get around this, you can take the following steps to modify the insert layer in After Effects:
Precompose the layer and move all attributes into the new composition.
- Go back to the original composition, select the precomposed layer and paste the data.
- You should see your corner pin correctly lined up now.
|You will now have an aligned corner pin.
Alternatively you can use Align Surface in mocha to define the full dimensions:
Apply a manual corner pin to your insert layer in After Effects and place it in the desired position for any frame.
On this frame, Precompose the layer and make sure all attributes are inside it.
You will now have a precomposed layer that is the same dimensions as the tracked footage.
In mocha, go to the same frame in the footage you applied the corner pin to in After Effects and select the track.
On this frame, turn on your surface and click “Align Surface” in the Layer Properties panel.
You will see the surface fit to the full dimensions of the footage.
Export this newly aligned track to After Effects corner pin.
Back in After Effects, select the precomposed layer and paste the data.
This will apply the tracking data relative to the full dimensions of the footage instead. If you need to adjust the insert, just open the precomposed layer and tweak the manual corner pin you made.
*As an aside, you can also use After Effects Transform Data to move objects inside of After Effects with mocha AE tracks:
- The transform data exports x and x positions as well as the scale and rotation for the whole surface.
- If you click Save, this will display a file browser for saving the tracking data for use later. By default, the files will take the name of the layer, so for a layer name Track_Layer the export will create a file named “Track_Layer.txt”.
- If you don’t need to save the export, you can press Copy to Clipboard, and then go straight to After Effects and paste the data. If you are saving to file you will need to open the text file you saved with the data, select the entire body of text and copy it.
- Sometimes, it helps if you delete the anchor point data.
Finishing Your Composite:
Finally, if you need to mask out any objects and finish your comp, you can complete that in After Effects.
- Adjust your screen insert opacity or blending mode.
- Remove any obstructions with masks made from keying or rotoscoping.
And from here all you need to do is render! If you get lost in After Effects, we recommend you check out Adobe’s documentation here