Tracking a Matte in Effect*
by Jon Carroll

This article is the first in a multi-part series on using Paint* and Effect* by Discreet Logic to composite a CGI character and special effects into live footage. My 3d software of choice happens to be Lightwave 3d. Even though Lightwave does not benefit from the close integration with Effect* and Paint* that 3d Studio Max does, they can still be used together quite well.  

There are times when you have a 'perfect' place to shoot.... but most of the time, you have to make do with somewhere that is less than perfect. In this case, I have a test shot for a client where my location had a small group of trees that the CGI character would need to be behind. Since I was in a public park, I couldn't cut down and/or strip the leaves from the trees so I knew I would have to create a matte around the trees so that they would seem to be the middle object. The matte will be used on a plane positioned at the approximate depth of the trees in the scene.  

I captured my footage on my DPS Perception Video Recorder, using an AD-2500 capture card that DPS was kind enough to loan me for this and other reviews. For work like this, it is best to use uncompressed video if you have the capability, or to capture your footage at the highest data rate possible if compressed video is your only option. In my case, I captured the video at 6 mb/s ('Digital' quality setting on the PVR) and then used the Perception Player utility to trim it down to the exact footage of the shot. Since the shot also involved the inclusion of some CG pyrotechnic effects, we used some filter paper attached to a portable reflector to provide some interactive lighting for it... maybe its not perfect, but its better than doing it all in post. Since Effect* does not directly support the DPS Perception, I loaded the video footage as a Targa sequence using the Perception virtual file system. (I was operating under WinNT 4.0 sp3. Note that the Perception virtual file system does not work under Win95/98.)

Creating The Matte 

Loaded footage

Start up Effect* and begin a new project. Set the length of the scene appropriate to your footage. In my case, my trimmed footage was 400 frames long (that's 00:00:13.10 in the timecode world). Import your footage into he project, deinterlacing it as necessary. Save the project with the footage imported into it.  While doing these procedures, you may want to turn the Display to 1/2 or even 1/3 resolution to speed up the previews, but all of my screen shots here are at full resolution for detail purposes. 

Next you need to go through the somewhat arcane procedure of finding the best frame to draw your matte on. This would probably be best done outside of Effect*, but you can flip through frames in Effect* just fine until you find the appropriate frame. You need to find a frame where the object you are creating a matte for is relatively stable, and undisturbed by things like motion blur or field separations. In my case, it was 00:00:06.22. In this particular frame, the attention of the viewer was focused on the creature immediately behind the tree, and the camera was still for a few frames in each direction. I did this intentionally, knowing I was going to have to pull a matte of the tree later.

The rough matte

Then go into the Effects menu -> Mask and select 'Freeform'. The Freeform Mask window will open. Set the matte to 'replace', and draw a rough matte around the object you are matting. At this point, it is sufficient to outline the object and its major features in a rough fashion, and worry about refining them once you've created a rough matte around the whole object. Clicking in position and dragging the mouse, you will create a control point for the matte and pull out spline handles. Don't worry about precision quite yet, just get the object or objects blocked out. Clicking on the start point of the matte will make the matte into a closed loop.

Reducing intensity

Next, reduce the intensity of the mask. This will make it possible for you to see areas on both sides of the spline that may need to be included in the matte.

Refine the edges

Now, you can carefully go along the edges of the matte and refine its detail. Clicking along the spline path between control points will create new control points you can use to refine the detailed areas. Use the rounding capability of the splines to circle rounded details instead of creating too many extra control points, as larger numbers of control points will make the matte render slower (And at this point, you will see the matte rendering. Areas outside the matte will be darker due to the reduced masking intensity. Otherwise they would be black) For creating more drastic changes  in the contour of the spline, press control before clicking on the point handles and you will 'break' the handles from the control point, allowing you to send the two control points in different directions. Save your project often as you work on the matte, and expect this procedure to take awhile. Don't worry if you need to come back to work on it later, just note the frame you are working on, and you can re-open the matte you are working on.

Make sub-mattes

In my matte, since it was trees, I had to make holes inside the matte where you could see through,  because the creature would need to cast a shadow onto the structure behind the trees, and because there were areas in the tree where you could see through it. In order to do this, create additional mattes on top of the first, and set it to Subtract from the previous matte. Invert is important otherwise you end up with just the area around the hole as your matte, instead of a hole IN your matte. You can still use the reduced intensity trick to refine the edges of these as well, setting the intensity of the 'holes' to lower just like you did with the original. In some places, because of the nature and positioning of the leaves, it looked better if I feathered the edges of the mattes by a few pixels. 

Tracking The Matte

If your matted object remains still in the scene, then you are done. If not, then you need to track the motion of the matted object through the camera's view. 

Tracker settings

Effect*'s 2d tracker is fairly powerful and resilient, and I have found it to be able to track footage that After Effects cannot. Discreet simply converted the motion tracker from their higher end tools like Flint* (now Effect* 5.5 for IRIX) and Flame* into something useable within Effect*'s environment. They even added a few things that aren't in the Flame* version (like multithreading, the new magnification modes for contrast and edge detection, and the result preview window). We will be using this 2d tracker to track the movement of the matte object (the tree) through the scene and use that data to move the matte appropriately. At this point, the Control Points button should be selected for your mask. You need to click on the Object button next to it so you can apply the Tracker to the mask. The Tracker in Effect* is context-sensitive, meaning that you can apply it to only a few things in a scene if you have them selected. This includes the capability to select a few control points of a mask and track them to the edge they are applied to if you so desire.

Select a tracking point

You need to find a tracking point, preferably one within the object you are matting. You need to find two fairly high contrast tracking point that remains consistent throughout your animation. In my case, the upper trunk of the larger background tree was the location of the points I chose. The first point (the red tracker box in the above picture) is for tracking position data, and the second (it will have a little '2' next to it, it is white above) is for tracking Scaling data. I used a longer and narrower reference point for the second tracker because the changing width of the tree gave the best reference for scaling data. I extended the reference box just beyond the edges of the trunk since the tracker would be using the edges of the trunk for most of its tracking information. It will also be helpful to temporarily turn off the mattes for this procedure. To do this, go to the timeline editor and click the arrow next to 'effects' under your footage. You should see your mattes listed in the order you created them. Click the yellow box next to the name of each matte (it looks kind of like a square LED) and it will go gray, turning off the matte. Turning them all off will greatly speed up the tracking process. Depending on how good your tracking point is, you may need to tweak the Tolerance setting to get a more accurate track of your object.  In my case, I cranked Tolerance down to 50% and achieved a more precise track.

Make sure the tracker is set to track Position and Scale, set to Relative mode. Select Tracker one by clicking on it, then track the duration of the frames for your matte. You should see the matte moving with the tracker, thought the matte may fall out of scale with the feature it is tracking. If the tracker loses your object too much, you will want to go back to the start frame of the track, reset the tracker, and then lower the Tolerance further. You may need to change your tracking point, or manually set the tracking for the points where it loses the reference point. After you have a good track for the primary matte, you need to export the track data for Tracker 2 so you can then re-load this track data for the subsequent mattes.

Importing and exporting tracker data

For subsequent mattes (the 'holes' in your matte),  set up a tracker to track position, but instead of tracking the matte, simply import the tracking data from the main matte, and the sub-mattes will move with the larger matte. Do this for every sub matte and they will all move together. This works especially well on my matte since some of  the sub-mattes move off screen. 

To be fair, I will tell you you may need to go in at this point and hand-tweak the position, scale and rotation of your matte or sub mattes to account for things like lens distortion (or even a light breeze). But, your tracked mattes should still be reasonable useful.

If you've followed the instructions, as well as taking some of my suggested actions, (including tweaking where necessary), you should have a tracked rotoscoped matte for your use. You can output it and use it for a variety of things, in my case, I am exporting just the Alpha channel and using it as an Alpha channel for the main footage mapped onto a large polygon positioned in from of the camera in Lightwave. This means I can let my character cast its shadows onto the objects in the scene without its shadows falling across the trees it should be 'behind'.

Click the image to see a small mpeg movie

Jon Carroll is a freelance animator, illustrator and production designer located in Richmond, VA. He can be reached at draven@infi.net.