Masks inside of after effects are a great way of selecting specific and often finicky areas. They can also be used to create shapes. Masks can be drawn on top of solid, text, adjustment and footage/image layers.
Create a new composition and a new solid. We can now draw a mask on our solid by selecting it and using the pen tool from the toolbar. If we expand the pen tool options we can see quite a few. Let’s just stick to the top one for now.
This is the pen tools basic function, it simply allows you to draw a mask. All other options are for editing the mask. For starters, begin drawing a mask. It will take some experimentation and perseverance to become competent with the pen tool.
The first point you draw is the start of the mask and no matter what shape you draw, you need to end the mask at the same point. For example, I have drawn my first point as the top of a tree, made many other points but have come back to that original point to complete the mask.
You’ll notice two things:
- the points are roughly placed
- the points are all stiff, none are smooth
This is because I like to roughly place my points to get an idea of the shape, then I go back and refine and smooth them. To fix my first problem, I can individually select points by clicking on the vertices and moving them either with my mouse, or my keyboard by using the directional arrow keys. Holding shift whilst moving points using the directional arrow keys moves them 10x further.
You can select multiple points by clicking on one, and then shift clicking on additional points to select them. Alternatively you can click and drag to draw a selection box around many points. When you release the mouse, all points within that box are selected.
Here I have gone back and refined the position of my vertices. Before we add some curves to this very stiff tree, let’s take a look at all the mask options we have available. Whilst selecting the red solid, press M twice to display all the options.
The four options are:
- Mask path: keyframe this if you need to animate between different shapes in a mask (more on this later).
- Mask feather: feathers the edge. This can be broken up into an X and Y feather, allowing you to selectively blur each axis by clicking the chain icon.
- Mask opacity: lower to make the mask transparent.
- Mask expansion: positive values expand the mask and negative values shrink the mask.
Just by having a play with these settings you can create some interesting results:
Other important settings are the options from this dropdown menu, as well as the invert button.
Whilst they may seem simple, once you add multiple masks, some inverted, subtracting, others intersecting, it can become a headache.
To get the basics, select Mask 1 and press control + D to duplicate it. Set the mode to subtract, and lower the expansion into the negative values. You should have something like this:
Using mask expansion we’ve created the appearance of round edges. Let’s delete mask 2 and set the feather and mask expansion back to 0 on mask 1. Now let’s select the convert vertex tool from the pen tool dropdown menu on the toolbar.
Using this tool we can turn the sharp points into curves by converting a vertex into a Bezier curve. To do this, click and drag on a vertex using this tool to draw a curve. You can clean the curve up using the selection tool (hotkey V) and manipulating the curves.
Note the mask colour can also be changed by selecting the little green box next to mask 1 and selecting a colour.
As with many things, Bezier curves take a little while to get acquainted with. Practice and perseverance is key.
Let’s move to visual effects now, and a process called rotoscoping. Very often the need will arise to extract an element (often a person) from live action footage. This requires drawing often quite an intricate mask with varying levels of feathering around the mask, and it has to be modified every frame, 25 frames per second, 60 seconds per minute. Very painstaking!
Learning how to efficiently create and animate masks takes a lot of practice. Here are some helpful hints when rotoscoping:
- Drawing the mask on the first frame isn’t always the best idea. For example if you had to rotoscope a man who takes his hand out of his pocket and waves, it would be easier to draw the mask at the frame where his hand is visible. This is because if you created the mask when his hand in his pocket, as the complex shape of a human hand emerges, you’ll be constantly adding and reshaping additional vertices. Alternatively, it is often easier to create a new mask for objects that are revealed momentarily so that you don’t have to worry about the additional vertices when the object is obscured.
- Once you’ve created an outline for the object you are rotoscoping, you don’t need to animate every frame unless there is very fast paced action. You can easily get away with settings a keyframe every 5 frames. After effects will interpolate between the keyframes, and if need be you can clean up the in-between frames. Oftentimes the clean-up will be painless because the interpolation got you most of the way.
- You can use motion blur on the roto layer and after effects will variably blur different parts of the mask based on how fast they are moving.
- Don’t delete points unless you know you have to. Once you delete a point, it disappears from the entire clip and will often mess up the work you’ve already done. If you have too many vertices, I sometimes set a keyframe and then on the next frame move the mask off screen. That way you can create a new, clean mask. Just make sure that you aren’t using motion blur on the roto layer because such a sudden move will create unwanted results.
In the next lesson we will look at alpha compositing.