Masking/Compositing in AE

Discussion and help related to Adobe video software goes here. e.x. Premiere, After Effects, Photoshop, etc.
Post Reply
User avatar
Bebop0083
Joined: Wed Oct 30, 2002 4:27 pm
Status: That's What She Said.
Org Profile

Masking/Compositing in AE

Post by Bebop0083 » Sat May 24, 2003 12:29 am

im using a program called video factory 2.0 to make most of my videos. i was wondering if you could cut and paste characters from other anime in a different anime like crossover vids examples: Tainted Dounuts and Dragon Bebop Z. if i can do it in video factory how can i?? if its only in adobe after affects how do i do it?? thanks. i have both premiere and after affects by the way. Premiere version 6.0. is there a difference between 6.0 and 6.5???

TaranT
Joined: Wed May 16, 2001 11:20 pm
Org Profile

Post by TaranT » Sat May 24, 2003 3:35 am

Vlad and E-Ko are probably the best sources on how to do this sort of thing. I believe what happens is that you mask the moving character and turn the background into a blue or green screen. Neither VideoFactory nor Premiere will do this kind of masking except in simple, usually static image cases. But it can be done with After Effects and a handful of other programs. The masked clip is then put on an overlay track in the video eidtor.

VideoFactory has an overlay track and the clip properties (right-click on the event) allow for setting an alpha channel type which would make the blue or green transparent - assuming the alpha channel was defined in the clip. I was never able to get this to work, although I never really tried that hard to figure it out. The only time I needed it, I made the clip in Premiere and brought it back to VideoFactory for the final render.

If you're composing with several of these characters, then Premiere is the way to go since you'll need several video tracks.

BTW, the crude way to mask the characters would be to export the image sequence and edit each frame in an image editor. Guaranteed to work, but don't use VideoFactory. It's not fun with large numbers of single frame pics.

E-Ko
Joined: Wed Dec 20, 2000 4:12 am
Org Profile

Post by E-Ko » Wed Jun 04, 2003 6:08 am

To create crossovers in videos like "Tainted Donuts", I use Photoshop and either Premier or After Effects. Premier and After Effects both recognize the alpha channels and layers of Photoshop images. As such, Premier and After Effects can import a specific layer from a Photoshop image, preserving the transparent and semi-transparent areas of the layer.

I refer to these layers as cutouts, though this probably isn't the official jargon.

The following is the method I currently use. It's a bit more sophisticated than the methods used to create "Tainted Donuts." The idea is the same, though. In fact, steps 1-4 and 29-30 are identical to the steps used to create many of the scenes in "Tainted Donuts". The other steps were added to adjust the darkness and antialiasing of the borders around the cutouts.

----------------------Cutout Instructions-------------------------

Disclaimer:
I'm mostly self-taught. I'm also a dumbass. As such, there are likely better, quicker, more competant ways to perform these tasks. Also, I probably misuse terminology. I don't claim to have all the right answers. At the very least, these instructions will provide a useful jumping-off point. If you'd like to send feedback, flames, or corrections, please send them to ekobet@juno.com.

Miscelaneous notes:

Several steps in these instructions are intended to quickly and
systematically darken the borders around of cutouts. If a darkened border is undesired, omit steps 5-17.

This guide does not cover color correction.

Some warnings about pixel aspect ratio: It is my understanding that full frame video (DV 4:3 720 X 480 DV/D1 pixel) has a pixel aspect ratio of 0.9. If you export a frame as a still, edit it in Photoshop, and import it back into either Premier or After Effects, Premier or AE may interpret the still image as having a 1.0 pixel aspect ratio (especially of you resize the image - I've found that when I keep the image 720 X 480, it's usually interpreted as having a 0.9 pixel aspect ratio. If I change the size of the image, it's usually interpreted as having a 1.0 pixel aspect ratio). When this happens, you will likely want to change the pixel aspect ration back to 0.9. Easily done in either Premier or After Effects. In Premier, this option is in the "pixel aspect ratio…" dialog box (under the Clip menu, advanced options submenu). In AE, this option is in the interpret footage dialog box.

When dealing with a sequence of stills (like a running motion), you may
wish to merge each cutout into a single layer before performing steps
5-28. This accomplishes two things. First, it speeds up the process by
allowing you to manipulate each clip at the same time rather than
seprately. Second, it insures that you are treating each cutout in the
same manner. Some steps, such as 25, are subjective and may be difficult to reproduce. Merging the cutouts into a single layer ensures that each cutout is treated the same (levels and curves features of Photoshop also allow you to "save" and "load", allowing one reproduce results without merging cutouts into a single layer). Merging the images also makes it easier uniformly color correct/adjust and apply filters to the images. After finishing your cutouts, they may be cut and pasted back into their own layers. Before merging cutouts into a single layer, it may be necessary to increase the size of the canvas.

Some values (such as border thickness) are subjective. A show with thick borders (like the ill-fated "Clerks" TV series) would use considerably different values (or methods) than a show with thin/no borders.

O.K. On to the current ritual:

1) Export image you want to composite as a bitmap

2) Open the exported bitmap in Photoshop

3) Copy and past the image into a new layer. The image should now have
two layers, a background layer and an image layer. From here on out,
we'll be working with the image layer - this contains the image you want to
composite, your cutout. For the remainder of these instructions, the background image layer will be ignored. If you wish, feel free to hide it.

4) Using the eraser tool, erase the unwanted area around the edges of your cutout (oftentimes, it is more convenient to use the lasso tool to select unwanted areas and press the delete key to remove them).

5) Activate magic wand tool. Set it as follows:
tolerance = 255
anti-aliased = yes (check the box)
contiguous = no (don't check the box)
use all layers = no (don't check the box)

6) Use the magic wand tool to select the cutout image. At this point,
marching ants should be marching around the borders of your cutout.

7) With the selection still active (ants should still be marching), use
the eyedropper tool to select the darkest color in cutout. This will color
serve as a border for the cutout, so select something like a deep
shadow, a border around a facial feature, etc. Try to sample from the
blackest part of the cutout (or just use black - it works in most cases).

8) With the selection still active (ants should still be marching),
activate the paint bucket tool. Use these settings:
Mode = normal (experiment if you wish)
Opacity = 100 % (experiment if you wish)
tolerance = 255
anti-aliased = yes (check the box)
contiguous = no (don't check the box)
use all layers = no (don't check the box)

9) With the selection still active (ants should still be marching), go to
the select menu, go to the modify submenu, select the "border..." option. Enter a
value of 2 into the "Width: _____ pixels" dialog box. There should now be
two columns of marching ants. Click "OK"

10) Use the paint bucket tool to fill in the selected area. Clicking
anywhere within the cutout should work - only the selected area (between
the two columns of marching ants) will be filled.

11) Deselect everything. The cutout will now have a jagged, dark border
around its edges.

12) Activate magic wand tool. Set it as follows:
tolerance = 255
anti-aliased = yes (check the box)
contiguous = no (don't check the box)
use all layers = no (don't check the box)

13) Use the magic wand tool to select the cutout image. At this point,
marching ants should be marching around the borders of your cutout.

14) With the selection still active (ants should still be marching), go
to the select menu, go to the modify submenu, select the "contract..." option. Click "OK." Enter a
value of 1 into the "Contract by _____ pixels" dialog box.

15) With the selection still active (ants should still be marching), go
to the select menu, go to the modify submenu, select the "border..." option. Enter a
value of 2 or 3 (I'm a wuss so I usually use 2) into the "Width: _____ pixels" dialog box. Click "OK." There should now be two columns of marching ants.

16) Apply 1.0 pixels of Guassian Blur (Filter menu, Blur submenu,
Gaussian Blur, Radius of 1.0 pixels) to the selected area (only the area
between the two columns of marching ants should be blurred).

17) Deselect everything. At this point, the borders should be a little softer and there
should be some antialiasing.

18) Activate magic wand tool. Set it as follows:
tolerance = 255
anti-aliased = yes (check the box)
contiguous = no (don't check the box)
use all layers = no (don't check the box)

19) Use the magic wand tool to select the cutout. At this point,
marching ants should be marching around the borders of your cutout.

20) With the selection still active (ants should still be marching), go
to the select menu and choose the "save selection" option. This opens a
dialog box. Change "Channel" to
"Layer 1 Mask" (I'm generalizing that the layer name of your cutout is "layer 1"). (This will gray out the "Name" portion of the dialog box, leaving only the "new channel" operation.) Click OK.

21) Deselect everything. We're done with the marching ants.

22) In the layers pallet, select the layer containing your cutout.

23) Go to the channels pallet. In addition to the usual channels (RGB, Red, Green, and Blue), you'll notice that
there is ablack-and-white channel labeled "Layer 1 Mask" (I'm generalizing
that the layer name of your cutout is "layer 1"). This black
and white outline is actually an 8 bit channel defining the transparency of
your cutout. The black areas (which should correspond with the area outside your cutout)
represent 100 transparent (0 % opaque) areas of the image. The white
areas (which should correspond with the inside of your cutout) represent 0 % transparent
(100% opaque) areas of the image. Gray areas (which shouldn't be present, yet) will indicate intermediate levels of transparency. Intuitively,
darker gray values are more transparent than lighter gray levels.

24-1) Channels may be edited individually. With this in mind, highlight the
"Layer 1 Mask" (I'm generalizing that the layer name of your cutout is "layer 1") channel. The other four channels (RGB, Red, Green, and Blue) should NOT be highlighted.

OR

24-2) In the levels pallet, the layer containing the cutout will have two
boxes. The left box contains a small image of the cutout. The right box contains a gray-level image representing the mask. Each box may be selected and edited separately. Select the gray-level box on the right by clicking it.

25) Apply 2 pixels of Guassian Blur (Filter menu, Blur submenu, Gaussian
Blur, Radius of 2.0 pixels) to the "Layer 1 Mask" (I'm generalizing that
the layer name of your cutout is "layer 1") channel. This
will create a thin gradient of gray between the Black and white areas of
"Layer 1 Mask" (I'm generalizing that
the layer name of your cutout is "layer 1") and a slight feathering (transparency gradient) around the overall
image.

26) Go to the Image menu, Adjustments submenu, and select "levels."

27) This histogram may be used to adjust the border thickness and antialiasing of your cutout. There should be a black, gray, and white arrow under the
histogram. Slide the black arrow (the on the left side) to the right. You
should see the edges of the composite contract as you do this. Slide the
white arrow (on the right side) to the left to darken the border of the cutout. Adjust these arrows until you're satisfied with the cutout (borders and antialiasing look acceptable). Click OK.

28) In the layers pallet, context click (right click) the gray-level box on the right side of the cutout. A dialog box will open. Select the "apply layer mask option."

29) Save the image as a Photoshop file.

30) The cutout/composite is now ready to be color corrected/adjusted and
imported into Premier, After Effects, etc. When importing, be sure to only
import the Photoshop layer containing your cutout. This is fairly intuitive - when importing a multi-layer Photoshop file, both After Effects and Premier will ask you to "Choose a layer." Select the layer corresponding to your cutout.

After the cutout has been imported, it may be placed on the timeline and edited just like any other still image footage file.

In an After Effects timeline, place the cutout in a layer above the background footage value (in other words, the foreground cutout will have a lower layer number than that of the background footage. A simple scenario would place the cutout in layer 1, background in layer 2). The transparency of the cutout will be maintained. Through the use of After Effects's motion graphics capabilities, there are plenty of ways in which you can position and animate the cutout.

In Premier timeline, place the cutout in a video track above the one containing your background footage (in other words, the video track corresponding to the cutout will have a higher track number than that of the background footage. A simple scenario would place the cutout in video track 2, background in video track 1). Use the "motion" settings in Premier to position (for a static image, starting point = ending point) and animate the cutout.

It looks like a long process. Step 4 is the only particularly tedious step.

-------------------------------------------------------------------------------------

User avatar
Ashyukun
Medicinal Leech
Joined: Wed Sep 04, 2002 12:53 pm
Location: KY
Contact:
Org Profile

Post by Ashyukun » Wed Jun 04, 2003 9:52 am

:shock:

Wow. That's quite a write up, E-ko. You should (if you haven't already) write that up as a guide.

When I do rotoscoping of characters into other series and such (WLtFO, Samurai Windu) I tend to do it entirely in After Effects using its internal masking capabilities. This has both its advantages and its disadvantages. Disclaimer: I'm writing this up from memory, I don't have a computer with AE in front of me, so hopefully I won't screw things up too badly. Also note the keys I talk about are the PC shortcut keys... I imagine they're similiar on a Mac, but I don't know for sure ... I'd also advise reading through all of this before trying it...

Basically, import the footage you want (both the overlay and the background) and put them on the timeline. You only really need the overlay, but the background helps by making it easier to see how much edge you should leave on the overlay when you mask it. The overlay should (somewhat obviously) be the upper layer so it sits on top of the background (think cels).

The 'G' key is going to be your friend for quite some time now. 'G' is the shortcut key for the masking tool. When you hit it for the first time, it brings up the generic masking tool, which looks like the head of a fountain pen. Pick a point on the edge of what you want to mask out of the original foreground picture to put onto the new background and click on it. A Yellow square should appear on the image (or a colored square- I believe yellow is the default these days). Pick another point along the outline of what you want to cut out and click on it as well. A second point will appear, with a line between them. You can use this to go all the way around what you want to cut out. When you get back around to the beginning (a mask must, as far as I know, be a closed loop), the odds are that the section you just masked around will disappear, leaving the everything but what you wanted to have left in. Don't Panic. Down on the timeline, expand the menus under the overlay track, and then expand the 'mask' item. You should see a drop-down menu that should say (I hope i'm not getting this backwards) "Subtract". Click on this menu, and switch it to "Add". You should now just see what you wanted to leave in the screen from the cutout. You've just masked your first object out of one scene and into another. You can also set the mask to 'None', which doesn't add or subtract anything and just shows you where the mask is. This is usually the best way to work on masks.

Now. This works just fine if you want to take a box out of one scene and into another, but unfortunately most characters, ships, etc. are not composed of straight lines (yes, I know that at some level, they are, but that's getting really technical). You need to be able to make the mask go around the curves of the object. You could do this with lots of short, straight lines, but there's a much more convenient way: Bezier Curves. If you hit the 'G' key a few times to cycle through the masking tool options, you'll eventually get a tool that has a cursor that looks like a greater than or less than sign. Click and hold on one of the points in the mask you already made, and drag the cursor around a bit. You should see either one line coming out from the point to your cursor, or more likely a line on either side of the point. The line should bend (still anchored at the point) as you drag around the cursor. When you release the cursor, the line should stay bent. Pick another point and try it again. What you're doing is adjusting the parameters for the curve on either side of the point. The curve between two points is determined by the controls at the points that define it. You can make some pretty complex curves between points once you get good at manipulating the point controls. You don't have to wait until you've already done all the masking though to adjust the curves. As you put down the points for a mask, if you hold the mouse button down when you click to put down a point, you can drag around the control bars for the curves and adjust them then. Just be aware that the controls you set influence the way the next line you lay down will go... The best way to learn this, all told, is just to experiment with it. It's fairy easy to figure out how the controls work. One thing to note though- sometimes (I've never really figured out exactly what determines it... there has to be some logic to it, I just don't know it) when you adjust the control bars, you'll adjust both sides and some times it will just to one side. Usually, if you adjust the control for a point and it moves both control bars (and adjusts that curves on both sides), if you then click on and drag the other control bar, it should just move that side.

The next important masking tool is the add and delete point tools- a fountain pen head with a plus sign and a pen head with a minus sign, respectively. The Add tool is actually far more useful than the delete tool, because you can do more than just add points with it. You may have noticed when you played with the curves above that the point you were adjusting became a solid square, while the other points were just the outline of a square. This denotes which points are active, and will be influenced by whatever you do. If you click on any outlined point with the add point tool, it will make that point active. You can also click off the line and drag out a box to select a number of points with the add tool. This will select multiple points, which if you then click on and drag around one of those points, all the points selected will move (as will the lines connecting them to each other and the non-moving points). If you find you can't make a single Bezier curve go around a curve on the part of the scene you want to take out, you can use the add point tool and click on the line between the two existing points. A new point will be formed, wihch you can move and adjust the curve controls on. The delete point tool does exactly what you'd expect it to.. it deletes points from the curve.

Before we move on to the part that makes this work so well for video work, I want to point out that it is possible to change the color of the masking curves and points. Bright Yellow is quite good for working on dark scenes, but if you're, say, masking Faye in her normal outfit, it will be hard to see when you're around her yellow clothes. If you look down under the 'Mask' controls on the timeline, to the left of the drop-down menu that controls the Add, Subtrack, or None, there is a colored box. If you click on this, you can change the color of the mask.

What makes masking in AE so good for rotoscoping characters in and out of anime is that you can manipulate the masks over time. The odds are you started off on the first frame of your scene, which is good. If you fully expand the options under the mask you're working with (you can have multiple masks), you'll see a number of options. They should be (but not necessarily in this order, this is from memory and I may miss one...) Mask Shape, Feather, Expansion, and Opacity. 'Mask Shape' is what makes this all work. To the left of "Mask Shape" there should be what looks like a stop-watch. If you've not worked with AE before, this activated the keyframe setting. If you've worked with AE a decent bit before, you probably already know well what this does. If you haven't though, a keyframe setting locks in a particular parameter at a point in time. You can change these parameters over time by setting different keyframes on the timeline for the parameters you want to vary. Click on the stop-watch, and you should see a diamond appear on the timeline at the time point you're currently at. EIther drag the time slider a few frames to the right or hit Page-Down a few times to advance a few frames. Now, there should be check box that appeared when you clicked the stop-watch to the left of it. Click on this, and another diamond point (keyframe) will appear at you current time on the timeline. Pick a few of the points in the mask and drag them from their original positions to someplace new. Now, go back to the original time by stepping backwards with Page-Up. The points should move from where you just moved them to back to their original positions when you reach the time you set the original keyframe at.

Normally when masking characters and such out, you will want to set a keyframe for the mask shape every frame or very close to it. For our purposes, using AfterEffect's ability to interpolate between keyframes is not that useful, as it will likely not be doing the same thing the scene object you want to mask it. If you need to hold the mask steady for a while, just set identical keyframes at either side of the hold. Another useful thing to know is that if you drag the Add Point tool select box to select all the points in the curve, you can then move the whole mask around (this can also be done by clicking on something other than the 'mask' on the timeline, and then clicking directly on the mask name (which selects all the points) and dragging it or 'nudging' it (using the arrow keys) to a new position. This is very useful when working with animated footage, as in many cases the general features of a character, object, or whatever will not change a whole lot between frames and you'll only have to adjust a few of the points once you move the entire mask. Also, the first time you try and either add or delete a point once you've started setting the keyframes, the program will pop up a dialogue box complaining about keeping a constant number of keypoints. It will tell you where to find the option to turn this off- I can't remember exactly where it is. You'll want to turn it off- you'll pretty much certainly be varying the number of keypoints over time.

So basically, to mask out a moving character or such, create the initial mask, set the keyframe, advance to the next frame- if the character (or whatever) has moved at all or moves i nthe next frame, set a keypoint (as you need to lock down the keypoints so they don't interpolate between keypoints, always keyframe at the beginning and end of a hold. i.e., if a character is in one position for frame 1, changes to another in frame 2 and stays there until frame 5 before moving to a new position in frame 6, you would put a keyframe down in frames 1, 2, 5, and 6. You won't change anything on frame 5, but you have to tell the program you want it to hold the same mask position/shape from 2-5), change the mask as necessary (or not), move to the next frame, and repeat as necessary. It is slow- but once you get good at creating the masks and curves for them, it works quite well. You can then move the position of the footage around however you want to place the character where you want them in the scene, and do things like adjust the size, opacity, etc. of the layer as usual.

There are a few other important controls I've not talked about- Mask Expansion and and Feather. Mask Expansion is exactly what is sounds like... it expands the mask either positively or negatively depending on the value you put in by that many pixels. This is useful if you realize you put the mask a bit too close in and want it to be a bit bigger, or not close enough and need to constrict it a bit. Feather sets the number of pixels on the edge of the mask that are feathered, or made semi-transparent (best way I know to describe it. Basically, Feathering allows you to eliminate the sharp 'edge' the sometimes happens when you're masking. I usually use a feather of 1 or 2 depending on the size of what I'm masking. You can have multiple masks on one layer, but you'll want to play with how you have them set Add and Subtact wise. For example, if you were masking out a donut, you would likely use an "Add" on the mask the encompasses the entire donut, and a 'Subtract' for the donut hole where you should be able to see through to the scene behind.

That's my somewhat-quick run down of how to do masks entirely in AE. It's a bit less intuitive than doing it in Photoshop, as far more people are probably used to just deleting things in Photoshop than making Bezier Curve masks in AE, but I think once you get used to it you should be able to mask at least as fast as in Photoshop, with less steps pulling things between programs. You're best off choosing whichever works best for you...
Bob 'Ash' Babcock
Electric Leech Productions

User avatar
OmniStrata
Joined: Wed Aug 01, 2001 4:03 pm
Status: Wealthy
Location: Chicago
Contact:
Org Profile

Post by OmniStrata » Wed Jun 04, 2003 10:40 am

Heh, I'm too lazy to rotoscope so :p to you all ^_^
"Strength lies in action. Let the weak react to me..." - Kamahl, Pit Fighter from Magic: the Gathering
"That is a mistake many of my enemies make. They think before they act. I act before I think!" - Vortigern from Merlin ('98)
"I AM REBORN!" - Dark Schneider Bastard!! OAV

Gallup
Joined: Sun Jan 23, 2005 11:57 pm
Org Profile

Post by Gallup » Tue May 03, 2005 8:32 am

Does anyone know if the new Ulead 9.0 has this ability as well?

User avatar
AMV_4000
Joined: Wed Apr 10, 2002 6:29 am
Location: USA
Contact:
Org Profile

Post by AMV_4000 » Mon Jul 04, 2005 2:55 am

Gallup wrote:Does anyone know if the new Ulead 9.0 has this ability as well?
i know the older Ulead media Studio Pro 7.0 has a Video Paint feature where it breaks down the frames and allows you to paint them.. although its sometimes easier to use AE.. I've used this paint feature for Various Color screening in videos such as Girlfriend (many of the cut outs), Prison Bitch (the um.. stuff), and in Fappage (blood parachute head), one slight problem is that you can only import 30 seconds of footage at a time (atleast in UMS 7.0), but i would go crazy frame by frame painting on a video that was minutes long...

its a nifty tool to have on hand, but make sure you dont mistakenly get Ulead Video Studio! i have that one too, and well.. theres no video paint feature that im aware of..

i think you might be able to download a trial version at Ulead.com i say try it, and if you like it.. get it.. its a good tool to have!

Gallup
Joined: Sun Jan 23, 2005 11:57 pm
Org Profile

Post by Gallup » Tue Jul 26, 2005 1:22 am

How do I acess this video paint feature?

User avatar
ssj4lonewolf
Joined: Fri Jan 28, 2005 11:24 am
Location: Stuck in Hell, i mean Phoenix....
Org Profile

Post by ssj4lonewolf » Fri Aug 26, 2005 10:59 am

hmm...what about sony vegas video 5.0?
Oh god, that black dude with the afro is always making those damn trash ass music hip hop amvs...he needs to do something with techno or rock....
.......as if I would do something like that.
おおかみなく

TaranT
Joined: Wed May 16, 2001 11:20 pm
Org Profile

Post by TaranT » Fri Aug 26, 2005 11:40 am

ssj4lonewolf wrote:hmm...what about sony vegas video 5.0?
Yes. Probably. I don't know anyone who has tried it. Look for "keyframed Bezier masking" in the Help menu.

Post Reply

Return to “Adobe Software”