Now in Living Color – The Process of Colorization

One of my colorized shots used in They Shall Not Grow Old

I haven’t posted a lot of work lately, but not because I’ve been lazy. I spent the better part of 2018 as a Color Compositor and Matte Painter on Peter Jacksons’s colorized documentary They Shall Not Grow Old, that examines the life of soldiers during World War I. My experience in the past year really helped me analyze my own work, and recognize issues that I might not have discovered myself until several revisions later. Each day at the studio, I submitted drafts of my shots and we analyzed them in the theater to work them down to the smallest detail. I think one of the biggest improvements I gained through the WWI project is attention to highlights and shadows. Previously, I had been adding color to the highlights as a way to bring color into the overall image, but for the WWI footage, they wanted clean, white highlights and clean, black shadows; no tinting, no compensating for the atmosphere of the shot. Since I now incorporate this into my own projects, I think it forces me to work more color into the areas where there would naturally be more or less saturation, instead of relying on an overall base tint to mimic the look of film stocks.

The most popular question I receive is “How do you do this? Can I do this myself?” When I first started colorizing footage, I could hardly find any information on how to do it. There still isn’t much information out there, and there certainly aren’t any walkthroughs or 101 tutorials. In a nutshell, it requires manual application of every individual color in the shot. There isn’t a special program that does it for me; it’s a totally manual process. (There are some impressive algorithms and automated colorization programs right now, but they are not at a point yet where they can colorize footage in fine detail.) Since there isn’t one way, or a right way to do it, it’s up to the artist to choose the best and most efficient method. The largest task at hand isn’t getting color into the image, but getting all of those colors to move naturally within the footage. The most straightforward method is to paint and animate every frame one at a time, and I used this method on the very first piece of footage I colorized (see The Munsters – Season 2 opening.) Painting frame by frame was exhausting and frustrating, but out of the grief, it gave me a better understanding of how these areas of color need to move within the image and it pushed me to find a better, more efficient way to do this.

Since then and currently, I rotoscope the footage using motion tracking. Rotoscoping involves outlining an area of footage, tracking its motion through the shot, and making sure it stays in the appropriate area. Rotoscoping is usually done on a smaller scale to remove wires from a shot, change the color of an object, or to place new objects/characters in a shot. Rarely are they rotoscoped entirely unless it’s for something like colorization or 3D conversion. Motion tracking still requires frame-by-frame analysis, but it only requires manual intervention every 3 or 4 frames instead of every single frame. It might not sound like a lot, but that cuts down the task by about 75% and depending on the shot, some objects can be tracked with almost no intervention at all. This has greatly increased the efficiency of getting color into the shot, and with all the time saved, it gives an opportunity to work more detail into the image. Here is a quick step by step of the process I use:

The Process

      1. Select and edit the footage – Since I do everything myself, the effort it will take to rotoscope a shot is important to consider. I usually try to find a 30-60 second scene that has a good joke or punchline, and also has the type of movement I can feasibly rotoscope on my own. Backgrounds are important to consider, but they are a little easier since there isn’t usually a lot of detailed movement. I also get a chance to break these scenes down, add some music cues, a title card, adjust the laugh track, etc. which is a fun bonus.

        Editing My Fair Munster in Adobe Premiere

      2. Rotoscope – This is by far the most time-consuming part of the process. I start with a main character in the scene, usually with the face, and work out from there. There is no secret, just a lot of motion tracking, watching frame by frame to make sure the mask stays in line, and making adjustments as it changes. I’ve spent anywhere from 2 days to several weeks on some shots. Many times a shot will look simple enough like I could finish it in an hour or two, but then I’ll sit down and realize it’s going to take days for one reason or another.

        Grandpa Munster’s roto masks

      3. Assemble masks – This takes a long time and is also not the most fun part of the process, but importing the masks and arranging them in front of/behind each other is a crucial part of the process.
      4. Add color (finally!) – This is actually the fastest part of the process, and definitely the most fun. I can do color on a shot in one day, but it is always helpful to let it sit and come back later to pinpoint issues that stick out or require adjustments. Color is added in various ways depending on the brightness/contrast of the shot and how it absorbs the color. I use everything from solid colors, to levels to curves when adding the color. This is where an eye for color is essential. I sometimes use reference photos but if it’s for The Munsters, I Love Lucy or something else I know very well, I go without a reference and just make sure it looks good (Unless it’s something like a floral dress that can be matched to a real color reference.)

        Assembling the color masks in After Effects

      5. Detail it out – After the shot has color, I watch it over and over to look for masks that aren’t layered properly, or ones that need edge adjustments because they are bleeding over into other areas. At this point, the project file is getting pretty heavy and it takes longer to process changes so it’s important to do the rotoscoping correctly at the beginning.
      6. Render & Archive – When everything is finished, I render out a high quality, high resolution master file that I use to upload online. I save all the project files onto backup hard drives so I can always go back and make changes later.

The Results