Digital Visual Effects Production Notes

The biggest visual effects challenge for the first Spider-Man film was convincing the audience that Tobey Maguire wearing a Spider-Man costume could jump off a 25 story building and land without shattering all the bones in his body. The audience was willing to take the leap and Sony Pictures Imageworks' crew was only too happy to push them over the edge. "When you think about that final swing, the emotional content was pretty compelling for a character wearing a mask," says Oscar award winning visual effects designer John Dykstra. "You got a sense of joy and excitement. That's where the visual effects had to pick up for the second movie."

For Spider-Man 2, Dykstra wants the audience to not only believe that Spider-Man could swing through Manhattan on a silky thread, he wants moviegoers to feel it, to step off the edge and swing with the superhero.

"I think the audience will get a real sense of vertigo," Dykstra says. "We explored what it feels like to be up high, moving fast and falling. I think this film will give the audience some idea of what truly makes Spider-Man a superhero. I think they'll feel what it's like to actually be in those upper climbs in freefall and in flight."

"We're benefiting a lot from the advances in CG technology," adds "Spider-Man 2" director Sam Raimi. "The environment that Spider-Man lives in is greatly enhanced because of those advances in technology. The buildings seem more real; the swinging shots of Spider-Man are more exhilarating."

All told, 222 people at Imageworks helped create a total of 836 shots, 300 more than the first Spider-Man film. "We had more work to do in the same amount of time," says Lydia Bottegoni, visual effects producer. "The CG characters were more complex. The CG environments hold up to greater scrutiny. And, we have more effects animation than in the first film."

In the first film, the villain wore a mask and rode a hover-board. In Spider-Man 2, Dr. Otto Octavius (Alfred Molina), more commonly known as Doc Ock, has four octopus-like tentacles that act as additional arms and legs. And, this villain doesn't wear a mask. Thus, the digital effects crew had to add CG tentacles to the live action actor in 178 shots, and create photorealistic Doc Ock digital doubles for 38 shots.

Puppets created by Edge FX were used for the CG tentacles when the metal arms had to interact with Molina-when he lights a cigar, for example, or brushes back his hair. But, each arm required four puppeteers which meant 16 puppetteers had to work alongside Molin when all four tentacles were active. In addition, because the tentacles could expand and contract, the practical, puppeteered versions were created in several lengths that had to be swapped as a tentacle reached for something. Thus, the CG tentacles were often called into action. "Our CG tentacles were useful for shots where we had close action, where Doc Ock had to stretch a tentacle out to reach for something, and in the dynamic fighting shots. There was spacing between the vertebrae in our CG model, so we could do subtle length changes," says Oscar-nominated visual effects supervisor Scott Stokdyk.

To create the digital tentacles, Imageworks started with Cyberscans of the puppets taken at Gentle Giant studios, and then modeled the appendages by stringing individual vertebrae on cables like beads on a necklace, adding various claws and devices at the end as needed-a death flower, a foot claw, a camera.

"We didn't build them with a rig like most articulated CG models," says Peter Nofz, CG supervisor. "Aaron Campbell created a method for building them that was like a big Lego set. That way, we had maximum flexibility to use what we needed for each shot."

Animating the appendages offered new challenges for the Imageworks crew. "Animating Doc Ock was like animating five characters-the central character [when he's CG] and the four arms," says Oscar-nominated animation director Anthony LaMolinara, who praises Aaron Campbell for his work on the tentacles and Chris Wagner for modeling Doc Ock's digital body.

"The tentacles have a unique movement," says LaMolinara, "it's not like an octopus. We had to determine what kind of power they had, how much weight they could pick up, and the effect of Doc Ock's body on them whether he was CG or live action. We had to shoot live action plates with Alfred's [Molina] body moving as if it were affected by the tentacles. When all those things are not handled right, it feels false. I think we hit our stride in the hospital sequence and when the tentacles smash into buildings."

Visually matching the CG tentacles with the practical ones built by Edge FX also proved to be a labor intensive task. John Monos and John Wallace used the latest "subdivision surface" features in Renderman and worked for months on dialing in the proper maps to make the CG tentacles indistinguishable from the practical ones on set. Kui Han Lee and Aaron Campbell continued refining and simplifying the model and it's animation controls to make sure that the resulting geometry could easily flow through the pipeline. The CG tentacles were so successful that they were used more and more, and were featured in many extreme closeups.

Because the audience can see his face, for shots where Doc Ock is fully digital the team decided to pull no punches in creating a believable, photorealistic digital double of the villain - and, similarly, of Spider-Man. Rather than use such commonly practiced rendering techniques as subsurface scattering to create digital flesh, the effects wizards chose a more innovative approach. And, rather than use keyframe animation or motion capture techniques to duplicate facial expressions, they, in effect, sampled Molina and Maguire's movements using the visual and kinetic equivalents of audio sampling.

"I knew from the start this would be our biggest challenge because there were shots where the camera was very close to the digital Doc Ock," says Stokdyk. "We've gotten good at creating CG hair. The big thing was his face - animating it and rendering his skin."

After considering many alternatives, Stokdyk decided to use the Light Stage system developed by Paul Debevec and implemented at USC's Institute of Creative Technology. With the Light Stage, the reflectance field of a human face is captured photographically. The resulting images can be used to create an exact replica of a subject's face from various viewpoints and in arbitrary, changing lighting conditions. To create Doc Ock's face, Molina was seated in a chair and surrounded with four film cameras running at 60 frames per second. Above his head was an armature that rotated around the chair. On the armature, strobe lights fired at 60 frames per second. "At the end of eight seconds, we had 480 images from each camera of his head in one position, but with different lighting conditions in each," Stokdyk says. The same procedure was used for Maguire.

While the core technique was designed by Mark Sagar, the real challenge was making it production worthy, which fell mostly into the hands of lighting leads John Monos and Jeff Stern, using Renderman work that Sing Foo implemented. In the cases where the data acquired from the lightstage was corrupt, John Schmidt and Dan Ziegler came up with an innovative "virtual lightstage" technique to recreate needed imagery.

Separately, to match the lighting on the set, the crew took high dynamic range images (HDRI) using fish-eye lenses to create 360-degree environments. Once a CG character's head was surrounded with those global environments, a software algorithm could calculate which of the 480 images from each camera to blend together to create the character's face. If the set was dark on one side, for example, the algorithm wouldn't access images taken on that side of the actor's face.

To create photorealistic movement for the actors' photorealistic faces, Imageworks developed a system based on motion capture; however, Remington Scott, senior technical director, is quick to note that the result is not a motion-captured performance.

Both actors were recorded at the House of Moves using Vicon motion capture cameras. Each wore 150 tiny markers in his face, each marker a 1.5 diameter, retro-reflective sphere flattened on one side.

"We wanted to create an alphabet of facial expressions," says Scott. "So, we had each actor pronounce selected phonemes, act out ranges of expressions, and do about 80 muscle actions." The muscle actions-puckering the lips, wrinkling the nose, raising a brow, and so forth-were based on the Facial Action Coding system (FACS) developed by Paul Ekman and Wallace Friesen.

Once the captured data was stabilized, Imageworks used it to create the relationships of individual muscles to each other and to the typography of the face through the application of proprietary scripts and tools written for Alias' Maya. The muscles, in turn deformed the skin. So, when an animator pulls or pushes points on a digital model to make Doc Ock smile, the muscles move the flesh on the digital model's face in exactly the same way that Molina's muscles move the skin on his face…or, Maguire's muscles move the skin on his face. And, because the texture of the actors' faces was captured exactly with the Light Stage system, the faces move and look real.

"We couldn't simply record an actor's performance," says Scott. "We had to create a flexible system that let us alter expressions and timing. Our digital doubles are not an animator's representation of an actor's performance. They are the actor's performance sampled through the computer."

The digital double's bodies, however, relied on more traditional keyframe animation techniques. No one, after all, can climb a building or swing through a city like Spider-Man can.

The crew kept Spider-Man's original model, changing only his costume to accommodate a new Spidey suit developed for Maguire that allowed his jaw to move. The physiquing created by Koji Morihiro, which provides the rig animators use to perform the digital character, also remained largely unchanged. The digital double's performance was enhanced, however.

"We more or less established Spider-Man's behavior and the way he moved in the first film," says LaMolinara. "But one thing that changed his performance was the involvement of martial arts stunt coordinators. I didn't want the film to look like a Hong Kong meat grinder movie, but their influence on what I was doing gave him a unique movement. Also, dealing with Dr. Octopus caused him to perform differently: The way he used the tentacles against Doc Ock caused a different kind of performance. And, the camera was flying around a bit more. There was a little more wildness on this film."

Leading a team of 18 animators who created the digital Spider-Man's performance were Jason Reising, Peter Giliberti and Chris Hurtt. The CG Spidey stars in 127 shots,

For some of the wilder Spider-Man shots, Dykstra used a computer-controlled system developed by Spydercam Systems that sends cable-based cameras on repeatable, complex 3D paths at great distances and speeds. "We used a Spydercam with a remote head that traveled up Wall Street on a cable for almost three-quarters of a mile," says Dykstra. "It went down something like 25 stories to the ground and back up again. At one point, the camera came so close to the ground that it hit the back window of a stunt driver's car."

Many of the shots, however, relied on virtual sets rather than the real streets of New York to help Spider-Man with his stunts, whether Spidey was Tobey Maguire, his stunt double or his digital double. The goal for this film was to make the virtual buildings more detailed so that Spider-Man could swing ever closer. "My idea was to make the buildings like characters," says Dan Abrams, effects lead, "individualized and fleshed out. We wanted the characters much closer to the buildings and wanted the buildings to look more realistic when Spider-Man streaks past them at 100 miles per hour. And, because the characters are moving so fast, even in 100-frame shots, the buildings in the distance are in the foreground in seconds."

To add detail, head modeler Alex Wong worked with photographic reference, matching detailed ornamentation in building facades and even the graffiti on the backs of buildings. "We weren't just painting bricks," says Abrams. "We added dirt, grunge, patterns. We did a lot of motion blur tests to see what type of details work best when a character is whizzing past. Otherwise, the buildings looked plastic." For room interiors, fish-eye photos taken of various office and apartment interiors were taken apart and reassembled in 3D space.

Ultimately, the modelers built 30 highly detailed buildings that the team placed in layouts for particular shots, arranging the buildings in various ways to create close-up urban settings with as many as 150 buildings. For shots where the camera pointed toward the street, the team created a full 3D city block with streets, intersections, bicycle racks, newspaper stands and other props. By changing textures procedurally, the generic city could take on different characteristics. "Remington Scott even added motion captured 3D people," says Abrams. "They hail cabs, look up and point at Spidey and Doc Ock on the side of a building."

One shot actually had a 30-block long virtual set with traffic. Of course, virtual buildings extremely far in the distance in many shots were matte paintings, but for one building, in one shot, the team actually modeled every brick. "Spidey gets thrown through it, so we see every brick offset and grout lines," says Abrams. "The camera is about one foot from the façade.

That facade was one of many shattered by Doc Ock and Spider-Man as they crashed into buildings and onto an elevated train in live action plates and virtual sets. Francisco de Jesus, digital effects supervisor managed the demolition for the film's 266 destruction shots.

Some of the most complicated shots took place during the train sequence fight. "Doc Ock is walking on top of the train, digging his claws in, so we had to dent the metal," says de Jesus. In some shots, a few of the train cars are real. In a few shots, the entire train is real. In many shots, however, the train is all CG, and CG cars are attached to real train cars.

In addition to denting trains, both real and CG, the digital destruction crew blew out windows during the fight, often matching virtual shards of glass to practical safety glass broken on set. "Even when the train car is real, the windows are usually CG," de Jesus says. "When Doc Ock reaches through a window to grab someone, there was no window on stage because Ock's arm would be CG. So we broke a lot of glass in CG."

The most dramatic destruction happens at the end of the sequence. "The train plows through what I can only describe as a gigantic spring with a stopper aimed at it," says de Jesus. "It shatters into thousands of pieces big and small."

To accomplish the destruction for these shots and others, de Jesus developed techniques that could be used whether the villain and superhero destroyed a plate or a CG model.

"I started with custom software to shatter solid volumes based on an algorithm that works in 2D or 3D, but it's easier to explain in 2D," he says. "Imagine if you were to draw points on a blank paper. It would calculate the halfway mark between every point and a neighbor so you end up with a cellular diagram where the center of each cell is the point you gave it. So, a high density of points yielded a lot of shattered pieces of debris and fewer points farther apart produced bigger chunks."

To break a building, dent a train, or shatter a window in a live-action plate, the team started with a match move to derive the camera and then created simple geometry to represent the building. This geometry was destroyed with the shatter software. "We adapted a ray tracing algorithm so that an impact point could automatically calculate all the seed points," says de Jesus. To make the effect look real, the plate was projected onto exterior surfaces and locked there before it was shattered so that the live action image would move with and stick to the debris.

The train fight also caused to team to rethink the system used on previous films for cloth simulation. "We knew Doc Ock would be a challenge because he would be running around on a train moving at 60 miles per hour," says Nofz. The team had been considering using a technique called object cloth, which simplifies the methods used to create digital costumes. Doc Ock provided the push they needed. "We asked Alias to help us create a new cloth pipeline," says Nofz. The simplification helped make possible the computationally intense task of simulating all the layers of Doc Ock's clothes simultaneously, even when the villain was on top of the fast-moving train. "We couldn't simulate the layers of clothing separately because they interacted with each other," Nofz says.

As on most effects films, the unsung heroes of Spider-Man 2 were the artists in the match move department. Led by Rachel Nicoll they derived the wild camera moves from live action plates so that the digital characters, buildings and effects could be inserted.

Doc Ock and Spidey, for example, have their own unique weapons, each of which is created with visual effects. As in the first film, Spider-Man's web provides his means of transport, but in this film he also uses it to net evil doers and to fling as a web ball. "The web ball is just a bunch of webs," says Theo Vandernoot, digital effects supervisor, who created the Spidey's web for the first film. "The web didn't change from the first movie."

Doc Ock, though, was given a different sort of ball to play with-an energy ball. "It starts off as a nuclear pellet, then grows and gets more and more sophisticated until it's the size of a Volkswagen," Vandernoot says. "People lose control of it; it consumes things."

Designed to look like the sun, the CG fire ball has bubbling hot spots and "prominences" that erupt from its surface. "The mini prominences, which are loops, are actually pieces of fur, that, like Spider-Man's web, are turned into pipes so we can have several thousand of them," he says. "Hero prominences that look like solar flares, like big loops that change shape, twist, grow and shrink, were rendered as larger pipes with textures flowing through them."

The energy ball is sometimes contained in a bag that's pushed and pulled by the prominences and also by Doc Ock's tentacles. "When a prominence pushes the bag, the bag pushes Doc Ock's hand," says Vandernoot. "But the tentacles also drive the animation, so we had to find a balance between character animation and effects. I think the interconnection between the two was a milestone for this company."

Because the energy ball was made of hundreds of elements, the technical crew decided to render it in black and white. Compositors led by compositing supervisor J.D. Cowles added colors later using brightness to determine which elements were light yellow and which were hot red as they combined elements to create the energy ball. Imageworks has developed its own compositing software to handle work such as this.

During the final battle, the energy ball lands (along with various other objects) in the river, which meant the crew needed to create yet another effect: CG water. "We needed to art direct the water, so we used a surface deformed by forces with a shader and particles layered on top-all done through Side Effects Software's Houdini and Pixar's RenderMan," says Vandernoot. "If the water interpenetrates the objects being dropped into it, we put foam around the boundary so you never see it."

All these effects contribute to the overall magic-toward helping the audience believe that a superhero really can swing between skyscrapers on Wall Street. "The magic we do is figuring out how to capture the 3D world in a 2D medium in a way that gives the viewer a sense of reality," says Dykstra. "I think we did that with Doc Ock - with his tentacles even though they have no parallel in reality and with the ground-breaking methods we used for creating CG flesh. And, I think the audience will have their appetite sated for being up high and moving fast, for that vertiginous sense of falling. Audiences are so sophisticated, when you do a movie like this you have to promise to do things you have no idea how to do. If you rely on known or mature techniques they'll be obsolete by the time the movie comes out."