This time I am making a digression from my usual posts about the making of One Soul, to discuss cinematic set extensions. I had the fortune to be able to get some drone footage of the neighborhood, and wanted to practice working with an actual shot, as opposed to pure CG environments.
SHOOTING WITH THE DRONE
First of all, a big thanks to Rentaro Hashimoto, who was kind enough to spend his holiday shooting drone footage for this project! He was using a DJI Mini 3 Pro, as shown in the picture below.
It is very capable, comparable to the popular DJI Mavic 2 Pro. We used the 10-bit D-Cinelike colorspace to have enough to work with later in the compositing. It was a very windy day, so not the most ideal for shooting, and there were some hiccups with the communication with the drone and the remote control.
SELECTING THE SHOT
Going through the footage, there was one section that l liked where the drone was backing up, catching a good view over the Great Buddha. This was perfect to use if played in reverse. Due to the Golden Week holidays in Japan, there were many tourists, and as they were walking backwards in the reversed footage, they all had to be removed.
In place of the Great Buddha, I would place a pagoda I previously modeled in Houdini.
EXTRACTING THE IMAGE SEQUENCE
The movie files were encoded as H.265 (HEVC). This is not supported by Nuke, so to extract the image sequence, I used Premier Pro. It can convert to both DPX and EXR, but I found EXR to be easier to work with.
COMPENSATING FOR LENS DISTORTION
I solved the lens distortion in Nuke, using a photo of a grid pattern shot using the drone camera.
MATCHING THE CAMERA MOVEMENT
My first attempt was to use the camera tracker in Nuke. It can be used to automatically detect features, as well as to calculate lens distortion. Though, the result was not very accurate. I considered using 3DEqualizer, which is the industry standard software for match move. It is very expensive, but you can subscribe for a week, and do all your match move.
I decided to see how far I could push it using Nuke only, and got a decent result by solving the lens distortion separately, and manually tracking the features using the 2D tracker. These tracks could then be imported into the camera tracker, with an error of just 0.32, which is much better than when using just the camera tracker itself.
To help with aligning CG set extension elements like the pagoda, I also generated a point cloud in Nuke.
PROJECTION PAINTING
The full image sequence is 101 frames. That makes it quite slow to iterate when using full 3D renders. I started by rendering out a one frame AO pass of the pagoda, and projected it onto the geometry. Then, I painting the frame using roto paint in Nuke, and later rendered it out and continued painting in Photoshop.
This projection will cover the full pagoda at the particular camera angle at that frame. Though, when viewed from another angle at a different frame, some areas will lack cover, as shown in blue (and a tiny bit in red) below.
I rendered out AO passes for the first and last frame as well, to cover these. When I am happy with the main projection, I will paint in the complementary projections as well.
All the people in the shot were removed using the same projection technique.
THE FOREGROUND TREES
As can been seen in the image of the original plate with the Great Buddha above, there are some trees in the foreground with many tourists walking under the canopy, that is also swaying heavily in the wind. After painting out the people, I attempted to get a key of the canopy, and put the canopy back on top.
After endless attempts, the best result I could get was to convert the frame to YUV, and use the chroma to get the key.
The result was rather good when looking at a single frame, but when animated, the people walking under the branches caused a flickering artifact. Frame blending the key improved it a little, but the result was still not convincing.
ADDING SPEED TREES
I had to give up, and use some Japanese Cedar trees from the SpeedTree Cinema library instead. I blocked out the placement to fill the shot where the original trees had been, and rendered the full image sequence in Clarisse. The lower part was cut off using a plane, and the result blends in fairly well.
As full CG renders are very time consuming, I only used a low quality render for the first iteration, so you can see some noise. Also, the trees should be replaced with animated versions, as it was a windy day.
ADDING A CROWD
After removing all the original tourists, the shot looked a little bit empty. Being a big fan of Houdini, this was a good opportunity to get familiar with the crowd simulation feature. I did a simple setup with Mixamo animations.
FUTURE IMPROVEMENTS
There is still a huge list of things to improve. Refining the pagoda painting, and adding the missing projections. Grounding it a little better in the shot. Filling out the empty court yard with props. Adding variation and animations to the trees. Render them at higher quality. Make the crowd simulation more interesting. Work a bit on the color grading. Hopefully I'll have the time to add some of these things in the future.