3D To 2D Tracking In Nuke

How to create 2D tracks using a matchmove camera.

Free Tracks

If you have a matchmove camera and a 3D scene, you can get accurate 2D tracks for free.

Let’s say you want to track a roto shape to your scan, and you have a matchmove camera and geometry (or a point cloud to reference the scene). Typically, you would line up a Card to the area of the scene that you want to roto, project your roto onto the Card, and output the projection via a ScanlineRender node using the matchmove camera.

Working with projected rotos can feel a bit unwieldy and difficult, though, especially if you have to animate the roto shapes. As you get further away from the projection frame, moving the splines into place becomes more and more guesswork.

Instead of this trial and error approach, it’s very useful to instead convert the matchmove data into 2D tracks and link those directly to the Roto node. That way, you can work with the splines pretty much in the same way as you would during a normal roto task, without any projection.

Reconcile3D

Nuke has a great node which you can use to convert 3D object positions into 2D tracks using your matchmove camera. It’s called Reconcile3D.

The Reconcile3D node needs two inputs to generate a 2D track: a Camera and a 3D position (XYZ).

For the camera, just connect your matchmove camera to the Reconcile3D node’s camera input.

For the 3D position, you can either type or paste a value into the Reconcile3D's point knob (under Input), or create an Axis node and connect it to the axis input. Here, we're going to use an Axis node.

The Axis node has to be positioned at the point in 3D space which you want to track. If you want to track the corner of a wall, the Axis has to be placed exactly at the corner of the wall in 3D space in order to generate an accurate 2D track.

View your geometry (or the point cloud of the scene) and move your Axis to the position you want to track. A very easy way to do this is to go into Vertex selection mode in the Viewer, select a vertex on the geometry, and use the snap function in the Axis node to snap it into position:

Snapping an Axis into position in 3D space.

Once you have positioned your Axis and have connected both it and the Camera to the Reconcile3D node, hit the create keyframes button in the Reconcile3D node's properties. Select your frame range and hit OK. It will calculate the 2D track for you in the XY output (output knob).

If you want to create another 2D tracking point, create a new Reconcile3D node and connect a new Axis node to it which is positioned where you want to track the point in 3D space. Hit create keyframes in the new Reconcile3D node to generate another 2D tracking point. Rinse and repeat for as many 2D tracking points you need.

Once you have your 2D tracking points, create a Tracker node. Click add track to add a blank track to the Tracker. Then, copy/paste the animation from the XY output in the Reconcile3D node to the new track in the Tracker node:

Copying the animation from the Reconcile3D node to a new track in the Tracker node.

If you have more Reconcile3D nodes with 2D tracking points, copy their XY output animations into new tracks in the same Tracker node.

You now have a Tracker node with accurate 2D tracks based on your 3D scene and your Camera, and you can use that however you like. You can change your reference frame, select which points you want to contribute to the tracking output, select if you want translation, rotation, and scale, and so on.

Going back to the roto example, you can link the Tracker directly to a Roto node:

Linking a Tracker to a Roto node.

You can now roto as you normally would, without using a projection, and it will track with your scan. However, you have to add lens distortion after your Roto node to accurately line it up at the edges of the frame.

I hope you found this tutorial useful. For more Nuke tips & tricks, see Nuke.