Camera calibration and 3D voxel reconstruction

Three connected assignments in C++ with OpenCV: calibrate the cameras, rebuild a person in 3D from four video views, then tell four people apart by the colour of their clothes.

When
Early 2022
Where
MSc coursework, Computer Vision course at Utrecht University, in a group of two
Built with
C++, OpenCV, K-means, Gaussian mixture models
Made as
Team project

About the project

The problem. Rebuild a person in 3D from ordinary video, and keep track of who is who when several people walk around. The course split this into three connected assignments, in C++ with OpenCV: calibrate the cameras, reconstruct a person from four camera views, and tell four people apart by the colour of their clothes.

Step 1: camera calibration. Before a camera can measure anything, you need to know how it turns the 3D world into a 2D image. That mapping is described by the intrinsic matrix: the focal length and the image centre, in pixels. We estimated it from twenty photos of a checkerboard, a pattern whose corners sit at known positions. The quality measure is the reprojection error: project the known corners back into each photo with the estimated camera and measure how many pixels they land from the corners actually found. Photos that raised this error were dropped. We also tried median filtering the photos first (replacing each pixel by the median of its neighbours, to remove noise); it changed the error only in the fifth decimal. As a visual check, we drew a 3D cube onto a live webcam image using the calibration.

Step 2: 3D voxel reconstruction. Four calibrated cameras film the same scene. Each camera first separates the person from the room, which is called background subtraction: a model of the empty room is trained on a video of the empty room, and every pixel that does not fit it is foreground. The model is a mixture of Gaussians for each pixel, a few bell curves that together describe the colours that pixel normally shows. The space is then divided into voxels, small cubes that are the 3D version of pixels. A voxel is kept only if all four cameras see it as foreground; the voxels that remain form the person.

  • Cleaner masks: a morphological opening (shrink the foreground mask, then grow it back, which erases small specks) and contour filtering (keep only the large outlines) removed a hole in the person and the clutter around the chair.
  • A smaller voxel space: trimming it made it 21.7% smaller, so there are fewer voxels to check.
  • Speed: preparing the background model once per camera, instead of again for every frame, cut that step from about 240 ms to about 2 ms per frame.

Step 3: colour labelling. Now four people walk through the scene, and the program has to say who is who in every frame. K-means clustering, which splits points into a chosen number of groups (here four) around group centres, groups the voxels by their position on the floor plane: one cluster per person. To know which cluster is which person, each person gets a colour model: a Gaussian mixture model of their upper-body colours, trained on one frame where everyone stands well apart, seen from two cameras. In every later frame these models decide which cluster is who, which gives each person a fixed colour and a path over time (third picture).

My part. I set up and kept our group's repository, and most of the code went in through me. In the reconstruction I made the background step fast: the background model is now built once per camera when it starts, instead of again for every frame, which took the step from about 240 ms to about 2 ms. I also wrote the clean-up of the foreground masks with contour filtering. In the colour labelling I added the handling of occluded voxels to the offline phase, where the colour model of each person is built.

More from Computer vision, AI and image processing

Get in touch

Let’s talk if you have a project in mind.