Generate a Point Cloud From Depth Data

Introduction

A point cloud is a set of vertices in a three-dimensional coordinates system.These vertices are defined in the coordinates (x,y,z) and are used by the description of the outerof an object.

In this tutorial you will learn how to transform a depth image into a 3D point cloud, we will also learn how to assign color to the point cloud and finally how to visualize this point cloud, all using Python. But before going to the code, let's first learn some concepts.

Intrinsic Matrix

Intrinsic parameters are camera specific. Includes information such as focal length (f_x, f_y), optical centers (c_x, c_y), etc. Also called a camera array. It depends only on the camera, so once calculated, it can be stored for future purposes. It is expressed as a 3 × 3 matrix:

These parameters are found through the calibration of a camera, with the calibration it is possible to relate these measurements from the camera with measurements from the real three-dimensional world. This is important because scenes are not just three-dimensional, they are also physical spaces with physical units, therefore the relationship between the natural units of the camera (pixels) and the units of the physical world is a critical component in any attempt to construct a three-dimensional scene as it is in this case.

This tutorial does not explain how to calibrate a camera, it is assumed that you already have the intrinsic matrix of your camera.

Homography Matrix

A Homography is a projective transformation or perspective transformation that determines a bijective correspondence between the points of two planes

This transformation of perspective is necessary, since the rgb camera with the depth camera are not exactly alienated due to their physical conditions. This transformation can be represented by a 3x3 matrix:

The intention of the homography is that the two cameras are referenced to the same coordinate system, so the data fusion will simply be a color correspondence to the point cloud.

Data Fusion

With the point cloud already generated, the process for assigning color to the point cloud is now explained. The coordinate system of interest is that of the depth camera, RGB images are transformed to this system, with the images related to the depth camera coordinate system, the color of each pixel in position (u, v) in the image resulting from the transformation, it corresponds to the color of each vertex at the position (u, v) of the depth image, that is: