K.U.Leuven > ESAT > PSI > Visics > Research > Topics > Item 1.3

Augmented reality: fusion of virtual objects with real video

L. Van Gool, M. Pollefeys, K. Cornelis

Augmented reality is a fast growing research domain at the edge of computer vision and computer graphics. Its goal is to incorporate seamlessly virtual entities into the real environment in which we live. The augmented world can be observed through special head-mounted displays or through a recorded video. Augmented reality applications range from computer-aided surgery were graphical models or photographs can be overlaid onto the patient, to movie special effects were the virtual entities add to the increase in entertainment value.

The 3D modelling software developed in the group, that extracts both the shape of the scene and the motion of the camera, was adapted to deal with continuous video sequences instead of sets of separate images. The most important difference lies in the search for corresponding features. While for separate views a matching algorithm is used that can deal with relatively large changes of viewpoints between images, in the context of video it is much more efficient to use a tracking algorithm that follows salient features from one frame to the next. The tracking of a feature can persist a long time leading to the determination of corresponding features in video frames that can be widely separate in time. The structure and motion computation that recovers both the 3D structure of the feature set and the motion and calibration of the camera, is initialised from two well-separated views that are selected for this purpose. Next, the pose of the camera is determined for all other frames and the final result is optimised through a bundle-adjustment procedure. This yields a globally optimal result that also takes into account distortion effects such as radial distortion.

The estimate camera parameters can be used to instantiate virtual cameras in a computer graphics package. The virtual entities that we want to merge with the video can then also be imported in this package and rendered from the viewpoints of each frame's camera. These generated virtual images can then be overlaid onto the real video material to generate an augmented version of the video.

A movie showing an augmented video fragment is available as augm.mpg. A frame from this movie is shown below.