NVIDIA Instant NeRF, the artificial intelligence that turns 2D photos into 3D scenes

At GTC 2022, the NVIDIA research team unveiled an impressive technology called Instant NeRF that could turn 2D photos into 3D scenes in seconds.

To do this, the researchers harnessed the power of AI, more specifically through so-called neural radiance fields. LNeRFs use neural networks to reconstruct the scene from the images by predicting the color of light radiating in any direction. NVIDIA claims that Instant NeRF is the fastest technology of its kind to date, enabling multiplicative acceleration of over 1000x in some cases, so that 1080p rendering happens in just milliseconds.

As NVIDIA says, this is a reverse render, since instead of using a three-dimensional model to launch rays that bounce off surfaces directly and indirectly and thus create a photorealistic two-dimensional image, as they do in video games, reverse rendering analyzes two-dimensional photographs to try to reconstruct the scene in three dimensions. The technology analyzes the images and artificial intelligence is capable of deducing the origin of each pixel in the image, reconstructing all of reality even though it cannot see it.

Nvidia is evolving these techniques at incredible speed.

Nvidia has evolved this method to make this process almost instantaneous. Once the neural network has reconstructed the scene, you can move freely through it, in real time, and re-photograph the scene from any angle, even from behind a person, in real time.

Unsurprisingly, Instant NeRF was developed using the CUDA Toolkit and the Tiny CUDA Neural Networks library. They can access your code on this page, as according to NVIDIA, the neural network is light enough to easily run on a single GPU, especially if equipped with Tensor Cores.

David Luebke, vice president of graphics research at NVIDIA, said:

If traditional 3D representations like polygon meshes are similar to vector images, NeRFs are like bitmap images: They densely capture the way light radiates from an object or within a scene. In that sense, Instant NeRF may be as important to 3D as digital cameras and JPEG compression were to 2D photography, vastly increasing the speed, ease, and scope of 3D capture and sharing.

Applications for Instant NeRF technology can be many, from quickly scanning real people or environments so game creators can use digital scans in their designs, to training autonomous cars or robots to understand the shape and size of real objects.