Nvidia GANverse3D: from an image to a 3D model, the power of neural networks

From 2D images to 3D animated objects, effortlessly. GANverse3D, being developed at Nvidia’s AI Research Lab in Toronto, it allows images to be brought to life, turning them into objects that can be viewed and controlled in virtual environments within Nvidia Omniverse.

For now this is possible with well-defined and non-articulated shapes, such as cars, but in the future it could even mature to give life to a person’s image, immediately creating a 3D counterpart.

GANverse3D could help architects, game developers and designers to easily add new elements to their models without the need for 3D modeling experience or a large budget to spend on rendering. Nvidia researchers illustrated their work in un paper su ArXiv.

The original photo

The single photo of a car can be turned into a 3D model complete with headlights, taillights and everything in between, thus quickly creating an object that you can move within a virtual scene. To generate the training dataset, the researchers leveraged a opposing generative network (generative adversarial network), O GAN, to synthesize images that depict the same object from multiple points of view, such as a photographer walking around a parked vehicle, shooting from different angles.

These images from multiple viewpoints are then inserted into a rendering framework for reverse graphics, which allows to infer 3D mesh models from 2D images. Once trained with images from multiple views, GANverse3D only needs a single 2D image to obtain a 3D mesh model. This model can be used with a 3D neural renderer that allows developers to customize objects and change backgrounds.

When imported as an extension into the Omniverse platform and rendered by an Nvidia RTX GPU, GANverse3D can be used to recreate any 2D image in 3D; the company demonstrated this technology with KITT, the car from the popular TV show Knight Rider (Supercar) from the 1980s.

The 3D model obtained with GANverse3D

Previous models for reverse graphics relied on 3D shapes as training data. Instead, without the help of 3D assets, “we have transformed a GAN model into a very efficient data generator in order to create 3D objects from any 2D image on the web“said Wenzheng Chen, Nvidia researcher leading the project.” Because we trained on real images instead of the typical pipeline, which is based on synthetic data, the AI ​​model generalizes better with real world applications. ” added researcher Jun Gao, an author on the project.

Game creators, architects and designers rely on virtual environments such as Nvidia Omniverse to test new ideas and visualize prototypes before creating final products. With Omniverse Connectors, developers can use their favorite 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

Comparison between real image and GANverse3D

The problem is that not all content creators have the time and resources to create 3D models of every object they design. The cost of capturing the number of multi-view images required to render a car in a showroom or street buildings can be prohibitive. And this is where GANverse3D comes in.

To recreate KITT, Nvidia researchers fed the trained model with an image of the car, leaving to GANverse3D the task of predicting the corresponding 3D textured meshas well as various parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the intended texture into high-quality materials that give KITT a more realistic look, and then put the car in a dynamic sequence alongside other cars.

The GAN model allows researchers to manipulate the layers (the layers) of its neural network to turn it into a data generator. More precisely, the researchers found that by opening the first four layers of the neural network and freezing the remaining 12, the GAN network rendered images of the same object from different points of view. On the contrary, freezing the first four layers and leaving the other 12 variables led the neural network to generate different images from the same point of view.

By manually assigning standard viewpoints, with vehicles depicted at a specific camera height and distance, researchers can quickly generate a multi-view dataset from single 2D images. The final model, trained with images of 55,000 cars generated by GAN, proved superior to an inverse graphical network trained with the popular Pascal3D dataset.