Is Nvidia also thinking about actual 3D images?

[user33559]

Hello,

Are generative or predictive models being considered for images that are actually 3D in size? For example, 3D scans of objects or human bodies (e.g., gray-scale 3D bodies of [256, 256, 256, 1])? Such problems usually require fully 3D-aware models.

If so, what are the challenges on top of those of 2D images?

This will have significant applications in scientific fields such as medicine and geoscience.

Many thanks,
Reza

[jung1]

Hi, GET3D is able to genearte 3D meshes as it output, but generating actual 3D voxels is not considered at the moment. It would also be possible try to use similar idea from GET3D for this task (e.g. rendering the 3D bodies into 2D images and apply discriminator on the 2D images for supervision) we’d love to see how this can work!