Point E is the new AI of OpenAI for image generation
OpenAI has expanded the capabilities of its text-to-image software from two to three dimensions with the release of Point-E, an AI that generates 3D images from text prompts. According to a document published with the codebase, Point-E does not require a high-end computer to run and can generate a model in under two minutes using a single Nvidia V100 GPU.
With this we can see that 3D model generators could be the next breakthrough that will take the world of AI by storm. OpenAI has been very active this year, since for example here on the blog we have talked about the company for its two major projects: Dall-E 2 and ChatGPT.
Between these two massive AI platforms, the company helped generate images and long stretches of text from a simple text message, and now the company is back with a third concept, launched just before Christmas to generate buzz. This third concept, called Point-E, follows a similar pattern, creating 3D content from simple prompts.
Announced in a research article published by the OpenAI team, Point-E works in two stages: first it uses a text-to-image AI to convert your verbal request into an image, then it uses a second function to transform this image into a 3D model.
Point-E is open source and the source code is available on GitHub . Even so, it can be a bit tricky to get started, as users need to be reasonably familiar with command-line tools and the system must have Python installed, unlike ChatGPT, where users can register on a website and test their skills.
Point-E does not create 3D objects in the traditional sense; rather, it generates point clouds, that is, discrete sets of data points in space that represent a 3D shape, hence the cheeky abbreviation.
(The "E" in Point-E is short for "efficiency" as it is apparently faster than previous approaches to rendering 3D objects.) According to the developers, point clouds are computationally easier to synthesize, but they don't capture the shape or fine texture of an object, which is a major limitation of Point-E currently.
To overcome this limitation , the Point-E team trained an additional AI system to convert Point-E's point clouds into meshes. (Meshes—collections of vertices, edges, and faces that define an object—are commonly used in 3D modeling and design.) But the researchers note in their paper that the model can sometimes miss parts of objects , resulting in distorted or blocked shapes. In addition to the mesh-generating model, which is self-contained, Point-E consists of two models: a text-image model and a 3D image model.
The text-to-image model, similar to artwork generation systems like DALL-E 2 and Stable Diffusion, was trained on labeled images to understand the associations between words and visual concepts . The 3D image model, on the other hand, was fed a set of images associated with 3D objects to learn how to translate the two effectively. When given a text message, for example, "a 3D-printable gear, a single gear 7 cm in diameter and 1 cm thick," Point-E's text-to-image model generates a rendered synthetic object that is then passed to the 3D image model.
The latter then generates a cloud of points. After training the models on a dataset of "several million" 3D objects and associated metadata, Point-E was able to produce colored point clouds that frequently matched text prompts, the OpenAI researchers say. Not perfect: Point-E's 3D image model sometimes doesn't understand the image from the text image model, resulting in a shape that doesn't match the text message. Still, it's much faster than the previous state of the art, at least according to the OpenAI team.
The result is far from reaching the quality of a commercial 3D rendering in a movie or a video game. But it's not supposed to be. Once they are fed into a 3D application like Blender, they can be turned into textured meshes that look more like normal 3D images.
"Although our method is still not up to scratch in terms of sampling quality, it is one or two orders of magnitude faster, which is a practical compromise for some use cases," the OpenAI researchers explain in the paper describing the project.
Finally, if you are interested in learning more about Point-E or the source code, you can find the details at the following link.