Fast single image to 3d objects generation

Abstract

Disclosed are systems and methods for generating a 3D model from a single 2D image, the method comprising: receiving a single 2D input image of an object; generating a set of consistent multi-view images based on the single 2D input image using a fine-tuned 2D diffusion model that processes multiple views together in a tiled configuration; constructing a 3D feature volume by projecting 2D patch features from the generated multi-view images using corresponding camera pose information; generating a 3D mesh using a pair of 3D diffusion networks conditioned on the multi-view images, wherein the pair of 3D diffusion networks comprises a first network for generating a coarse occupancy volume and a second network for generating a high-resolution sparse volume; and refining a texture of the generated 3D mesh to produce a textured 3D mesh.

What is claimed: 
     
         1 . A method for generating a three-dimensional (3D) model from a single two-dimensional (2D) image, the method comprising:
 receiving a single 2D input image of an object;   generating a set of consistent multi-view images based on the single 2D input image using a fine-tuned 2D diffusion model that processes multiple views together in a tiled configuration;   constructing a 3D feature volume by projecting 2D patch features from the generated multi-view images using corresponding camera pose information;   generating a 3D mesh using a pair of 3D diffusion networks conditioned on the multi-view images, wherein the pair of 3D diffusion networks comprises a first network for generating a coarse occupancy volume and a second network for generating a high-resolution sparse volume; and   refining a texture of the generated 3D mesh to produce a textured 3D mesh.   
     
     
         2 . The method of  claim 1 , wherein generating the set of consistent multi-view images comprises:
 tiling six views into a single image with a 3×2 layout;   defining camera poses for the multi-view images using fixed absolute elevation angles alternating between about 30° and about −20°, coupled with azimuths commencing at about 30° and incrementing by about 60° for each subsequent pose; and   generating the tiled image using the fine-tuned 2D diffusion model conditioned on the single 2D input image.   
     
     
         3 . The method of  claim 1 , wherein the fine-tuned 2D diffusion model incorporates:
 local conditioning through reference attention that appends self-attention key and value matrices from the 2D input image to corresponding attention layers for the multi-view image;   global conditioning using contrastive language-image pre-training image embedding as a global semantic understanding of the object; and   a linear noise scheme for the diffusion process.   
     
     
         4 . The method of  claim 1 , wherein generating the 3D mesh comprises:
 initializing a low-resolution 3D grid with Gaussian noise;   denoising the grid using the first diffusion network to produce a coarse occupancy volume;   subdividing each predicted occupied voxel into smaller voxels to construct a high-resolution sparse volume;   initializing the sparse volume with Gaussian noise;   denoising the sparse volume using the second diffusion network to predict signed distance function (SDF) values and color for each voxel; and   applying a Marching Cubes algorithm to extract a textured mesh from the denoised volume.   
     
     
         5 . The method of  claim 1 , wherein refining the texture of the generated 3D mesh comprises:
 rendering the 3D mesh from multiple views that match the camera poses of the generated multi-view images;   comparing the rendered views with the generated multi-view images;   optimizing a color field to minimize differences between the rendered views and the generated multi-view images using a l2 loss function; and   transferring the optimized color field to the surface of the textured 3D mesh surface.   
     
     
         6 . The method of  claim 1 , further comprising:
 processing the generated textured 3D mesh to create simplified collision models for physical simulation;   analyzing one or more geometric properties of the 3D mesh to identify potential grasping points and manipulation affordances; and   using the identified affordances to guide robot interaction planning when working with objects resembling the generated 3D model.   
     
     
         7 . The method of  claim 1 , further comprising training a robot to interact with physical objects by using the generated textured 3D mesh in a simulation environment that simulates physical interactions between the robot and virtual representations of the physical objects. 
     
     
         8 . The method of  claim 7 , further comprising:
 generating multiple different 3D meshes of various objects from corresponding 2D input images;   populating a virtual training environment with the generated 3D meshes; and   training the robot in the virtual training environment to perform grasping, manipulation, and navigation tasks with respect to objects represented by the 3D meshes.   
     
     
         9 . The method of  claim 8 , wherein training the robot in the virtual environment further comprises:
 generating variations of the 3D meshes with different textures, sizes, and orientations to enhance the robustness of the robot's learned policies;   simulating different lighting conditions and environmental factors to improve generalization capability of the robot; and   gradually increasing the complexity of manipulation tasks to enable progressive learning.   
     
     
         10 . The method of  claim 7 , wherein training the robot in the simulation environment comprises:
 simulating physics-based interactions between the robot and the generated 3D objects;   collecting training data from the simulated interactions;   using the collected data to train machine learning models that predict optimal grasping points, manipulation strategies, and object recognition capabilities; and   deploying the trained models on physical robots to enable effective interaction with real-world counterparts of the simulated objects.   
     
     
         11 . A system for generating a three-dimensional (3D) model from a single two-dimensional (2D) image, the system comprising:
 one or more processors; and   at least one memory storing instructions that, when executed by the one or more processors, cause the system to:   receive a single 2D input image of an object;   generate a set of consistent multi-view images based on the single 2D input image using a fine-tuned 2D diffusion model that processes multiple views together in a tiled configuration;   construct a 3D feature volume by projecting 2D patch features from the generated multi-view images using corresponding camera pose information;   generate a 3D mesh using a pair of 3D diffusion networks conditioned on the multi-view images, wherein the pair of 3D diffusion networks comprises a first network for generating a coarse occupancy volume and a second network for generating a high-resolution sparse volume; and   refine a texture of the generated 3D mesh to produce a textured 3D mesh.   
     
     
         12 . The system of  claim 11 , wherein generating the set of consistent multi-view images comprises:
 tiling multiple views of the object into a single composite image arranged in a grid layout; and   generating the composite image using the fine-tuned 2D diffusion model conditioned on the single 2D input image, enabling cross-view attention during the diffusion process.   
     
     
         13 . The system of  claim 11 , wherein the fine-tuned 2D diffusion model is configured to generate the multi-view images with predetermined camera poses comprising:
 fixed absolute elevation angles; and   relative azimuth angles with respect to the input image view.   
     
     
         14 . The system of  claim 11 , wherein the first network of the pair of 3D diffusion networks generates a low-resolution occupancy volume using 3D convolution, and the second network generates a high-resolution sparse volume using 3D sparse convolution. 
     
     
         15 . The system of  claim 11 , wherein the instructions further cause the system to refine the texture of the generated 3D mesh by:
 optimizing a color field represented by a tensor radiance field (TensoRF) while maintaining a geometry of the generated 3D mesh; and   baking the optimized color field onto the mesh.   
     
     
         16 . The system of  claim 11 , wherein constructing the 3D feature volume comprises:
 extracting 2D patch features from each of the multi-view images using a pre-trained vision model;   projecting each 3D voxel within the 3D feature volume onto the multi-view images using known camera poses; and   aggregating corresponding 2D patch features through a shared-weight multilayer perceptron followed by max pooling.   
     
     
         17 . The system of  claim 11 , wherein the instructions further cause the system to perform text-to-3D generation by:
 receiving a text prompt describing an object;   synthesizing a reference image based on the text prompt using a text-to-image model; and   processing the synthesized reference image through the consistent multi-view generation and 3D diffusion pipeline to produce a textured 3D mesh corresponding to the text prompt.   
     
     
         18 . The system of  claim 11 , wherein the instructions further cause the system to train a robot to interact with physical objects by using the generated textured 3D mesh in a simulation environment that simulates physical interactions between the robot and virtual representations of the physical objects. 
     
     
         19 . The system of  claim 18 , wherein the instructions further cause the system to:
 generate multiple different 3D meshes of various objects from corresponding 2D input images;   populate a virtual training environment with the generated 3D meshes; and   train the robot in the virtual training environment to perform object manipulation tasks before deployment in a physical environment.   
     
     
         20 . The system of  claim 18 , wherein training the robot using the simulation environment comprises:
 using reinforcement learning to train robot control policies based on interactions with the generated 3D meshes; and   transferring the learned policies to a physical robot for real-world object manipulation.