Fusing neural radiance fields by registration and blending

Abstract

A method for fusing neural radiance fields (NeRFs) is described. The method includes re-rendering a first NeRF and a second NeRF at different viewpoints to form synthesized images from the first NeRF and the second NeRF. The method also includes inferring a transformation between a re-rendered first NeRF and a re-rendered second NeRF based on the synthesized images from the first NeRF and the second NeRF. The method further includes blending the re-rendered first NeRF and the re-rendered second NeRF based on the inferred transformation to fuse the first NeRF and the second NeRF.

What is claimed is: 
     
         1 . A method for fusing neural radiance fields (NeRFs), the method comprising:
 re-rendering a first NeRF and a second NeRF at different viewpoints to form synthesized images from the first NeRF and the second NeRF;   inferring a transformation between a re-rendered first NeRF and a re-rendered second NeRF based on the synthesized images from the first NeRF and the second NeRF; and   blending the re-rendered first NeRF and the re-rendered second NeRF based on the inferred transformation to fuse the first NeRF and the second NeRF.   
     
     
         2 . The method of  claim 1 , in which the first NeRF and the second NeRF comprise representations of a 3D scene. 
     
     
         3 . The method of  claim 1 , in which blending comprises compositing predictions from the first NeRF and the second NeRF to form blended images having an image quality greater than images individually rendered by any one of the first NeRF or the second NeRF. 
     
     
         4 . The method of  claim 1 , in which a scene is represented by a plurality of NeRFs. 
     
     
         5 . The method of  claim 1 , in which each of the plurality of NeRFs having a neighbor NeRF. 
     
     
         6 . The method of  claim 1 , further comprising training the first NeRF and the second NeRF on a separate set of images. 
     
     
         7 . The method of  claim 6 , in which the separate set of images capture different, overlapping, portions of the same scene. 
     
     
         8 . The method of  claim 1 , further comprising planning an object grasp by a robot according to a fused image from the first NeRF and the second NeRF. 
     
     
         9 . A non-transitory computer-readable medium having program code recorded thereon for fusing neural radiance fields (NeRFs), the program code being executed by a processor and comprising:
 program code to re-render a first NeRF and a second NeRF at different viewpoints to form synthesized images from the first NeRF and the second NeRF;   program code to infer a transformation between a re-rendered first NeRF and a re-rendered second NeRF based on the synthesized images from the first NeRF and the second NeRF; and   program code to blend the re-rendered first NeRF and the re-rendered second NeRF based on the inferred transformation to fuse the first NeRF and the second NeRF.   
     
     
         10 . The non-transitory computer-readable medium of  claim 9 , in which the first NeRF and the second NeRF comprise representations of a 3D scene. 
     
     
         11 . The non-transitory computer-readable medium of  claim 9 , in which the program code to blend comprises program code to composite predictions from the first NeRF and the second NeRF to form blended images having an image quality greater than images individually rendered by any one of the first NeRF or the second NeRF. 
     
     
         12 . The non-transitory computer-readable medium of  claim 9 , in which a scene is represented by a plurality of NeRFs. 
     
     
         13 . The non-transitory computer-readable medium of  claim 9 , in which each of the plurality of NeRFs having a neighbor NeRF. 
     
     
         14 . The non-transitory computer-readable medium of  claim 9 , further comprising program code to train the first NeRF and the second NeRF on a separate set of images. 
     
     
         15 . The non-transitory computer-readable medium of  claim 14 , in which the separate set of images capture different, overlapping, portions of the same scene. 
     
     
         16 . The non-transitory computer-readable medium of  claim 9 , further comprising program code to plan an object grasp by a robot according to a fused image from the first NeRF and the second NeRF. 
     
     
         17 . A system for fusing neural radiance fields (NeRFs), the system comprising:
 re-render module to re-render a first NeRF and a second NeRF at different viewpoints to form synthesized images from the first NeRF and the second NeRF;   transform inference model to infer a transformation between a re-rendered first NeRF and a re-rendered second NeRF based on the synthesized images from the first NeRF and the second NeRF; and   NeRF blending module to blend the re-rendered first NeRF and the re-rendered second NeRF based on the inferred transformation to fuse the first NeRF and the second NeRF.   
     
     
         18 . The system of  claim 17 , in which the NeRF blending module is further to composite predictions from the first NeRF and the second NeRF to form blended images having an image quality greater than images individually rendered by any one of the first NeRF or the second NeRF. 
     
     
         19 . The system of  claim 17 , in which a scene is represented by a plurality of NeRFs and in which each of the plurality of NeRFs having a neighbor NeRF. 
     
     
         20 . The system of  claim 17 , further comprising an object manipulation module to plan an object grasp by a robot according to a fused image from the first NeRF and the second NeRF.