Systems and methods for selective replacement of objects in images

Abstract

Exemplary embodiments are directed to a system for selective replacement of an object in an image. The system includes an interface configured to receive as input an original image and a background image, and a processing device in communication with the interface. The processing device is configured to process the original image using a neural network to detect one or more objects in the original image, generate a neural network mask of the original image for the one or more objects in the original image, generate a filtered original image including the original image without the one or more objects, generate a modulated background image including a replacement background based on the neural network mask, and generate a combined image including the filtered original image combined with the modulated background image.

1 . A system for selective replacement of an object in an image, the system comprising:
 an interface configured to receive as input an original image and a background image; and   a processing device in communication with the interface, the processing device configured to:
 (i) process the original image using a neural network to detect one or more objects in the original image; 
 (ii) generate a neural network mask of the original image for the one or more objects in the original image; 
 (iii) generate a filtered original image including the original image without the one or more objects; 
 (iv) generate a modulated background image including a replacement background based on the neural network mask; and 
 (v) generate a combined image including the filtered original image combined with the modulated background image. 
   
     
     
         2 . The system of  claim 1 , wherein the original image includes a foreground and a background, wherein the one or more objects include the background of the original input. 
     
     
         3 . The system of  claim 2 , wherein the one or more objects include a sky in the original image. 
     
     
         4 . The system of  claim 2 , wherein the processing device extracts the foreground from the original image. 
     
     
         5 . The system of  claim 2 , wherein the processing device generates a refined mask for each pixel of the original image associated with the background. 
     
     
         6 . The system of  claim 5 , wherein the processing device generates a dilated mask including a dilation or indentation from a border extending between the foreground and the background. 
     
     
         7 . The system of  claim 6 , wherein the processing device generates an interpolation grid corresponding to the foreground. 
     
     
         8 . The system of  claim 7 , wherein the processing device generates an extracted image including the original image with the foreground extracted based on the interpolation grid. 
     
     
         9 . The system of  claim 7 , wherein processing device generates the filtered original image by extracting the foreground of the original image based on the interpolation grid. 
     
     
         10 . The system of  claim 1 , wherein the processing device generates a blended image, the blended image including a smooth transition between the filtered original image and the modulated background image. 
     
     
         11 . The system of  claim 1 , wherein the processing device generates a toned image, the toned image including the combined image with adjustment of tone within the combined image. 
     
     
         12 . The system of  claim 1 , wherein the processing device generates a tint unified image, the tint unified image including tint correction at edges between the filtered original image and the modulated background image. 
     
     
         13 . The system of  claim 1 , wherein the processing device adjusts one or more characteristics of the filtered original image independently from one or more characteristics of the modulated background image. 
     
     
         14 . The system of  claim 1 , wherein the interface includes an image selection section with the combined image and one or more additional original images. 
     
     
         15 . The system of  claim 14 , wherein the interface includes a first submenu for selecting the combined image and copying the adjustments or enhancements applied to the combined image, and the interface includes a second submenu for selecting one or more of the additional original images and applying the copied adjustments or enhancements of the combined image to the selected one or more of the additional original images. 
     
     
         16 . A method for selective replacement of an object in an image, the method comprising:
 receiving as input at an interface an original image and a background image;   detecting one or more objects in the original image with a neural network;   generating a neural network mask of the original image for the one or more objects in the original image;   generating a filtered original image, the filtered original image including the original image without the one or more objects;   generating a modulated background image, the modulated background image including a replacement background based on the neural network mask; and   generating a combined image, the combined image including the filtered original image combined with the modulated background image.   
     
     
         17 . The method of  claim 16 , comprising adjusting one or more characteristics of the filtered original image independently from one or more characteristics of the modulated background image. 
     
     
         18 . The method of  claim 16 , comprising receiving at the interface one or more additional original images, wherein the interface includes an image selection section with the combined image and the one or more additional original images. 
     
     
         19 . The method of  claim 18 , comprising selecting the combined image and copying the adjustments or enhancements applied to the combined image at a first submenu of the interface, and selecting one or more of the additional images and applying the copied adjustments or enhancements of the combined image to the selected one or more of the additional images at a second submenu of the interface. 
     
     
         20 . A non-transitory computer-readable medium storing instructions at least for selective replacement of an object in an image that are executable by a processing device, wherein execution of the instructions by the processing device causes the processing device to:
 receive as input at an interface an original image and a background image;   detect one or more objects in the original image with a neural network;   generate a neural network mask of the original image for the one or more objects in the original image;   generate a filtered original image, the filtered original image including the original image without the one or more objects;   generate a modulated background image, the modulated background image including a replacement background based on the neural network mask; and   generate a combined image, the combined image including the filtered original image combined with the modulated background image.