Digital Media Environment for Style-Aware Patching in a Digital Image

Abstract

Techniques and systems are described for style-aware patching of a digital image in a digital medium environment. For example, a digital image creation system generates style data for a portion to be filled of a digital image, indicating a style of an area surrounding the portion. The digital image creation system also generates content data for the portion indicating content of the digital image of the area surrounding the portion. The digital image creation system selects a source digital image based on similarity of both style and content of the source digital image at a location of the patch to the style data and content data. The digital image creation system transforms the style of the source digital image based on the style data and generates the patch from the source digital image in the transformed style for incorporation into the portion to be filled of the digital image.

What is claimed is: 
     
         1 . In a digital medium environment for style-aware patching in a digital image, a method implemented by at least one computing device, the method comprising:
 generating, for a portion to be filled in a digital image, style data for the portion indicating a style of an area surrounding the portion to be filled and content data for the portion indicating content of the digital image of the area surrounding the portion to be filled;   selecting a source digital image, from multiple digital images, based on a both a similarity of style and a similarity of content between the source digital image and the digital image;   transforming a style of the selected source digital image based on the style data of the digital image; and   generating a patch from the selected source digital image in the transformed style for incorporation into the portion to be filled of the digital image.   
     
     
         2 . The method of  claim 1 , wherein the style data is generated using a style classifier trained on numerous digital images to determine one or more aesthetics or feelings to include in the style data. 
     
     
         3 . The method of  claim 2 , wherein the style classifier is a convolutional neural network trained to perform low-dimensional feature embedding for visual style. 
     
     
         4 . The method of  claim 1 , further comprising cropping the patch from the source digital image in the transformed style to match a size of a cell of a grid in the portion to be filled of the digital image. 
     
     
         5 . The method of  claim 4 , further comprising copying the cropped patch to the cell in a location of the grid within the portion to be filled of the digital image. 
     
     
         6 . The method of  claim 1 , wherein the selecting the source digital image further includes performing an image search based on both the style data and the content data to select a subset of digital images that are related to the digital image. 
     
     
         7 . The method of  claim 6 , wherein the selecting the source digital image further comprises sampling candidate patches in the portion to be filled of the digital image to produce a collection of candidate patches based on the similarity of style and the similarity of content, the collection of patches further reducing a number of source digital images in the subset of source digital images. 
     
     
         8 . In a digital medium environment for style-aware patching in a digital image in a digital image creation system, a system comprising:
 at least a memory and a processor to perform operations comprising:
 generating, for a portion to be filled in a digital image, style data for the portion indicating a style of an area surrounding the portion to be filled and content data for the portion indicating content of the digital image of the area surrounding the portion to be filled; 
 selecting a source digital image, from multiple digital images, based on a both a similarity of style and a similarity of content between the source digital image and the digital image; 
 transforming a style of the selected source digital image based on the style data of the digital image; and 
 generating a patch from the selected source digital image in the transformed style for incorporation into the portion to be filled of the digital image. 
   
     
     
         9 . The system of  claim 8 , wherein the style data is generated using a style classifier trained on numerous digital images to determine one or more aesthetics or feelings to include in the style data. 
     
     
         10 . The system of  claim 9 , wherein the style classifier is a convolutional neural network trained to perform low-dimensional feature embedding for visual style. 
     
     
         11 . The system of  claim 8 , wherein the operations further comprise cropping the patch from the source digital image in the transformed style to match a size of a cell of a grid in the portion to be filled of the digital image. 
     
     
         12 . The system of  claim 11 , wherein the operations further comprise copying the cropped patch to the cell in a location of the grid within the portion to be filled of the digital image. 
     
     
         13 . The system of  claim 8 , wherein the selecting the source digital image further includes performing an image search based on both the style data and the content data to select a subset of digital images that are related to the digital image. 
     
     
         14 . The system of  claim 13 , wherein the selecting the source digital image further comprises sampling candidate patches in the portion to be filled of the digital image to produce a collection of candidate patches based on the similarity of style and the similarity of content, the collection of patches further reducing a number of source digital images in the subset of source digital images. 
     
     
         15 . A computer-readable storage device having instructions stored thereon that, responsive to execution by one or more processors, perform operations comprising:
 generating, for a portion to be filled in a digital image, style data for the portion indicating a style of an area surrounding the portion to be filled and content data for the portion indicating content of the digital image of the area surrounding the portion to be filled;   selecting a source digital image, from multiple digital images, based on a both a similarity of style and a similarity of content between the source digital image and the digital image;   transforming a style of the selected source digital image based on the style data of the digital image; and   generating a patch from the selected source digital image in the transformed style for incorporation into the portion to be filled of the digital image.   
     
     
         16 . The computer-readable storage device of  claim 15 , wherein the style data is generated using a style classifier trained on numerous digital images to determine one or more aesthetics or feelings to include in the style data. 
     
     
         17 . The computer-readable storage device of  claim 16 , wherein the style classifier is a convolutional neural network trained to perform low-dimensional feature embedding for visual style. 
     
     
         18 . The computer-readable storage device of  claim 15 , wherein the operations further comprise:
 cropping the patch from the source digital image in the transformed style to match a size of a cell of a grid in the portion to be filled of the digital image; and   copying the cropped patch to the cell in a location of the grid within the portion to be filled of the digital image.   
     
     
         19 . The computer-readable storage device of  claim 15 , wherein the selecting the source digital image further includes performing an image search based on both the style data and the content data to select a subset of digital images that are related to the digital image. 
     
     
         20 . The computer-readable storage device of  claim 19 , wherein the selecting the source digital image further comprises sampling candidate patches in the portion to be filled of the digital image to produce a collection of candidate patches based on the similarity of style and the similarity of content, the collection of patches further reducing a number of source digital images in the subset of source digital images.