Detection and obfuscation of display screens in augmented reality content

Abstract

The subject technology receives first image data captured by a camera of an eyewear device. The subject technology detects, using a machine learning model, a representation of a display screen in the first image data. The subject technology selects at least a portion of the representation of the display screen. The subject technology adjusts a visual appearance of the portion of the representation of the display screen. The subject technology causes display of the adjusted visual appearance using a display system of the eyewear device.

What is claimed is: 
     
         1 . A method, comprising:
 receiving first image data captured by a camera of an eyewear device;   detecting, using a machine learning model, a representation of a display screen of an electronic device in the first image data;   selecting at least a portion of the representation of the display screen of the electronic device;   adjusting, while the representation of the display screen of the electronic device is detected in a field of view of a user while using the eyewear device, a visual appearance of the portion of the representation of the display screen, wherein adjusting, while the representation of the display screen of the electronic device is detected in the field of view of the user while using the eyewear device, the visual appearance of the portion of the representation of the display screen comprises at least:
 generating a representation of a bounding box around the representation of the display screen, the representation of the bounding box including a set of pixels corresponding to at least four sides, and 
 modifying a second set of pixels corresponding to the representation of the display screen by at least changing a first luminosity value of a third pixel to a second luminosity value, wherein the second luminosity value is greater than the first luminosity value; and 
   causing display of the adjusted visual appearance using a display system of the eyewear device.   
     
     
         2 . The method of  claim 1 , wherein detecting, using the machine learning model, the representation of the display screen comprises:
 performing an object detection process on the first image data to determine the representation of the display screen, wherein the machine learning model determines a prediction of the representation of the display screen being included in the first image data.   
     
     
         3 . The method of  claim 2 , wherein the machine learning model comprises a convolutional neural network (CNN). 
     
     
         4 . The method of  claim 3 , wherein the CNN determines a region of interest where the representation of the display screen is present, the region of interest comprising a portion of the first image data. 
     
     
         5 . The method of  claim 4 , wherein the region of interest comprises a candidate bounding box including the representation of the display screen. 
     
     
         6 . The method of  claim 4 , wherein the CNN generates a feature map based on the region of interest and provides a vector of values corresponding to the feature map as an output, the vector including respective values describing contents of the region of interest. 
     
     
         7 . The method of  claim 6 , wherein the feature map includes a set of features, and a classifier model generates a classification of the set of features from the feature map, the classification comprising a display screen object. 
     
     
         8 . The method of  claim 1 , wherein adjusting the visual appearance of the portion of the representation of the display screen further comprises:
 modifying the second set of pixels corresponding to the representation of the display screen by at least changing a particular luminosity value of a fourth pixel to a second particular luminosity value, wherein the second particular luminosity value is less than the particular luminosity value.   
     
     
         9 . The method of  claim 1 , wherein adjusting the visual appearance of the portion of the representation of the display screen further comprises:
 modifying the second set of pixels corresponding to the representation of the display screen by at least changing a color value of a fourth pixel to a second color value.   
     
     
         10 . The method of  claim 1 , further comprising:
 receiving second image data captured by the camera, the second image data being received upon movement of the camera of the eyewear device based on a change of a position of a head of a user wearing the eyewear device;   detecting that the representation of the display screen is no longer present in the second image data; and   modifying the second image data to indicate that the representation of the display screen is no longer present, the modifying comprising reducing a portion of the second image data to decrease at least one luminosity value of a pixel from the second image data.   
     
     
         11 . A system comprising:
 a processor; and   a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
 receiving first image data captured by a camera of an eyewear device; 
   detecting, using a machine learning model, a representation of a display screen of an electronic device in the first image data;   selecting at least a portion of the representation of the display screen of the electronic device;   adjusting, while the representation of the display screen of the electronic device is detected in a field of view of a user while using the eyewear device, a visual appearance of the portion of the representation of the display screen, wherein adjusting, while the representation of the display screen of the electronic device is detected in the field of view of the user while using the eyewear device, the visual appearance of the portion of the representation of the display screen comprises at least:
 generating a representation of a bounding box around the representation of the display screen, the representation of the bounding box including a set of pixels corresponding to at least four sides, and 
 modifying a second set of pixels corresponding to the representation of the display screen by at least changing a first luminosity value of a third pixel to a second luminosity value, wherein the second luminosity value is greater than the first luminosity value; and 
   causing display of the adjusted visual appearance using a display system of the eyewear.   
     
     
         12 . The system of  claim 11 , wherein detecting, using the machine learning model, the representation of the display screen comprises:
 performing an object detection process on the first image data to determine the representation of the display screen, wherein the machine learning model determines a prediction of the representation of the display screen being included in the first image data.   
     
     
         13 . The system of  claim 12 , wherein the machine learning model comprises a convolutional neural network (CNN). 
     
     
         14 . The system of  claim 13 , wherein the CNN determines a region of interest where the representation of the display screen is present, the region of interest comprising a portion of the first image data. 
     
     
         15 . The system of  claim 14 , wherein the region of interest comprises a candidate bounding box including the representation of the display screen. 
     
     
         16 . The system of  claim 14 , wherein the CNN generates a feature map based on the region of interest and provides a vector of values corresponding to the feature map as an output, the vector including respective values describing contents of the region of interest. 
     
     
         17 . The system of  claim 16 , wherein the feature map includes a set of features, and a classifier model generates a classification of the set of features from the feature map, the classification comprising a display screen object. 
     
     
         18 . The system of  claim 11 , wherein adjusting the visual appearance of the portion of the representation of the display screen further comprises:
 modifying the second set of pixels corresponding to the representation of the display screen by at least changing a particular luminosity value of a fourth pixel to a second particular luminosity value, wherein the second particular luminosity value is less than the particular luminosity value.   
     
     
         19 . The system of  claim 11 , wherein adjusting the visual appearance of the portion of the representation of the display screen further comprises:
 modifying the second set of pixels corresponding to the representation of the display screen by at least changing a color value of a fourth pixel to a second color value.   
     
     
         20 . A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
 receiving first image data captured by a camera of an eyewear device;   detecting, using a machine learning model, a representation of a display screen of an electronic device in the first image data;   selecting at least a portion of the representation of the display screen of the electronic device;   adjusting, while the representation of the display screen of the electronic device is detected in a field of view of a user while using the eyewear device, a visual appearance of the portion of the representation of the display screen, wherein adjusting, while the representation of the display screen of the electronic device is detected in the field of view of the user while using the eyewear device, the visual appearance of the portion of the representation of the display screen comprises at least:
 generating a representation of a bounding box around the representation of the display screen, the representation of the bounding box including a set of pixels corresponding to at least four sides, and 
 modifying a second set of pixels corresponding to the representation of the display screen by at least changing a first luminosity value of a third pixel to a second luminosity value, wherein the second luminosity value is greater than the first luminosity value; and 
   causing display of the adjusted visual appearance using a display system of the eyewear.