Representing Real-World Objects with a Virtual Reality Environment

Abstract

An image processing system enables a user wearing a head-mounted display to experience a virtual environment combined with a representation of a real-world object. The image processing system receives a captured scene of a real-world environment that includes a target object. The image processing system identifies the target object in the captured scene and generates a representation of the target object. In some cases, the image processing system may include a graphical overlay with the representation of the target object. The image processing system can generate a combined scene that includes the target object and the virtual environment. The combined scene is presented to the user, thereby allowing the user to interact with the real-world target object (or a representation thereof) in combination with the virtual environment.

What is claimed is: 
     
         1 . A computer-implemented method, comprising:
 receiving, by a virtual reality system, image data for an image frame of a real-world environment;   identifying, by the virtual reality system, a first representation of a target object in the received image data;   generating, by the virtual reality system, a scene from a perspective of a user wearing a head-mounted display of the virtual reality system;   applying a skin, to the first representation of a target object, that causes a visual effect which changes an appearance of the first representation of a target object into a second representation of the target object;   overlaying, by the virtual reality system, the second representation of the target object onto the scene, the second representation of the target object located to appear in a position corresponding to a position of the first representation of the target object in the received image data; and   presenting, by the virtual reality system, a combined scene, including both the scene and the overlaid second representation of the target object.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the applying the skin causes the second representation of the target object to match a determined theme of the scene. 
     
     
         3 . The computer-implemented method of  claim 1 ,
 wherein the target object was identified using an object recognition system, and   wherein the first representation of the target object is identified by determining edges of the target object, as identified by the object recognition system, in the image data.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein the first representation of the target object is identified based on the target object being associated with at least one of a marker, a tag, or a particular coloring. 
     
     
         5 . The computer-implemented method of  claim 4 ,
 wherein the target object is associated with a marker that emits a wireless signal, and   wherein the first representation of the target object is identified based on the wireless signal.   
     
     
         6 . The computer-implemented method of  claim 1 ,
 wherein the image data is received from an imaging sensor of the virtual reality system, and   wherein the combined scene is presented to the user via the head-mounted display.   
     
     
         7 . The computer-implemented method of  claim 1 , wherein presenting the combined scene, by the virtual reality system, comprises the virtual reality system rendering the second representation of the target object at a set of coordinates on the head-mounted display and virtual content of the scene at all other locations on a displayable area of the head-mounted display. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the virtual reality system includes preprogrammed logic, a predefined setting, and/or a default configuration defining the target object for object detection or recognition. 
     
     
         9 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to:
 receive, by a virtual reality system, image data for an image frame of a real-world environment;   identify, by the virtual reality system, a first representation of a target object in the received image data;   generate, by the virtual reality system, a scene from a perspective of a user wearing a head-mounted display of the virtual reality system;   apply a skin, to the first representation of a target object, that causes a visual effect which changes an appearance of the first representation of a target object into a second representation of the target object;   overlay, by the virtual reality system, the second representation of the target object onto the scene, the second representation of the target object located to appear in a position corresponding to a position of the first representation of the target object in the received image data; and   present, by the virtual reality system, a combined scene, including both the scene and the overlaid second representation of the target object.   
     
     
         10 . The computer-readable storage medium of  claim 9 , wherein the applying the skin causes the second representation of the target object to match a determined theme of the scene. 
     
     
         11 . The computer-readable storage medium of  claim 9 ,
 wherein the target object was identified using an object recognition system, and   wherein the first representation of the target object is identified by determining edges of the target object, as identified by the object recognition system, in the image data.   
     
     
         12 . The computer-readable storage medium of  claim 9 , wherein the first representation of the target object is identified based on the target object being associated with at least one of a marker, a tag, or a particular coloring. 
     
     
         13 . The computer-readable storage medium of  claim 12 ,
 wherein the target object is associated with a marker that emits a wireless signal, and   wherein the first representation of the target object is identified based on the wireless signal.   
     
     
         14 . The computer-readable storage medium of  claim 9 ,
 wherein the image data is received from an imaging sensor of the virtual reality system, and   wherein the combined scene is presented to the user via the head-mounted display.   
     
     
         15 . The computer-readable storage medium of  claim 9 , wherein presenting the combined scene, by the virtual reality system, comprises the virtual reality system rendering the second representation of the target object at a set of coordinates on the head-mounted display and virtual content of the scene at all other locations on a displayable area of the head-mounted display. 
     
     
         16 . The computer-readable storage medium of  claim 9 , wherein the virtual reality system includes preprogrammed logic, a predefined setting, and/or a default configuration defining the target object for object detection or recognition. 
     
     
         17 . A computing system comprising:
 one or more processors; and   one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to:
 receive, by a virtual reality system, image data for an image frame of a real-world environment; 
 identify, by the virtual reality system, a first representation of a target object in the received image data; 
 generate, by the virtual reality system, a scene from a perspective of a user wearing a head-mounted display of the virtual reality system; 
 apply a skin, to the first representation of a target object, that causes a visual effect which changes an appearance of the first representation of a target object into a second representation of the target object; 
 overlay, by the virtual reality system, the second representation of the target object onto the scene, the second representation of the target object located to appear in a position corresponding to a position of the first representation of the target object in the received image data; and 
 present, by the virtual reality system, a combined scene, including both the scene and the overlaid second representation of the target object. 
   
     
     
         18 . The computing system of  claim 17 , wherein the applying the skin causes the second representation of the target object to match a determined theme of the scene. 
     
     
         19 . The computing system of  claim 17 , wherein the first representation of the target object is identified based on the target object being associated with at least one of a marker, a tag, or a particular coloring. 
     
     
         20 . The computing system of  claim 17 , wherein the virtual reality system includes preprogrammed logic, a predefined setting, and/or a default configuration defining the target object for object detection or recognition.