Augmented, virtual and mixed-reality content selection & display

Abstract

Systems, methods and techniques for automatically recognizing two or 3-dimensional real world objects with an augmented reality display device (smartphone or glasses etc.), and augmenting or enhancing the display of such real world objects by superimposing virtual images such as a still or video advertisement, an opportunity to buy, a story or other virtual image presentation. In non-limiting embodiments, the real world object includes visible features including visible security features and a recognition process takes the visible security features into account when recognizing the object and/or displaying superimposed virtual images.

1 . A portable display device for automatically recognizing two or three dimensional real world objects and augmenting or enhancing display of such real world objects with an additional presentations superimposed thereon, the portable display device comprising:
 a portable housing having a surface,   a touch screen display disposed on the housing surface,   a push button disposed on the housing,   a camera disposed within the housing,   a memory disposed within the housing,   a wireless communications device disposed within the housing, and   a processor disposed within the housing and operatively coupled to the touch screen display, the memory, the camera, the push button and the wireless communications device, the processor configured to perform, in response to instructions stored in the memory, operations comprising:   (a) display, on the touch screen display, an image of a real world object acquired by the camera as the camera is aimed at the real world object;   (b) recognize at least a portion of the acquired image without requiring a bar code or AR marker or other special marking that is not intended to be recognized by humans but instead is designed or intended to be automatically recognized electronically by a machine; and   (c) in response to recognizing at least a portion of the acquired image, anchoring a presentation to the displayed image of the real world object currently being acquired by the camera such that if the housing is moved to change the position and/or orientation of the housing relative to the real world object, the presentation moves with apparent movement of the image of the real world object on the touch screen display,   wherein the presentation provides offering details concerning the recognized acquired image portion.   
     
     
         2 . The portable display device of  claim 1  wherein the processor is further configured to change position, orientation and/or perspective of the anchored presentation as the underlying real world object changes position, orientation and/or viewing angle relative to the housing, in order to provide a photorealistic image in which the anchored presentation appears to be part of the underlying real world object. 
     
     
         3 . The portable display device of  claim 1  further including at least one sensor disposed in the housing, the sensor configured to sense posture, orientation and/or position of the device, wherein the at least one sensor comprises an inertial sensor such as an accelerometer and/or a gyrosensor, and wherein the processor is configured to display, on the touch screen display, different portions of the presentation based on such sensed posture, orientation and/or position. 
     
     
         4 . The portable display device of  claim 1  further comprising an optical flow sensor disposed within the housing, the optical flow sensor providing an optical flow image analysis, and wherein the processor is configured to display, on the touch screen display, different portions of the presentation based on such optical flow image analysis. 
     
     
         5 . The portable display device of  claim 1  wherein the presentation comprises a mixed reality (MR) image that mixes virtual and real scenes. 
     
     
         6 . The portable display device of  claim 1  wherein the presentation comprises an augmented reality (AR) image that augments real scenes with virtual objects. 
     
     
         7 . The portable display device of  claim 1  wherein the presentation is animated. 
     
     
         8 . The portable display device of  claim 1  wherein the presentation is displayed with controls such as play/stop, fast-forward, rewind and mute that can be pressed on the touch screen display to control the presentation. 
     
     
         9 . The portable display device of  claim 1  wherein the presentation comprises a superimposed control bar displayed on the touch screen display that controls playing of the presentation. 
     
     
         10 . The portable display device of  claim 1  wherein the presentation comprises a superimposed virtual shopping cart to provide a virtual or mixed reality composite image and the processor is further configured to wireless transmit an acceptance of the offering details in response to touch screen manipulation of the superimposed virtual shopping cart. 
     
     
         11 . The portable display device of  claim 1  wherein the presentation is anchored as if it were glued or otherwise adhered or attached to the real world object. 
     
     
         12 . A portable display device for automatically recognizing two or three dimensional real world objects and augmenting or enhancing views of such real world objects with an additional presentation superimposed thereon, the portable display device comprising:
 a display,   a push button,   a camera,   a memory,   a wireless communications device, and   a processor operatively coupled to the display, the memory, the camera, the push button and the wireless communications device, the processor configured to perform, in response to instructions stored in the memory, operations comprising:   (a) acquire an image with the camera;   (b) recognize at least a portion of the acquired image without requiring a bar code or AR marker or other special machine-readable marking; and   (c) in response to recognizing at least a portion of the acquired image, anchoring a display of a presentation to a view of the real world object such that if the display device is moved to change the position and/or orientation of the real world object relative to the display device, the presentation moves on the display with movement of the view of the real world object,   wherein the presentation provides offering details concerning the recognized acquired image portion.   
     
     
         13 . The portable display device of  claim 1  wherein the processor is further configured to change position, orientation and/or perspective of the anchored presentation as the underlying real world object changes position, orientation and/or viewing angle relative to the display device, in order to provide a view in which the anchored presentation appears to be part of the underlying real world object. 
     
     
         14 . The portable display device of  claim 1  further including at least one sensor configured to sense posture, orientation and/or position of the device, wherein the at least one sensor comprises an inertial sensor such as an accelerometer and/or a gyrosensor, and wherein the processor is configured to display, on the display, different portions of the presentation based on such sensed posture, orientation and/or position. 
     
     
         15 . The portable display device of  claim 1  further comprising an optical flow sensor providing an optical flow image analysis, and wherein the processor is configured to display, on the display, different portions of the presentation based on such optical flow image analysis. 
     
     
         16 . The portable display device of  claim 1  wherein the presentation comprises a mixed reality (MR) image that mixes virtual and real scenes. 
     
     
         17 . The portable display device of  claim 1  wherein the presentation comprises an augmented reality (AR) image that augments real scenes with virtual objects. 
     
     
         18 . The portable display device of  claim 1  wherein the presentation is animated. 
     
     
         19 . The portable display device of  claim 1  wherein the presentation is displayed with controls such as play/stop, fast-forward, rewind and mute that can be actuated to control the presentation. 
     
     
         20 . The portable display device of  claim 1  wherein the presentation comprises a superimposed control bar that controls playing of audio.