Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking

Abstract

Techniques can conduct online visual searches through an augmented reality (AR) device having an optical see-through head mounted display (HMD). An apparatus identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD. The portion includes searchable content, such as a barcode. The user interaction may be an eye gaze or a gesture. A user interaction point in relation to the HMD screen is tracked to locate a region of the object that includes the portion and the portion is detected within the region. The apparatus captures an image of the portion. The identified portion of the object does not encompass the entirety of the object. Accordingly, the size of the image is less than the size of the object in the field of view. The apparatus transmits the image to a visual search engine.

What is claimed is: 
     
         1 . An apparatus for processing one or more images, the apparatus comprising:
 at least one memory; and   at least one processor coupled to the at least one memory and configured to:
 receive a first input from a user of the apparatus, wherein the first input comprises an eye gaze of the user; 
 determine a portion of interest within a scene based on the first input; 
 in response to the first input, cause content associated with the portion of interest to be displayed on a display; 
 receive a second input from the user of the apparatus, wherein the second input comprises a gesture, and wherein the second input is associated with the displayed content; and 
 initiate an action based on the second input. 
   
     
     
         2 . The apparatus of  claim 1 , wherein the gesture includes a hand gesture. 
     
     
         3 . The apparatus of  claim 2 , wherein the hand gesture comprises a physical interaction with the apparatus. 
     
     
         4 . The apparatus of  claim 3 , wherein the physical interaction with the apparatus comprises an interaction with a physical button of the apparatus. 
     
     
         5 . The apparatus of  claim 2 , wherein the at least one processor is configured to:
 receive a third input from the user of the apparatus; and   in response to the third input, cause additional content to be displayed.   
     
     
         6 . The apparatus of  claim 5 , wherein the third input comprises an interaction with a physical interface of the apparatus. 
     
     
         7 . The apparatus of  claim 6 , wherein the physical interface is a physical button. 
     
     
         8 . The apparatus of  claim 6 , wherein, to cause the content associated with the portion of interest to be displayed, the at least one processor is configured to cause displayed content to be adjusted based on the first input. 
     
     
         9 . The apparatus of  claim 2 , wherein, to initiate the action, the at least one processor is configured to launch an application. 
     
     
         10 . The apparatus of  claim 2 , wherein the first input includes an interaction with a search icon. 
     
     
         11 . The apparatus of  claim 10 , wherein, based on the interaction with the search icon, a search is performed based on a boundary around a portion in the scene. 
     
     
         12 . The apparatus of  claim 2 , wherein, to receive the first input, the at least one processor is configured to track the eye gaze of the user to determine a location of the eye gaze relative to the display, wherein the portion of interest within the scene corresponds to the location of the eye gaze. 
     
     
         13 . The apparatus of  claim 1 , wherein, to receive the second input, the at least one processor is configured to track a gesture of the user to determine a location of the gesture. 
     
     
         14 . The apparatus of  claim 1 , wherein the content comprises a boundary. 
     
     
         15 . The apparatus of  claim 1 , wherein the action comprises a visual search process associated with the portion of interest. 
     
     
         16 . The apparatus of  claim 2 , wherein the at least one processor is configured to obtain, from an image capture apparatus, at least one image of the scene. 
     
     
         17 . The apparatus of  claim 16 , further comprising the image capture apparatus. 
     
     
         18 . The apparatus of  claim 16 , wherein the at least one processor is configured to process the at least one image of the scene to generate data associated with the portion of interest. 
     
     
         19 . The apparatus of  claim 18 , wherein the at least one processor is configured to cause the data to be transmitted to a remote device configured to generate visual search results. 
     
     
         20 . The apparatus of  claim 19 , wherein the at least one processor is configured to:
 receive the visual search results from the remote device; and   cause the visual search results to be displayed on the display.   
     
     
         21 . The apparatus of  claim 16 , wherein the apparatus is a head mounted device, and wherein the at least one image of the scene corresponds to a field of view observed by the user through a head mounted device. 
     
     
         22 . The apparatus of  claim 1 , wherein the apparatus is a head mounted device. 
     
     
         23 . The apparatus of  claim 1 , further comprising the display configured to display the content. 
     
     
         24 . An apparatus for processing one or more images, the apparatus comprising:
 at least one memory;   at least one eye tracking camera;   at least one scene camera;   at least one infrared (IR) camera;   at least one display;   at least one battery;   at least one transceiver; and   at least one processor coupled to the at least one memory and configured to:
 receive a first input from a user of the apparatus, wherein the first input comprises an eye gaze of the user; 
 determine a portion of interest within a scene based on the first input; 
 in response to the first input, cause content associated with the portion of interest to be displayed on the at least one display; 
 receive a second input from the user of the apparatus, wherein the second input comprises a gesture, and wherein the second input is associated with the displayed content; and 
 initiate an action based on the second input. 
   
     
     
         25 . The apparatus of  claim 24 , wherein the gesture includes a hand gesture. 
     
     
         26 . The apparatus of  claim 25 , wherein the hand gesture comprises a physical interaction with the apparatus. 
     
     
         27 . The apparatus of  claim 26 , wherein the physical interaction with the apparatus comprises an interaction with a physical button of the apparatus. 
     
     
         28 . The apparatus of  claim 25 , wherein the at least one processor is configured to:
 receive a third input from the user of the apparatus; and   in response to the third input, cause additional content to be displayed on the at least one display.   
     
     
         29 . The apparatus of  claim 28 , wherein the third input comprises an interaction with a physical interface of the apparatus. 
     
     
         30 . The apparatus of  claim 29 , wherein the physical interface is a physical button. 
     
     
         31 . The apparatus of  claim 29 , wherein, to cause the content associated with the portion of interest to be displayed, the at least one processor is configured to cause displayed content to be adjusted based on the first input. 
     
     
         32 . The apparatus of  claim 25 , wherein, to initiate the action, the at least one processor is configured to launch an application. 
     
     
         33 . The apparatus of  claim 25 , wherein the first input includes an interaction with a search icon. 
     
     
         34 . The apparatus of  claim 33 , wherein, based on the interaction with the search icon, a search is performed based on a boundary around a portion in the scene. 
     
     
         35 . The apparatus of  claim 25 , wherein, to receive the first input, the at least one processor is configured to track the eye gaze of the user to determine a location of the eye gaze relative to the at least one display, wherein the portion of interest within the scene corresponds to the location of the eye gaze. 
     
     
         36 . The apparatus of  claim 24 , wherein, to receive the second input, the at least one processor is configured to track a gesture of the user to determine a location of the gesture. 
     
     
         37 . The apparatus of  claim 24 , wherein the content comprises a boundary. 
     
     
         38 . The apparatus of  claim 24 , wherein the action comprises a visual search process associated with the portion of interest. 
     
     
         39 . The apparatus of  claim 25 , wherein the at least one processor is configured to obtain, from the at least one scene camera, at least one image of the scene. 
     
     
         40 . The apparatus of  claim 39 , wherein the at least one processor is configured to process the at least one image of the scene to generate data associated with the portion of interest. 
     
     
         41 . The apparatus of  claim 40 , wherein the at least one processor is configured to cause the data to be transmitted, via the at least one transceiver, to a remote device configured to generate visual search results. 
     
     
         42 . The apparatus of  claim 41 , wherein the at least one processor is configured to:
 receive the visual search results from the remote device; and   cause the visual search results to be displayed on the at least one display.   
     
     
         43 . The apparatus of  claim 39 , wherein the apparatus is a head mounted device, and wherein the at least one image of the scene corresponds to a field of view observed by the user through a head mounted device. 
     
     
         44 . The apparatus of  claim 24 , wherein the apparatus is a head mounted device. 
     
     
         45 . A method of processing one or more images, the method comprising:
 receiving a first input from a user of a device, wherein the first input comprises an eye gaze of the user;   determining a portion of interest within a scene based on the first input;   displaying, in response to the first input, content associated with the portion of interest on a display;   receiving a second input from the user of the device, wherein the second input comprises a gesture, and wherein the second input is associated with the displayed content; and   initiating an action based on the second input.   
     
     
         46 . The method of  claim 45 , wherein the gesture includes a hand gesture. 
     
     
         47 . The method of  claim 46 , wherein the hand gesture comprises a physical interaction with the device. 
     
     
         48 . The method of  claim 47 , wherein the physical interaction with the device comprises an interaction with a physical button of the device. 
     
     
         49 . The method of  claim 46 , further comprising:
 receiving a third input from the user of the device; and   displaying, in response to the third input, additional content.   
     
     
         50 . The method of  claim 49 , wherein the third input comprises an interaction with a physical interface of the device. 
     
     
         51 . The method of  claim 50 , wherein the physical interface is a physical button. 
     
     
         52 . The method of  claim 50 , wherein displaying the content associated with the portion of interest comprises adjusting displayed content based on the first input. 
     
     
         53 . The method of  claim 46 , wherein initiating the action comprises launching an application. 
     
     
         54 . The method of  claim 46 , wherein the first input includes an interaction with a search icon. 
     
     
         55 . The method of  claim 54 , wherein, based on the interaction with the search icon, a search is performed based on a boundary around a portion in the scene. 
     
     
         56 . The method of  claim 46 , wherein receiving the first input comprises tracking the eye gaze of the user to determine a location of the eye gaze relative to the display, wherein the portion of interest within the scene corresponds to the location of the eye gaze. 
     
     
         57 . The method of  claim 45 , wherein receiving the second input comprises tracking a gesture of the user to determine a location of the gesture. 
     
     
         58 . The method of  claim 45 , wherein the content comprises a boundary. 
     
     
         59 . The method of  claim 45 , wherein the action comprises a visual search process associated with the portion of interest. 
     
     
         60 . The method of  claim 46 , further comprising obtaining, from an image capture device, at least one image of the scene. 
     
     
         61 . The method of  claim 60 , further comprising processing the at least one image of the scene to generate data associated with the portion of interest. 
     
     
         62 . The method of  claim 61 , further comprising transmitting the data to a remote device configured to generate visual search results. 
     
     
         63 . The method of  claim 62 , further comprising:
 receiving the visual search results from the remote device; and   displaying the visual search results on the display.   
     
     
         64 . The method of  claim 60 , wherein the device is a head mounted device, and wherein the at least one image of the scene corresponds to a field of view observed by the user through a head mounted device.