Augmented reality user interaction system

Abstract

An augmented reality user interaction system includes a wearable computer equipped with at least one camera to detect one or more fiducial markers worn by a user. A user-mounted visual display worn by the user is employed to display visual 3D information. The computer detects in an image a fiducial marker worn by the user, extracts a position and orientation of the fiducial marker in the image, and superimposes on the image a visual representation of a user interface component directly on or near the user based on the position and orientation.

1 . An augmented reality user interaction system, comprising:
 a wearable computer equipped with at least one camera to detect one or more fiducial markers worn by a user; and   a user-mounted visual display worn by the user and employed to display visual 3D information,   wherein said computer operably detects in an image including at least part of the user one or more fiducial markers worn by the user, extracts a position and orientation of a fiducial marker in the image, and superimposes on the image a visual representation of a user interface component directly on or near the user based on the position and orientation.   
   
   
       2 . The system of  claim 1 , wherein said computer operably displays visual 3D information to the user including the image having the visual representation of the user interface component. 
   
   
       3 . The system of  claim 2 , wherein said computer operably constructs a frame associated with the fiducial marker having a quadrilateral edge containing a valid square image, and operably renders graphics into the frame, including the visual representation of the user interface component. 
   
   
       4 . The system of  claim 3 , wherein said computer operably searches the image for edges, determines whether there are any candidate edges, and traces a candidate edge. 
   
   
       5 . The system of  claim 4 , wherein said computer operably determines whether there are any candidate edges by scanning across rows of the image and looking for local transitions from a background color to a predefined edge color. 
   
   
       6 . The system of  claim 3 , wherein said computer operably determines whether a traced edge approximates a quadrilateral, and computes corners of an edge that has been determined to approximate a quadrilateral. 
   
   
       7 . The system of  claim 6 , wherein said computer operably determines whether a traced edge approximates a quadrilateral by determining if a quadrilateral can be overlaid on the edge such that the edge does not deviate from the quadrilateral by more than a maximum distance determined by a noise tolerance of a capture process employed to capture the image. 
   
   
       8 . The system of  claim 6 , wherein said computer operably computes corners of an edge that has been determined to approximate a quadrilateral by employing all pixels along all four edges of the quadrilateral to determine an optimal line fitting in a least-squared sense by computing four such lines and interpreting intersections of the four lines as the corners of the quadrilateral. 
   
   
       9 . The system of  claim 3 , wherein said computer operably processes an interior of the quadrilateral edge into a square image, and determining whether the square image is valid. 
   
   
       10 . The system of  claim 9 , wherein said computer operably processes an interior of the quadrilateral edge into a square image by warping a portion of the image enclosed by the quadrilateral edge into the square image by determining a mapping of pixels within the quadrilateral edge to corresponding locations in the square image. 
   
   
       11 . The system of  claim 9 , wherein said computer operably determines whether the square image is valid by determining whether the square image matches one of plural predefined images selected to have low correlations to random image data. 
   
   
       12 . The system of  claim 1 , further comprising detecting user interaction with the visual representation of the user interface component based on detected position and orientation of another fiducial marker worn by the user. 
   
   
       13 . The system of  claim 12 , wherein said computer operably superimposes a visual representation of a cursor at or near the other fiducial marker; and detects user interaction with the visual representation of the user interface component when the user manipulates the other fiducial marker to cause the visual representation of the cursor to appear to the user to interact with the visual representation of the user interface component in a predetermined fashion. 
   
   
       14 . The system of  claim 12 , wherein said computer operably detects the fiducial marker on or near one hand of the user, and detects the other fiducial marker on a ring worn on another hand of the user. 
   
   
       15 . The system of  claim 1 , wherein said computer operably detects user interaction with the visual representation of the user interface component based on user occlusion of the fiducial marker with respect to which the visual representation of the user interface component is visually rendered. 
   
   
       16 . The system of  claim 1 , wherein said computer operably triggers a predetermined interaction when the user interacts with the visual representation of the user interface component. 
   
   
       17 . The system of  claim 1 , wherein said computer operably extracts and recognizes image content located inside or near the fiducial marker, and determines which of plural user interface components to display in relation to the fiducial marker based on the image content. 
   
   
       18 . The system of  claim 1 , wherein said computer operably detects the fiducial marker attached to skin of at least one of a hand, wrist, or arm of the user. 
   
   
       19 . The system of  claim 1 , wherein said computer operably detects the fiducial marker on clothing worn by the user. 
   
   
       20 . The system of  claim 1 , wherein said computer operably detects the fiducial marker on at least one of a watch or jewelry worn by the user. 
   
   
       21 . An augmented reality user interaction method, comprising:
 visually detecting in an image, including at least part of the user, one or more fiducial markers worn by the user;   extracting a position and orientation of a fiducial marker in the image; and   superimposing on the image a visual representation of a user interface component directly on or near the user based on the position and orientation.   
   
   
       22 . The method of  claim 21 , further comprising displaying visual 3D information to the user including the image having the visual representation of the user interface component. 
   
   
       23 . The method of  claim 22 , further comprising:
 constructing a frame associated with the fiducial marker having a quadrilateral edge containing a valid square image; and   rendering graphics into the frame, including the visual representation of the user interface component.   
   
   
       24 . The method of  claim 23 , further comprising:
 searching the image for edges;   determining whether there are any candidate edges; and   tracing a candidate edge.   
   
   
       25 . The method of  claim 24 , wherein determining whether there are any candidate edges includes scanning across rows of the image and looking for local transitions from a background color to a predefined edge color. 
   
   
       26 . The method of  claim 23 , further comprising:
 determining whether a traced edge approximates a quadrilateral; and   computing corners of an edge that has been determined to approximate a quadrilateral.   
   
   
       27 . The method of  claim 26 , wherein determining whether a traced edge approximates a quadrilateral includes determining if a quadrilateral can be overlaid on the edge such that the edge does not deviate from the quadrilateral by more than a maximum distance determined by a noise tolerance of a capture process employed to capture the image. 
   
   
       28 . The method of  claim 26 , wherein computing corners of an edge that has been determined to approximate a quadrilateral includes employing all pixels along all four edges of the quadrilateral to determine an optimal line fitting in a least-squared sense by computing four such lines and interpreting intersections of the four lines as the corners of the quadrilateral. 
   
   
       29 . The method of  claim 23 , further comprising:
 processing an interior of the quadrilateral edge into a square image; and   determining whether the square image is valid.   
   
   
       30 . The method of  claim 29 , wherein processing an interior of the quadrilateral edge into the square image includes warping a portion of the image enclosed by the quadrilateral edge into the square image by determining a mapping of pixels within the quadrilateral edge to corresponding locations in the square image. 
   
   
       31 . The method of  claim 29 , wherein determining whether the square image is valid includes determining whether the square image matches one of plural predefined images selected to have low correlations to random image data. 
   
   
       32 . The method of  claim 21 , further comprising detecting user interaction with the visual representation of the user interface component based on detected position and orientation of another fiducial marker worn by the user. 
   
   
       33 . The method of  claim 32 , further comprising:
 superimposing a visual representation of a cursor at or near the other fiducial marker; and   detecting user interaction with the visual representation of the user interface component when the user manipulates the other fiducial marker to cause the visual representation of the cursor to appear to the user to interact with the visual representation of the user interface component in a predetermined fashion.   
   
   
       34 . The method of  claim 32 , further comprising:
 detecting the fiducial marker on or near one hand of the user; and   detecting the other fiducial marker on a ring worn on another hand of the user.   
   
   
       35 . The method of  claim 21 , further comprising detecting user interaction with the visual representation of the user interface component based on user occlusion of the fiducial marker with respect to which the visual representation of the user interface component is visually rendered. 
   
   
       36 . The method of  claim 21 , further comprising triggering a predetermined interaction when the user interacts with the visual representation of the user interface component. 
   
   
       37 . The method of  claim 21 , further comprising employing a camera worn by the user to capture the image including at least part of the user. 
   
   
       38 . The method of  claim 21 , further comprising employing a visual display worn by the user to display visual 3D information to the user including the image having the visual representation of the of the user interface component. 
   
   
       39 . The method of  claim 21 , further comprising:
 extracting and recognizing image content located at least one of inside or near the fiducial marker; and   determining which of plural user interface components to display in relation to the fiducial marker based on the image content.   
   
   
       40 . The method of  claim 21 , further comprising detecting the fiducial marker attached to skin of at least one of a hand, wrist, or arm of the user. 
   
   
       41 . The method of  claim 21 , further comprising detecting the fiducial marker on clothing worn by the user. 
   
   
       42 . The method of  claim 21 , further comprising detecting the fiducial marker on at least one of a watch or jewelry worn by the user.