Natural user interfaces for mobile image viewing

Abstract

The mobile image viewing technique described herein provides a hands-free interface for viewing large imagery (e.g., 360 degree panoramas, parallax image sequences, and long multi-perspective panoramas) on mobile devices. The technique controls the imagery displayed on a display of a mobile device by movement of the mobile device. The technique uses sensors to track the mobile device's orientation and position, and front facing camera to track the user's viewing distance and viewing angle. The technique adjusts the view of a rendered imagery on the mobile device's display according to the tracked data. In one embodiment the technique can employ a sensor fusion methodology that combines viewer tracking using a front facing camera with gyroscope data from the mobile device to produce a robust signal that defines the viewer's 3D position relative to the display.

1 . A computer-implemented process for viewing large scale imagery on a mobile device, comprising:
 tracking a mobile device's orientation and position;   using a camera and viewer tracker on the mobile device to track a user's face looking at a screen on the mobile device;   computing a viewing angle and a viewing distance between the user and the screen on the mobile device by using the tracked orientation and position of the mobile device, and the tracked position of the user's face relative to screen of the mobile device; and   computing image transformations of an imagery rendered on the screen of the mobile device using the computed viewing angle and viewing distance to allow the user to control viewing of the rendered imagery.   
     
     
         2 . The computer-implemented process of  claim 1 , further comprising the user changing the viewpoint of imagery rendered on the screen by moving the mobile device relative to the user's face. 
     
     
         3 . The computer-implemented process of  claim 1 , further comprising zooming in or out of the imagery rendered on the screen by changing the distance of the mobile device relative to the user's face. 
     
     
         4 . The computer-implemented process of  claim 3  wherein the distance of the mobile device relative to the user's face is approximated using face width. 
     
     
         5 . The computer-implemented process of  claim 1 , further comprising panning around the imagery rendered on the screen by changing the position of the mobile device laterally in relation to the user's face. 
     
     
         6 . The computer-implemented process of  claim 1 , further comprising mapping the angular offset of the user's face from the normal to the screen of the mobile device and the change in rotation about the vertical axis tangent to the screen to a position in the imagery rendered in computing the image transformations. 
     
     
         7 . The computer-implemented process of  claim 6 , further comprising fusing the tracked mobile device's orientation and position and tracked user's face to map the angular offset of the user's face from the normal to the display of the mobile device and the change in rotation about the vertical axis tangent to the display of the mobile device to a position in the rendered imagery. 
     
     
         8 . The computer-implemented process of  claim 1  wherein the mobile device's orientation and position is determined by a gyroscope on the mobile device. 
     
     
         9 . The computer-implemented process of  claim 8  wherein the viewer tracker is used to correct for drift of the gyroscope. 
     
     
         10 . A computer-implemented process for viewing large scale imagery on a mobile device, comprising:
 tracking a mobile device's orientation and position with a gyroscope on the mobile device;   using a front-facing camera and viewer tracker on the mobile device to track a user's face looking at a screen on the mobile device;   using the mobile device's orientation and position from the gyroscope and the position of the user's face obtained by the viewer tracker to determine a combined position and rate control for viewing imagery on the screen of the mobile device.   using the combined position and rate control to compute image transformations of the imagery rendered on the screen of the mobile device to allow the user to display different viewpoints of the rendered imagery.   
     
     
         11 . The computer-implemented process of  claim 10 , wherein the imagery is a 360 degree panorama and wherein the user can pan to the left and to the right in the rendered imagery by changing the viewing angle between the user and the screen of the mobile device, and can zoom into the imagery by changing the distance between the user and the screen of the mobile device. 
     
     
         12 . The computer-implemented process of  claim 10 , wherein the imagery is a set of parallax images and wherein the combined position and rate control is used to determine a relative offset of a virtual camera. 
     
     
         13 . The computer-implemented process of  claim 10 , wherein the imagery comprises a series of 360 degree panoramas of the same scene taken at fixed intervals, and a set of long perspective strip panoramas created by clipping out and stitching parts of the series of  360  degree panoramas. 
     
     
         14 . The computer-implemented process of  claim 13 , wherein the user can view left and right in a  360  degree panorama of the series by changing the viewing angle between the user's face and the screen of the mobile device and can zoom into a different  360  degree panorama of the series by changing the viewing distance between the user's face and the screen of the mobile device. 
     
     
         15 . A system for viewing large scale imagery, comprising:
 a general purpose computing device;   a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,   track a mobile device's orientation and position;   use a camera and viewer tracker on the mobile device to track a user's face looking at a screen on the mobile device;   use the mobile device's tracked orientation and position, and the position of the user's face obtained by the viewer tracker, to determine a combined position and rate control for viewing imagery on the screen of the mobile device,   using the combined position and rate control to compute image transformations of the imagery rendered on the screen of the mobile device to allow the user to display different view points of the rendered imagery.   
     
     
         16 . The system of  claim 15 , wherein the module to determine the combined position and rate control for viewing imagery on the screen of the mobile device, further comprises a sub-module to:
 compute a viewing angle and a viewing distance between the user and the screen on the mobile device by using the tracked orientation and position of the mobile device, and the tracked position of the user's face relative to screen of the mobile device.   
     
     
         17 . The system of  claim 16  wherein the user can change the viewpoint of the imagery rendered on the screen of the mobile device by changing the viewing angle of the mobile device relative to the user's face. 
     
     
         18 . The system of  claim 16 , wherein the user's face can be outside of the field of view of the camera and wherein a gyroscope on the mobile device can be used to estimate the location of the face. 
     
     
         19 . The system of  claim 15 , wherein the viewer tracker tracks the viewer's face by:
 locating the viewer's face relative to the camera using a face finder which returns a rectangle for the size and location of the face;   recording a face template from the rectangle along with position and size;   matching the face template at varying positions and scales around the current position and scale at each subsequent frame recorded by the camera to find the face in subsequent frames;   if the face is lost, reacquiring the face with the face finder.   
     
     
         20 . The system of  claim 15 , wherein the large scale imagery is one of a group comprising:
 high resolution imagery;   wide field of view imagery; and   a multi-perspective panorama.