Using a Smartphone for Remote Interaction with Visual User Interfaces

Abstract

Methods and systems are provided for establishing an interaction session between a visual interface on any of a wide range of devices (e.g., POS, ATM, computer, television, or other visual display) and a smartphone having a camera by using the smartphone camera to scan or capture a special image (e.g., a QR code) presented on the visual interface. The special image serves as a token mark that can graphically encode information about the visual interface and the operator or owner of the device presenting the visual interface. Upon scanning the token mark, an application on the smartphone may establish a connection between the application and a server in communication with the visual interface. The application may then present some of actions provided by the visual interface on the smartphone display so that the user can select actions using the smartphone with the selected actions being performed on the visual interface.

What is claimed: 
     
         1 . A method, comprising:
 receiving, by an application executed by a user device, an image of a token mark presented in a user interface displayed at a remote computing device;   forwarding, to a server, information associated with the token mark effective to cause an interaction session between the application and the remote computing device to be established;   receiving, from the server, one or more selectable options associated with one or more data entry fields presented in the user interface displayed at the remote computing device;   displaying the one or more selectable options in a user interface of the application;   receiving a notification of selection of one of the one or more selectable options; and   forwarding, to the server, the notification of the selection effective to cause the remote computing device to perform an action associated with the notification, the action comprising an interaction with at least one of the one or more data entry fields.   
     
     
         2 . The method as recited in  claim 1 , further comprising:
 presenting, in the user interface of the application, a selectable scan control to enable scanning of the token mark;   receiving selection of the selectable scan control; and   causing, in response to said receiving selection of the selectable scan control, the scanning of the token mark to generate the image.   
     
     
         3 . The method as recited in  claim 1 , further comprising:
 presenting, in the user interface of the application, a selectable disconnect control to enable disconnecting the application from the interaction session.   
     
     
         4 . The method as recited in  claim 3 , further comprising:
 receiving selection of the selectable disconnect control; and   disconnecting, in response to receiving the selection of the selectable disconnect control, the interaction session.   
     
     
         5 . The method as recited in  claim 1 , wherein the application comprises a smartphone application. 
     
     
         6 . The method as recited in  claim 1 , further comprising:
 detecting a connection to the user interface displayed at the remote computing device; and   displaying, in the user interface of the application, an indication of the user interface displayed at the remote computing device based on said detecting.   
     
     
         7 . The method as recited in  claim 1 , wherein said forwarding the information associated with the token mark further comprises:
 generating updated token mark information that includes the information associated with the token mark and user device information; and   forwarding the updated token mark information to the server.   
     
     
         8 . The method as recited in  claim 1 , further comprising:
 receiving a list of commands supported by the user interface displayed at the remote computing device; and   presenting the list of commands as selectable buttons on the user interface of the application.   
     
     
         9 . One or more storage devices storing instructions that, responsive to execution by one or more processors, implement an application to perform operations comprising:
 receiving an image of a token mark presented in a user interface displayed at a remote computing device;   forwarding, to a server, information associated with the token mark effective to cause an interaction session between the application and the remote computing device to be established;   receiving, from the server, one or more selectable options associated with one or more data entry fields presented in the user interface displayed at the remote computing device;   displaying the one or more selectable options in a user interface associated of the application;   receiving a notification of selection of one of the one or more selectable options; and   forwarding, to the server, the notification of the selection effective to cause the remote computing device to perform an action associated with the notification, the action comprising an interaction with at least one of the one or more data entry fields.   
     
     
         10 . The one or more storage devices as recited in  claim 9 , wherein the application is further implemented to perform operations comprising:
 presenting, in the user interface of the application, a selectable scan control to enable scanning of the token mark;   receiving selection of the selectable scan control; and   causing, in response to said receiving selection of the selectable scan control, the scanning of the token mark to generate the image.   
     
     
         11 . The one or more storage devices as recited in  claim 9 , wherein the VIC application is further implemented to perform operations comprising:
 presenting, in the user interface of the application, a selectable disconnect control associated with disconnecting the application from the interaction session.   
     
     
         12 . The one or more storage devices as recited in  claim 11 , wherein the VIC application is further implemented to perform operations comprising:
 receiving selection of the selectable disconnect control; and   terminating, in response to receiving the selection of the selectable disconnect control, the interaction session.   
     
     
         13 . The one or more storage devices as recited in  claim 9 , wherein the application comprises a smartphone application. 
     
     
         14 . The one or more storage devices as recited in  claim 9 , wherein the application is further implemented to perform operations comprising:
 detecting a connection to the user interface displayed at the remote computing device; and   displaying, in the user interface of the application, an indication of the user interface displayed at the remote computing device based on said detecting.   
     
     
         15 . The one or more storage devices as recited in  claim 9 , wherein said forwarding the information associated with the token mark further comprises:
 generating updated token mark information that includes the information associated with the token mark and user device information; and   forwarding the updated token mark information to the server.   
     
     
         16 . The one or more storage devices as recited in  claim 9 , wherein the VIC application is further implemented to perform operations comprising:
 receiving a list of commands supported by the user interface displayed at the remote computing device; and   presenting the list of commands as selectable buttons on the user interface of the application.   
     
     
         17 . A user device, comprising:
 one or more processors; and   one or more storage devices storing instructions that, responsive to execution by the one or more processors, implement an application to perform operations comprising:
 receiving, by the application, an image of a token mark presented in a user interface displayed at a remote computing device; 
 forwarding, to a server, information associated with the token mark effective to cause an interaction session between the application and the remote computing device to be established; 
 receiving, from the server, one or more selectable options associated with one or more data entry fields presented in the user interface displayed at the remote computing device; 
 displaying the one or more selectable options in a user interface of the application; 
 receiving a notification of selection of one of the one or more selectable options; and 
 forwarding, to the server, the notification of the selection effective to cause the remote computing device to perform an action associated with the notification, the action comprising an interaction with at least one of the one or more data entry fields. 
   
     
     
         18 . The user device as recited in  claim 17 , wherein the application is further implemented to perform operations comprising:
 presenting, in the user interface of the application, a selectable scan control to enable scanning of the token mark;   receiving selection of the selectable scan control; and   causing, in response to said receiving selection of the selectable scan control, the scanning of the token mark to generate the image.   
     
     
         19 . The user device as recited in  claim 17 , wherein the application comprises a smartphone application. 
     
     
         20 . The user device as recited in  claim 17 , wherein said forwarding the information associated with the token mark further comprises:
 generating updated token mark information that includes the information associated with the token mark and user device information; and   forwarding the updated token mark information to the server.