Providing actions for onscreen entities

Abstract

Implementations provide an interface for initiating actions related to content displayed on a mobile device. A method includes performing recognition on data from input devices of a mobile device and identifying a plurality of entities in the data. The method may include determine an entity of interest of the plurality of entities by scoring each of the plurality of entities and selecting the entity with the highest score as the entity of interest. The scoring can be based on user query data or global query data. The method may also include determining attributes associated with the entity of interest and displaying the entity of interest and one or more of the determined attributes on the computing device.

What is claimed is: 
     
         1 . A system for automatically identifying entities based on data from input devices and for providing attributes based on the identified entities, the system comprising:
 an entity engine adapted to:   receive data from a plurality of input devices of a computing device;   identify a plurality of entities from the data;   determine an entity of interest of the plurality of entities by scoring each of the plurality of entities and selecting the entity with the highest score as the entity of interest, wherein the scoring is based on at least one of user query data or global query data, wherein the user query data includes a history of queries associated with a user of the computing device, and wherein the global query data includes a history of queries associated with a plurality of users other than the user;   determine attributes associated with the entity of interest, wherein the attributes comprise descriptive information about the entity of interest; and   display the entity of interest and one or more of the determined attributes on the computing device.   
     
     
         2 . The system of  claim 1 , wherein the computing device comprises one or more of a smart phone, a laptop, desktop, or a tablet computer and the plurality of input devices include one or more of a camera, a scanner, and a global positioning system (GPS). 
     
     
         3 . The system of  claim 1 , wherein the data from the plurality of input devices includes data from an operating system of the device or an application running on the computing device. 
     
     
         4 . The system of  claim 1 , wherein the entity engine is further adapted to determine the descriptive information about the entity of interest from at least one of global entity data or user entity data. 
     
     
         5 . The system of  claim 1 , wherein the entity engine is further adapted to use a location of the device to determine the entity of interest. 
     
     
         6 . The system of  claim 1 , wherein the entity engine is further adapted to:
 receive user input associated with the user of the device; and   determine the entity of interest of the plurality of entities based on the user input.   
     
     
         7 . The system of  claim 1 , wherein the entity engine is further adapted to:
 store a plurality of rules, wherein each rule is associated with an entity and an action;   determine one or more rules of the plurality of rules that match an entity of the plurality of entities; and   perform the action associated with at least one of the one or more rules.   
     
     
         8 . The system of  claim 7 , wherein the action comprises one or more of finding a location and generating a notification. 
     
     
         9 . A method for automatically identifying entities and for determining attributes associated with the identified entities, the method comprising:
 collecting, by a device, data from one or more input devices of the device;   identifying, by the device, a plurality of entities from the data;   receiving user input by the device;   determining one or more entities of interest from the plurality of entities based on the received user input by the device and by scoring each of the plurality of entities and selecting the entity with the highest score as the entity of interest, wherein the scoring is based on at least one of user query data or global query data, wherein the user query data includes a history of queries associated with a user of the device, and wherein the global query data includes a history of queries associated with a plurality of users other than the user;   determining a query based on the received user input by the device;   determining attributes associated with the one or more entities of interest based on the query by the device, wherein the attributes comprise descriptive information about the one or more entities of interest; and   displaying the one or more entities of interest and one or more attributes of the determined attributes on a display associated with the device by the device.   
     
     
         10 . The method of  claim 9 , further comprising storing indicators of one or more of the entities of the plurality of entities in a file associated with the user of the device. 
     
     
         11 . The method of  claim 10 , wherein each indicator is associated with a time when the indicated entity was identified and a location where the indicated entity was identified. 
     
     
         12 . The method of  claim 9 , wherein identifying the plurality of entities from the data comprises using the data, one or more previously identified entities, and previously received data from input devices. 
     
     
         13 . The method of  claim 12 , wherein using the data includes using a location of the device to determine the entity of interest. 
     
     
         14 . The method of  claim 9 , further comprising:
 determining one or more rules of a plurality of rules that match an entity of the plurality of entities; and   performing an action associated with each rule that is determined to match an entity of the plurality of entities.   
     
     
         15 . The method of  claim 9 , wherein the device comprises one or more of a smart phone, a laptop, desktop, or a tablet computer. 
     
     
         16 . The method of  claim 9 , further comprising determining the descriptive information about the entity of interest from at least one of global entity data or user entity data. 
     
     
         17 . The method of  claim 9 , wherein the data from the one or more input devices includes data from an operating system of the device or an application running on the device. 
     
     
         18 . A method comprising:
 identifying a plurality of entities recognized from data generated by a mobile device for presentation to a user of the mobile device;   selecting at least a first entity of interest from the plurality of entities;   querying a search log of an Internet search engine for the first entity to obtain search results;   identifying in the search results for the first entity include information previously requested about the first entity by other users;   generating an action card for the first entity, wherein the action card includes the information previously requested about the entity of interest; and   providing the action card for display to a user of the mobile device, wherein the mobile device displays the action card.   
     
     
         19 . The method of  claim 18 , wherein the first entity is selected based on association with a user profile. 
     
     
         20 . The method of  claim 18 , wherein the first entity is selected based on association with a search history user. 
     
     
         21 . The method of  claim 18 , wherein the first is selected based on a display location when presented to the user. 
     
     
         22 . The method of  claim 18 , wherein the data generated by the mobile device for presentation to the user is an image. 
     
     
         23 . The method of  claim 18 , wherein the data generated by the mobile device for presentation to the user is an image. 
     
     
         24 . A method comprising:
 identifying a plurality of entities recognized from data generated by a mobile device for presentation to a user of the mobile device;   selecting at least a first entity of interest to the user from the plurality of entities;   querying a search engine for the first entity to generate search results, the search results including facts about the entity;   generating an action card for the first entity, wherein the action card includes the facts about the first entity; and   displaying the action card to a user of the mobile device, wherein the mobile device displays the action card.   
     
     
         25 . The method of  claim 24 , wherein the facts about the entity are facts requested by other users about the entity. 
     
     
         26 . The method of  claim 24 , wherein querying the search engine includes providing the query to an Internet-based search engine. 
     
     
         27 . The method of  claim 24 , wherein the data generated by the mobile device for presentation to the user is an image. 
     
     
         28 . The method of  claim 24 , wherein the first entity is selected based on association with a search history user. 
     
     
         29 . The method of  claim 24 , wherein the first entity is selected based on association with a user profile.