Cloud-Enabled Architecture For On-Demand Native Application Crawling

Abstract

A search system includes a query receiver, a live scrape system, and a content provider. The query receiver is configured to receive a first query related to a first application state of a first application. The first query is based on a user-requested search from a user device. The live scrape system is configured to operate a device including an operating system in which an instance of the first application executes. The live scrape system is configured to, responsive to the first query, and within the device, (i) navigate to the first application state in the instance of the first application and (ii) extract content from the instance of the first application. The content provider is configured to provide the extracted content to the user device for presentation to a user of the user device in real time.

What is claimed is: 
     
         1 . A search system comprising:
 a query receiver configured to receive a first query related to a first application state of a first application, wherein the first query is based on a user-requested search from a user device;   a live scrape system configured to:
 operate a device including an operating system in which an instance of the first application executes; and 
 responsive to the first query, and within the device, (i) navigate to the first application state in the instance of the first application and (ii) extract content from the instance of the first application; and 
   a content provider configured to provide the extracted content to the user device for presentation to a user of the user device in real time.   
     
     
         2 . The search system of  claim 1  wherein the device comprises one of:
 an emulator configured to execute the operating system on emulated hardware; and 
 a simulator configured to simulate interaction of the operating system with the first application. 
 
     
     
         3 . The search system of  claim 1  wherein the live scrape system is configured to, responsive to the first query, and within the device:
 navigate to the first application state in the instance of the first application; 
 supply search parameters based on the first query to the first application state; 
 initiate a search action from the first application state; and 
 extract content from search results provided by the instance of the first application in response to the search action. 
 
     
     
         4 . The search system of  claim 1  further comprising a query understanding pipeline configured to parse the user-requested search and identify a set of entity types provided in the user-requested search, wherein the first query is generated in response to the identified set of entity types from the user-requested search. 
     
     
         5 . The search system of  claim 4  wherein:
 the user-requested search includes a text string; and 
 the query understanding pipeline is configured to identify one or more substrings of the text string, each substring corresponding to an entity having a corresponding entity type. 
 
     
     
         6 . The search system of  claim 4  further comprising a rules-based search module configured to:
 store a set of rules, each rule designating (i) an application, (ii) an application state of the designated application through which a search can be performed, and (iii) for each input parameter of the search, a type of entity that can be provided as the input parameter; 
 select a first rule from the set of rules based on a match between the designated entity types for the first rule and the identified set of entity types from the user-requested search; and 
 generate the first query based on the first rule. 
 
     
     
         7 . The search system of  claim 1  wherein the extracted content is provided to the user device in a form of at least one deep view card (DVC), wherein each DVC of the at least one DVC includes content to be displayed to the user including an identification of the first application, an identification of the first application state, and information extracted from the first application state. 
     
     
         8 . The search system of  claim 7  further comprising:
 a DVC builder configured to format the extracted content into the at least one DVC; and 
 a DVC template store configured to store a first template that specifies which portions of the extracted content correspond to fields in the at least one DVC. 
 
     
     
         9 . The search system of  claim 1  further comprising a query router configured to, responsive to the query receiver receiving a second query:
 determine whether results data corresponding to the second query have been cached; 
 in response to the results data having been cached, selectively provide the results data to the content provider; and 
 in response to the results data not having been cached, selectively provide the second query to the live scrape system. 
 
     
     
         10 . The search system of  claim 9  wherein the query router is configured to, responsive to the query receiver receiving the second query, selectively provide the second query to the live scrape system in response to the cached results data being out of date. 
     
     
         11 . The search system of  claim 1  further comprising a query router configured to, responsive to the query receiver receiving a second query:
 in response to an application programming interface (API) being available for an application state specified by the second query, selectively provide the second query to an API passthrough to be serviced by the available API; and 
 in response to no API being available for the application state specified by the second query, selectively provide the second query to the live scrape system. 
 
     
     
         12 . The search system of  claim 1  further comprising a query router configured to, responsive to the query receiver receiving a second query:
 in response to a web edition of an application state specified by the second query being available via HTTP (hypertext transfer protocol), selectively provide the second query to a web scrape system to be serviced by the available web edition; and 
 in response to no web edition being available for the application state specified by the second query, selectively provide the second query to the live scrape system. 
 
     
     
         13 . A computer-implemented method comprising:
 receiving a first query related to a first application state of a first application, wherein the first query is based on a user-requested search from a user device;   operating a device including an operating system in which an instance of the first application executes;   responsive to the first query, and within the device, (i) navigating to the first application state in the instance of the first application and (ii) extracting content from the instance of the first application; and   providing the extracted content to the user device for presentation to a user of the user device in real time.   
     
     
         14 . The method of  claim 13  wherein operating the device comprises one of:
 operating an emulator configured to execute the operating system on emulated hardware; and 
 operating a simulator configured to simulate interaction of the operating system with the first application. 
 
     
     
         15 . The method of  claim 13  further comprising, responsive to the first query, and within the device:
 navigating to the first application state in the instance of the first application; 
 supplying search parameters based on the first query to the first application state; 
 initiating a search action from the first application state; and 
 extracting content from search results provided by the instance of the first application in response to the search action. 
 
     
     
         16 . The method of  claim 13  further comprising:
 parsing the user-requested search; and 
 identifying a set of entity types provided in the user-requested search, wherein the first query is generated in response to the identified set of entity types from the user-requested search. 
 
     
     
         17 . The method of  claim 16  wherein:
 the user-requested search includes a text string; and 
 the method further comprises identifying one or more substrings of the text string, wherein each substring corresponds to an entity having a corresponding entity type. 
 
     
     
         18 . The method of  claim 16  further comprising:
 storing a set of rules, each rule designating (i) an application, (ii) an application state of the designated application through which a search can be performed, and (iii) for each input parameter of the search, a type of entity that can be provided as the input parameter; 
 selecting a first rule from the set of rules based on a match between the designated entity types for the first rule and the identified set of entity types from the user-requested search; and 
 generating the first query based on the first rule. 
 
     
     
         19 . The method of  claim 13  wherein:
 the extracted content is provided to the user device in a form of at least one deep view card (DVC); and 
 each DVC of the at least one DVC includes content to be displayed to the user including an identification of the first application, an identification of the first application state, and information extracted from the first application state. 
 
     
     
         20 . The method of  claim 19  further comprising:
 formatting the extracted content into the at least one DVC; and 
 storing DVC templates including a first template, wherein the first template specifies which portions of the extracted content correspond to fields in the at least one DVC. 
 
     
     
         21 . The method of  claim 13  further comprising, in response to receiving a second query:
 determining whether results data corresponding to the second query have been cached; 
 in response to the results data having been cached, selectively providing the cached results data to the user device; and 
 in response to the results data not having been cached, selectively navigating to an application state based on the second query, extracting content, and providing the extracted content to the user device. 
 
     
     
         22 . The method of  claim 21  wherein the selectively navigating to an application state based on the second query, extracting content, and providing the extracted content to the user device are performed in response to the cached results data being out of date. 
     
     
         23 . The method of  claim 13  further comprising, in response to receiving a second query corresponding to a second application:
 in response to an application programming interface (API) being available for an application state specified by the second query, selectively (i) preparing an API call, (ii) transmitting the API call to a backend server for the second application, and (iii) providing content to the user device based on a response to the API call from the backend server; and 
 in response to no API being available for the application state specified by the second query, selectively navigating to an application state based on the second query, extracting content, and providing the extracted content to the user device. 
 
     
     
         24 . The method of  claim 13  further comprising, in response to receiving a second query corresponding to a second application:
 in response to a web edition of the second application being available, selectively (i) sending an HTTP (hypertext transfer protocol) request to a backend server for the second application, (ii) receiving an HTTP response from the backend server, (iii) scraping data from the HTTP response, and (iv) providing content to the user device based on the scraped data; and 
 in response to no web edition being available for the second application, selectively navigating to an application state based on the second query, extracting content, and providing the extracted content to the user device.