Reinforcement learning techniques to improve searching and/or to conserve computational and network resources

Abstract

Implementations are related to observing user interactions in association with searching for various files, and modifying a model and/or index based on such observations in order to improve the search process. In some implementations, a reinforcement learning model is utilized to adapt one or more search actions of the search process. Such search action(s) can include, for example, updating an index, reweighting terms in an index, modifying a search query, and/or modifying one or more ranking signal(s) utilized in raking search results. A policy of the reinforcement learning model can be utilized to generate action parameters that dictate performance of search action(s) for a search query, dependent on an observed state that is based on the search query. The policy can be iteratively updated in view of a reward function, and observed user interactions across multiple search sessions, to generate a learned policy that reduces duration of search sessions.

We claim: 
     
         1 . A method implemented by one or more processors, the method comprising:
 receiving, at a search interface, a search request that includes a plurality of terms;   determining, based on one or more of the terms, a state for the search request;   processing the state utilizing a reinforcement learning model to generate action parameters for use in performing a search in response to the search request,   wherein the action parameters include retrieval parameters that dictate how files are to be identified, using a search engine index, as responsive to the search request;   performing the search based on the search request and the action parameters,   wherein performing the search based on the action parameters comprises selecting, based on the retrieval parameters and from a plurality of candidate options for identifying files as responsive to the search request, a particular option to utilize for identifying files as responsive to the search request;   providing search results based on performance of the search that is based on the search request and the action parameters, the search results corresponding to the identified files;   determining a reward as a function of at least a duration of the search, wherein the reward is increased as the duration decreases;   modifying the reinforcement learning model based on the determined reward; and   subsequent to modifying the reinforcement learning model:
 receiving an additional search request, 
 determining an additional state for the additional search request; 
   processing the particular state utilizing the modified reinforcement learning model to generate additional action parameters, including additional retrieval parameters, and   performing an additional search based on the additional search request and the additional action parameters,
 wherein performing the additional search based on the additional action parameters comprises selecting, based on the additional retrieval parameters and from the plurality of candidate options for identifying files as responsive to the search request, an additional particular option to utilize for identifying additional files as responsive to the additional search request. 
   
     
     
         2 . The method of  claim 1 , wherein the selected particular option to utilize for identifying files as responsive to the search request comprises causing one or more additional terms, that are in addition to the terms of the search query, to be utilized in searching the search engine index. 
     
     
         3 . The method of  claim 2 , wherein the plurality of options for identifying files as responsive to the search request further comprises an alternate option that comprises causing one or more alternative terms to be utilized, in lieu of one or more of the terms of the search query, in searching the search engine index. 
     
     
         4 . The method of  claim 1 , wherein the particular option to utilize for identifying files as responsive to the search request comprises causing one or more alternative terms to be utilized, in lieu of one or more of the terms of the search query, in searching the search engine index. 
     
     
         5 . The method of  claim 4 , wherein the plurality of options for identifying files as responsive to the search request further comprises an alternate option that comprises causing one or more additional terms, that are in addition to the terms of the search query, to be utilized in searching the search engine index. 
     
     
         6 . The method of  claim 1 ,
 wherein the retrieval parameters dictate a modification to be made to the search request to generate a modified search request, and   wherein utilizing the particular option for identifying files as responsive to the search request comprises searching the search engine index using one or more components of the modified search request.   
     
     
         7 . The method of  claim 1 , wherein the action parameters further include one or more ranking parameters that dictate how search results are to be ranked, and wherein performing the search based on the action parameters further comprises:
 ranking, based on the ranking parameters, the search results identified based on the retrieval parameters,
 wherein providing the search results comprises providing the search results based on the ranking of the search results. 
   
     
     
         8 . A method implemented by one or more processors, the method comprising:
 receiving, at a search interface, a search request that includes a plurality of terms;   determining, based on one or more of the terms, a state for the search request;   processing the state utilizing a reinforcement learning model to generate action parameters for use in performing a search in response to the search request,   wherein the action parameters include retrieval parameters that dictate how files are to be identified, using a search engine index, as responsive to the search request;   performing the search based on the search request and the action parameters,   wherein performing the search based on the action parameters comprises selecting, based on the retrieval parameters and from a plurality of candidate options for identifying files as responsive to the search request, a particular option to utilize for identifying files as responsive to the search request;   providing search results based on performance of the search that is based on the search request and the action parameters, the search results corresponding to the identified files;   wherein the reinforcement learning model is trained based at least in part on:   receiving a prior search request that includes a plurality of prior terms;   determining, based on one or more of the prior terms, a prior state for the prior search request;   processing the prior state utilizing a prior version of the reinforcement learning model to generate prior action parameters for use in performing a prior search in response to the prior search request,
 wherein the prior action parameters include prior retrieval parameters; 
   performing the prior search based on the prior search request and the prior action parameters, including the prior retrieval parameters; and   determining a reward as a function of at least a duration of the prior search, wherein the reward is increased as the duration decreases; and   modifying the prior version of the reinforcement learning model based on the determined reward, wherein the reinforcement learning model is generated based on modifying the prior version.   
     
     
         9 . The method of  claim 8 , wherein the selected particular option to utilize for identifying files as responsive to the search request comprises causing one or more additional terms, that are in addition to the terms of the search query, to be utilized in searching the search engine index. 
     
     
         10 . The method of  claim 9 , wherein the plurality of options for identifying files as responsive to the search request further comprises an alternate option that comprises causing one or more alternative terms to be utilized, in lieu of one or more of the terms of the search query, in searching the search engine index. 
     
     
         11 . The method of  claim 8 , wherein the particular option to utilize for identifying files as responsive to the search request comprises causing one or more alternative terms to be utilized, in lieu of one or more of the terms of the search query, in searching the search engine index. 
     
     
         12 . The method of  claim 11 , wherein the plurality of options for identifying files as responsive to the search request further comprises an alternate option that comprises causing one or more additional terms, that are in addition to the terms of the search query, to be utilized in searching the search engine index. 
     
     
         13 . The method of  claim 8 ,
 wherein the retrieval parameters dictate a modification to be made to the search request to generate a modified search request, and   wherein utilizing the particular option for identifying files as responsive to the search request comprises searching the search engine index using one or more components of the modified search request.   
     
     
         14 . The method of  claim 8 , wherein the action parameters further include one or more ranking parameters that dictate how search results are to be ranked, and wherein performing the search based on the action parameters further comprises:
 ranking, based on the ranking parameters, the search results identified based on the retrieval parameters,
 wherein providing the search results comprises providing the search results based on the ranking of the search results. 
   
     
     
         15 . A system comprising:
 memory storing instructions;   one or more processors executing the instructions to:   receive, at a search interface, a search request that includes a plurality of terms;   determine, based on one or more of the terms, a state for the search request;   process the state utilizing a reinforcement learning model to generate action parameters for use in performing a search in response to the search request,
 wherein the action parameters include retrieval parameters that dictate how files are to be identified, using a search engine index, as responsive to the search request; 
   perform the search based on the search request and the action parameters,
 wherein in performing the search based on the action parameters one or more of the processors are to select, based on the retrieval parameters and from a plurality of candidate options for identifying files as responsive to the search request, a particular option to utilize for identifying files as responsive to the search request; 
   provide search results based on performance of the search that is based on the search request and the action parameters, the search results corresponding to the identified files;   determine a reward as a function of at least a duration of the search, wherein the reward is increased as the duration decreases;   modify the reinforcement learning model based on the determined reward; and   subsequent to modifying the reinforcement learning model:
 receive an additional search request, 
 determine an additional state for the additional search request; 
   process the particular state utilizing the modified reinforcement learning model to generate additional action parameters, including additional retrieval parameters, and   perform an additional search based on the additional search request and the additional action parameters,
 wherein in performing the additional search based on the additional action parameters one or more of the processors are to select, based on the additional retrieval parameters and from the plurality of candidate options for identifying files as responsive to the search request, an additional particular option to utilize for identifying additional files as responsive to the additional search request. 
   
     
     
         16 . The system of  claim 15 , wherein the selected particular option to utilize for identifying files as responsive to the search request comprises causing one or more additional terms, that are in addition to the terms of the search query, to be utilized in searching the search engine index. 
     
     
         17 . The system of  claim 16 , wherein the plurality of options for identifying files as responsive to the search request further comprises an alternate option that comprises causing one or more alternative terms to be utilized, in lieu of one or more of the terms of the search query, in searching the search engine index. 
     
     
         18 . The system of  claim 15 , wherein the particular option to utilize for identifying files as responsive to the search request comprises causing one or more alternative terms to be utilized, in lieu of one or more of the terms of the search query, in searching the search engine index.