Systems and Methods for Slate Optimization with Recurrent Neural Networks

Abstract

Systems and methods for generating a slate of ranked items are provided. In one example embodiment, a computer-implemented method includes inputting a sequence of candidate items into a machine-learned model, and obtaining, in response to inputting the sequence of candidate items into the machine-learned model, an output of the machine-learned model that includes a ranking of the candidate items that presents a diverse set of the candidate items at the top positions in the ranking such that one or more highly relevant candidate items can be demoted in the ranking.

What is claimed is: 
     
         1 . A computer system comprising:
 one or more processors; and   one or more non-transitory computer readable media that collectively store:   a machine-learned pointer network for generating an output sequence from a list of candidate items, the machine-learned pointer network comprising:
 an encoder network configured to receive the list of candidate items and provide an output that includes a sequence of latent memory states; 
 a decoder network configured to receive a previously-selected candidate item for the output sequence and provide an output vector based at least in part on the previously-selected candidate item; and 
 an attention network configured to receive the sequence of latent memory states and a query including the output vector from the decoder network, the attention network configured to produce a probability distribution associated with a next candidate item to include in the output sequence, wherein the attention network produces the probability distribution based at least in part on candidate items that already appear in the output sequence; and 
   instructions that, when executed by the one or more processors, cause the computer system to perform operations, the operations comprising:
 providing an input associated with the list of candidate items to the machine-learned pointer network; 
 implementing the machine-learned pointer network to process the list of candidate items; 
 receiving an output generated by the machine-learned pointer network as a result of processing the list of candidate items; and 
 selecting the next candidate item to include in the output sequence based at least in part on the probability distribution. 
   
     
     
         2 . The computing system of  claim 1 , wherein:
 the attention network assigns scores to the list of candidate items as part of producing the probability distribution; and   the scores assigned to candidate items that already appear in the output sequence are lower than the scores assigned to candidate items that do not already appear in the output sequence.   
     
     
         3 . The computing system of  claim 2 , wherein:
 the scores assigned to candidate items that already appear in the output sequence enforces the attention network to output permutations.   
     
     
         4 . The computer system of  claim 1 , wherein:
 the decoder network operates over a plurality of decoding steps including a first decoding step; and   a learned vector is provided as an input to the first decoding step, and in response the decoder network provides an output vector based at least in part on the learned vector.   
     
     
         5 . The computer system of  claim 4 , wherein:
 the plurality of decoding steps includes a second decoding step; and   the operations further comprise inputting an embedding that corresponds to the previously-selected candidate item into the second decoding step of the decoder network.   
     
     
         6 . The computer system of  claim 1 , wherein the list of candidate items comprises a sequence of features associated with the list of candidate items. 
     
     
         7 . The computer system of  claim 1 , wherein:
 the encoder network includes a first recurrent neural network that utilizes long short-term memory cells; and   the decoder network includes a second recurrent neural network that utilizes long short-term memory cells.   
     
     
         8 . The computer system of  claim 1 , wherein the machine-learned pointer network is trained to point to candidate items in the list of candidate items based on a plurality of learned parameters. 
     
     
         9 . A computer-implemented method to train a machine-learned pointer network for generating an output sequence from a list of candidate items, the method comprising:
 obtaining, by one or more computing devices, data descriptive of the machine-learned pointer network, wherein the machine-learned pointer network comprises an encoder network configured to receive the list of candidate items and provide an output that includes a sequence of latent memory states, a decoder network that operates over a plurality of decoding steps and is configured to receive a previously-selected candidate item for the output sequence and provide an output vector based at least in part on the previously-selected candidate item, and an attention network configured to receive the sequence of latent memory states and a query including the output vector from the decoder network, wherein the attention network is configured to produce a probability distribution associated with a next candidate item to include in the output sequence;   training, by the one or more computing devices, the machine-learned pointer network based on a set of training data, wherein training, by the one or more computing devices, the machine-learned pointer network comprises:   determining, by the one or more computing devices, a per-step loss for two or more of the plurality of decoding steps, the per-step loss representing a performance evaluation of the machine-learned pointer network based on the set of training data; and   modifying, by the one or more computing devices, one or more parameters of the machine-learned pointer network based at least in part on the per-step loss.   
     
     
         10 . The computer-implemented method of  claim 9 , wherein training, by the one or more computing devices, the machine-learned pointer network comprises:
 enforcing a permutation constraint on an output of the machine-learned pointer network.   
     
     
         11 . The computer-implemented method of  claim 9 , wherein modifying, by the one or more computing devices, one or parameters of the machine-learned pointer network comprises backpropagating the per-step loss to train the machine-learned pointer network end-to-end. 
     
     
         12 . The computer-implemented method of  claim 9 , wherein feedback based on the per-step loss is provided to the machine-learned pointer network at each of a plurality of decoding steps of the decoder network. 
     
     
         13 . A computer-implemented method to generate an output sequence from a list of candidate items, the method comprising:
 ranking, by one or more computing devices, the list of candidate items to generate an initial sequence of candidate items;   inputting, by the one or more computing devices, the initial sequence of candidate items into an encoder network configured to receive the initial sequence of candidate items and provide an output that includes a sequence of latent memory states;   inputting, by the one or more computing devices, a previously-selected candidate item for the output sequence into a decoder network configured to receive the previously-selected candidate item and provide an output vector based at least in part on the previously-selected candidate item; and   inputting the sequence of latent memory states and a query including the output vector from the decoder network into an attention network configured to receive the sequence of latent memory states and the query including the output vector from the decoder network, the attention network configured to produce a probability distribution associated with a next candidate item to include in the output sequence.   
     
     
         14 . The computer-implemented method of  claim 13 , wherein:
 the decoder network operates over a plurality of decoding steps including a first decoding step; and   a learned vector is provided as an input to the first decoding step, and in response the decoder network provides an output vector based at least in part on the learned vector.   
     
     
         15 . The computer-implemented method of  claim 14 , wherein:
 the plurality of decoding steps includes a second decoding step; and   the method further comprises inputting an embedding that corresponds to the previously-selected candidate item into the second decoding step of the decoder network.   
     
     
         16 . The computer-implemented method of  claim 13 , wherein the list of candidate items comprises a sequence of features associated with the list of candidate items. 
     
     
         17 . The computer-implemented method of  claim 13 , wherein:
 the encoder network includes a first recurrent neural network that utilizes long short-term memory cells; and   the decoder network includes a second recurrent neural network that utilizes long short-term memory cells.   
     
     
         18 . The computer-implemented method of  claim 13 , wherein the encoder network, the decoder network, and the attention network are part of a machine-learned pointer network that is trained to point to candidate items in the list of candidate items based on a plurality of learned parameters. 
     
     
         19 . The computer-implemented method of  claim 13 , wherein the attention network is configured to produce the probability distribution based at least in part on candidate items that already appear in the output sequence. 
     
     
         20 . The computer-implemented method of  claim 13 , wherein a score assigned to each candidate item that already appears in the output sequence enforces the attention network to output permutations.