Reinforcement learning using target neural networks

Abstract

We describe a method of reinforcement learning for a subject system having multiple states and actions to move from one state to the next. Training data is generated by operating on the system with a succession of actions and used to train a second neural network. Target values for training the second neural network are derived from a first neural network which is generated by copying weights of the second neural network at intervals.

What is claimed is: 
     
         1 . (canceled) 
     
     
         2 . A method performed by one or more computers, the method comprising:
 maintaining a replay memory that stores a plurality of transitions, each transition comprising respective starting state data defining a respective starting state of an environment, respective action data defining a respective action from a set of actions, and respective next state data defining a respective next state of the environment resulting from the respective action being performed in the environment when the environment is in the respective starting state;   sampling a transition from the replay memory in accordance with a sampling strategy that prioritizes different transitions in the replay memory differently; and   training a second neural network on at least the sampled transition, wherein the second neural network is a deep neural network that is configured to receive an input comprising input state data and to generate as output a respective action-value parameter for each action in the set of actions.   
     
     
         3 . The method of  claim 2 , wherein the sampling strategy prioritizes transitions based on how much the second neural network is likely to learn from being trained on the transitions. 
     
     
         4 . The method of  claim 2 , wherein training a second neural network on at least the sampled transition comprises:
 processing an input comprising the respective starting state data in the sampled transition using the second neural network to generate an action-value parameter for the respective action defined by the respective action data in the sampled transition;   generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network; and   training the second neural network to optimise a loss function that depends on a difference between the target value and the action-value parameter for the respective action defined by the respective action data in the sampled transition.   
     
     
         5 . The method of  claim 4 , wherein the first neural network is an instance of the second neural network with different weights from the first neural network. 
     
     
         6 . The method of  claim 4 , wherein generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network comprises:
 processing an input comprising the respective next state data in the sampled transition using the first neural network to generate a respective next action-value parameter for each action in the set of actions.   
     
     
         7 . The method of  claim 6 , wherein generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network comprises:
 identifying a maximum next action-value parameter from the respective next action-value parameters.   
     
     
         8 . The method of  claim 7 , wherein the sampled transition further comprises a reward value and wherein generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network comprises:
 determining the target value from the reward value and the identified maximum next action-value parameter.   
     
     
         9 . The method of  claim 2 , further comprising:
 generating a new transition using the second neural network; and   storing the new transition in the replay memory.   
     
     
         10 . The method of  claim 9 , further comprising:
 after storing the new transition in the replay memory, discarding one or more of the plurality of transitions from the replay memory.   
     
     
         11 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
 maintaining a replay memory that stores a plurality of transitions, each transition comprising respective starting state data defining a respective starting state of an environment, respective action data defining a respective action from a set of actions, and respective next state data defining a respective next state of the environment resulting from the respective action being performed in the environment when the environment is in the respective starting state;   sampling a transition from the replay memory in accordance with a sampling strategy that prioritizes different transitions in the replay memory differently; and   training a second neural network on at least the sampled transition, wherein the second neural network is a deep neural network that is configured to receive an input comprising input state data and to generate as output a respective action-value parameter for each action in the set of actions.   
     
     
         12 . The method of  claim 11 , wherein the sampling strategy prioritizes transitions based on how much the second neural network is likely to learn from being trained on the transitions. 
     
     
         13 . The system of  claim 11 , wherein training a second neural network on at least the sampled transition comprises:
 processing an input comprising the respective starting state data in the sampled transition using the second neural network to generate an action-value parameter for the respective action defined by the respective action data in the sampled transition;   generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network; and   training the second neural network to optimise a loss function that depends on a difference between the target value and the action-value parameter for the respective action defined by the respective action data in the sampled transition.   
     
     
         14 . The system of  claim 13 , wherein the first neural network is an instance of the second neural network with different weights from the first neural network. 
     
     
         15 . The system of  claim 13 , wherein generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network comprises:
 processing an input comprising the respective next state data in the sampled transition using the first neural network to generate a respective next action-value parameter for each action in the set of actions.   
     
     
         16 . The system of  claim 15 , wherein generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network comprises:
 identifying a maximum next action-value parameter from the respective next action-value parameters.   
     
     
         17 . The system of  claim 16 , wherein the sampled transition further comprises a reward value and wherein generating, from at least the respective next state data in the sampled transition, a target value for the second neural network using a first neural network comprises:
 determining the target value from the reward value and the identified maximum next action-value parameter.   
     
     
         18 . The system of  claim 11 , the operations further comprising:
 generating a new transition using the second neural network; and   storing the new transition in the replay memory.   
     
     
         19 . The system of  claim 18 , the operations further comprising:
 after storing the new transition in the replay memory, discarding one or more of the plurality of transitions from the replay memory.   
     
     
         20 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
 maintaining a replay memory that stores a plurality of transitions, each transition comprising respective starting state data defining a respective starting state of an environment, respective action data defining a respective action from a set of actions, and respective next state data defining a respective next state of the environment resulting from the respective action being performed in the environment when the environment is in the respective starting state;   sampling a transition from the replay memory in accordance with a sampling strategy that prioritizes different transitions in the replay memory differently; and   training a second neural network on at least the sampled transition, wherein the second neural network is a deep neural network that is configured to receive an input comprising input state data and to generate as output a respective action-value parameter for each action in the set of actions.   
     
     
         21 . The non-transitory computer-readable storage media of  claim 20 , wherein the sampling strategy prioritizes transitions based on how much the second neural network is likely to learn from being trained on the transitions.