Multi-agent simulations

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a simulation of an environment that is being interacted with by a plurality of agents over a plurality of time steps, wherein the simulation comprises a respective simulation state for each time step that specifies a respective state of each agent at the time step. In one aspect, a method comprises, for each time step: obtaining a current simulation state for the current time step; generating a plurality of candidate next simulation states for a next time step; determining, for each candidate next simulation state, a discriminative score characterizing a likelihood that the candidate next simulation state is a realistic simulation state; and selecting a candidate next simulation state as the simulation state for the next time step based on the discriminative scores for the candidate next simulation states.

1 . (canceled) 
     
     
         2 . A method performed by one or more computers for generating one or more simulations of an environment over a plurality of time steps, the method comprising, for each of one or more time steps:
 obtaining a plurality of current simulation states for the time step;   generating, for each of the plurality of current simulation states, a plurality of candidate next simulation states for a next time step based on the current simulation state, wherein generating each candidate next simulation state comprises:
 determining, for each of a plurality of agents included in the environment, a respective next state of the agent at the next time step if the agent performs a respective action in response to the current simulation state at the current time step; and 
   selecting a proper subset of the plurality of candidate next simulation states as current simulation states for the next time step.   
     
     
         3 . The method of  claim 2 , wherein for each of the plurality of current simulation states, generating each candidate next simulation state comprises:
 sampling, for each of the plurality of agents, the respective action corresponding to the agent from a set of possible actions that can be performed by the agent.   
     
     
         4 . The method of  claim 2 , wherein selecting a proper subset of the plurality of candidate next simulation states as current simulation states for the next time step comprises:
 determining, for each candidate next simulation state, a discriminative score characterizing a likelihood that the candidate next simulation state is a realistic simulation state; and   selecting the proper subset of the plurality of candidate next simulation states based on the discriminative scores for the plurality of candidate next simulation states.   
     
     
         5 . The method of  claim 4 , wherein for each candidate next simulation state, determining the discriminative score for the candidate next simulation state comprises:
 generating, for each agent, an agent-specific discriminative score characterizing a likelihood that the next state of the agent is a realistic state; and   generating the discriminative score for the candidate next simulation state based on the agent-specific discriminative scores.   
     
     
         6 . The method of  claim 5 , wherein generating the discriminative score for the candidate next simulation state based on the agent-specific discriminative scores comprises:
 computing a minimum of the agent-specific discriminative scores.   
     
     
         7 . The method of  claim 5 , wherein for each agent, generating an agent-specific discriminative score characterizing a likelihood that the next state of the agent is a realistic agent state comprises:
 obtaining, for each agent, a representation of the next state of the agent at the next time step corresponding to the candidate next simulation state; and   processing, for each agent, the representation of the next state of the agent at the next time step corresponding to the candidate next simulation state using a discriminator neural network to generate an agent-specific discriminative score characterizing a likelihood that the next state of the agent is a realistic agent state.   
     
     
         8 . The method of  claim 7 , wherein the discriminator neural network is trained to generate discriminative scores that characterize an environment state as being realistic if the environment state is a real-world environment state. 
     
     
         9 . The method of  claim 4 , wherein selecting the proper subset of the plurality of candidate next simulation states as current simulation states for the next time step comprises:
 selecting a predefined number of candidate next simulation state with highest discriminative scores, from among the plurality of candidate next simulation states, as current simulation states for the next time step.   
     
     
         10 . The method of  claim 2 , wherein the agents are vehicles in the environment. 
     
     
         11 . The method of  claim 10 , wherein a set of possible actions that can be performed by an agent comprise actions that adjust a steering angle of the agent. 
     
     
         12 . The method of  claim 10 , wherein a set of possible actions that can be performed by an agent comprise actions that adjust an acceleration of the agent. 
     
     
         13 . The method of  claim 2 , wherein each of the plurality of candidate next simulation states defines a respective state of each of the plurality of agents at the next time step, wherein the state of an agent defines: (i) a position of the agent at the time step, and (ii) a motion of the agent at the time step. 
     
     
         14 . The method of  claim 13 , wherein the position of the agent comprises: (i) a spatial location of the agent in the environment, and (ii) a heading of the agent in the environment. 
     
     
         15 . The method of  claim 13 , wherein the motion of the agent comprises: (i) a speed of the agent, and (ii) an acceleration of the agent. 
     
     
         16 . The method of  claim 3 , further comprising, for each of the plurality of agents:
 obtaining a representation of a current state of the agent in the environment; and   processing the representation of the current state of the agent in the environment using a policy neural network to generate a corresponding probability distribution over the set of possible actions for the agent;   wherein sampling, for each of the plurality of agents, the respective action from the set of possible actions comprises, for each agent:
 sampling an action from the set of possible actions in accordance with the probability distribution over the set of possible actions for the agent. 
   
     
     
         17 . The method of  claim 2 , wherein determining, for each of the plurality of agents, the respective next state of the agent at the next time step if the agent performs a respective action in response to the current simulation state at the current time step comprises, for each agent:
 processing data characterizing: (i) a current state of the agent, and (ii) the respective action, using a motion model to generate the next state of the agent at the next time step.   
     
     
         18 . A system comprising:
 one or more computers; and   one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for generating one or more simulations of an environment over a plurality of time steps, the operations comprising, for each of one or more time steps:   obtaining a plurality of current simulation states for the time step;   generating, for each of the plurality of current simulation states, a plurality of candidate next simulation states for a next time step based on the current simulation state, wherein generating each candidate next simulation state comprises:
 determining, for each of a plurality of agents included in the environment, a respective next state of the agent at the next time step if the agent performs a respective action in response to the current simulation state at the current time step; and 
   selecting a proper subset of the plurality of candidate next simulation states as current simulation states for the next time step.   
     
     
         19 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for generating one or more simulations of an environment over a plurality of time steps, the operations comprising, for each of one or more time steps:
 obtaining a plurality of current simulation states for the time step;   generating, for each of the plurality of current simulation states, a plurality of candidate next simulation states for a next time step based on the current simulation state, wherein generating each candidate next simulation state comprises:
 determining, for each of a plurality of agents included in the environment, a respective next state of the agent at the next time step if the agent performs a respective action in response to the current simulation state at the current time step; and 
   selecting a proper subset of the plurality of candidate next simulation states as current simulation states for the next time step.   
     
     
         20 . The non-transitory computer storage media of  claim 19 , wherein for each of the plurality of current simulation states, generating each candidate next simulation state comprises:
 sampling, for each of the plurality of agents, the respective action corresponding to the agent from a set of possible actions that can be performed by the agent.   
     
     
         21 . The non-transitory computer storage media of  claim 19 , wherein selecting a proper subset of the plurality of candidate next simulation states as current simulation states for the next time step comprises:
 determining, for each candidate next simulation state, a discriminative score characterizing a likelihood that the candidate next simulation state is a realistic simulation state; and   selecting the proper subset of the plurality of candidate next simulation states based on the discriminative scores for the plurality of candidate next simulation states.