Object localization using machine learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a location of a particular object relative to a vehicle. In one aspect, a method includes obtaining sensor data captured by one or more sensors of a vehicle. The sensor data is processed by a convolutional neural network to generate a sensor feature representation of the sensor data. Data is obtained which defines a particular spatial region in the sensor data that has been classified as including sensor data that characterizes the particular object. An object feature representation of the particular object is generated from a portion of the sensor feature representation corresponding to the particular spatial region. The object feature representation of the particular object is processed using a localization neural network to generate the location of the particular object relative to the vehicle.

What is claimed is: 
     
         1 . A method performed by one or more data processing apparatus, the method comprising:
 obtaining sensor data captured by one or more sensors of a vehicle;   generating a sensor feature representation of the sensor data by processing an input comprising the sensor data using an encoding neural network;   obtaining data defining a spatial region in the sensor data that has been classified as including sensor data that characterizes an object in an environment in a vicinity of the vehicle;   generating an object feature representation of the object from a portion of the sensor feature representation corresponding to the spatial region; and   processing an input comprising the object feature representation of the object using a localization neural network to generate a localization output characterizing a location of the object in the environment relative to the vehicle;   wherein the encoding neural network and the localization neural network have been jointly trained to optimize a loss function by training operations comprising:
 using the encoding neural network and the localization neural network to generate a predicted localization output; 
 determining gradients of the loss function with respect to current parameter values of the encoding neural network and the localization neural network, wherein the loss function measures an error in the predicted localization output; and 
 updating the current parameter values of the encoding neural network and the localization neural network using the gradients of the loss function. 
   
     
     
         2 . The method of  claim 1 , wherein the predicted localization output defines a location of a training object in a training environment relative to a training vehicle;
 generating the predicted localization output comprises processes training sensor data captured by one or more sensors of the training vehicle using the encoding neural network; and   the training operations further comprise training the encoding neural network to generate auxiliary outputs characterizing a geometry of the training environment.   
     
     
         3 . The method of  claim 2 , wherein the auxiliary outputs characterizing the geometry of the training environment comprise one or more of: (i) an auxiliary output defining a predicted depth of the training environment, (ii) an auxiliary output that define spatial regions in the training sensor data that include sensor data characterizing objects in the training environment, or (iii) an auxiliary output defining a spatial region in the sensor data that defines a location of an area of ground beneath the training object. 
     
     
         4 . The method of  claim 2 , wherein the training operations further comprise training the localization neural network to generate auxiliary outputs characterizing the training object. 
     
     
         5 . The method of  claim 4 , wherein the auxiliary outputs characterizing the training object comprise one or more of: (i) an auxiliary output defining a size of the training object, (ii) an auxiliary output defining a type of the training object, (iii) an auxiliary output that defines a future location of a center of the training object relative to the training vehicle after a predefined length of time has elapsed, or (iv) an auxiliary output that defines a future appearance of the training object after a predefined length of time has elapsed. 
     
     
         6 . The method of  claim 2 , further comprising masking a portion of the training sensor data prior to processing the training sensor data using the encoding neural network. 
     
     
         7 . The method of  claim 6 , wherein the training sensor data comprises data captured by a plurality of sensors, and masking a portion of the training sensor data comprises:
 masking a portion of the training sensor data captured by one or more specified sensors of the plurality of sensors.   
     
     
         8 . The method of  claim 6 , wherein masking a portion of the training sensor data comprises setting the portion of the training sensor data equal to a constant value. 
     
     
         9 . The method of  claim 1 , wherein obtaining sensor data captured by one or more sensors of the vehicle comprises:
 aligning and combining sensor data captured by one or more of: a laser sensor of the vehicle, a radar sensor of the vehicle, and a camera sensor of the vehicle.   
     
     
         10 . The method of  claim 1 , wherein obtaining data defining a particular spatial region in the sensor data that has been classified as including sensor data that characterizes a particular object in the environment in the vicinity of the vehicle comprises:
 obtaining an object bounding box with a rectangular geometry.   
     
     
         11 . The method of  claim 1 , wherein obtaining data defining a particular spatial region in the sensor data that has been classified as including sensor data that characterizes a particular object in an environment in a vicinity of the vehicle comprises:
 obtaining data generated by processing at least part of the sensor data using an object detection neural network to generate data defining the particular spatial region.   
     
     
         12 . The method of  claim 1 , further comprising:
 generating the object feature representation of the object from a portion of the sensor data corresponding to the spatial region in addition to the portion of the sensor feature representation corresponding to the spatial region.   
     
     
         13 . The method of  claim 1 , wherein generating an object feature representation of the object from a portion of the sensor feature representation corresponding to the spatial region comprises:
 cropping the portion of the sensor feature representation corresponding to the spatial region; and   transforming the cropped portion of the sensor feature representation to a fixed size using one or more pooling operations.   
     
     
         14 . The method of  claim 1 , wherein processing an input comprising the object feature representation of the object using a localization neural network to generate an output characterizing a location of the object in the environment relative to the vehicle comprises:
 processing the input comprising the object feature representation using the localization neural network to generate an output comprising coordinates characterizing a position of a center of the object in the environment, wherein the coordinates are expressed in a coordinate system which is defined relative to the vehicle.   
     
     
         15 . The method of  claim 1 , wherein processing an input comprising the object feature representation of the object using a localization neural network to generate an output characterizing a location of the object in the environment relative to the vehicle comprises:
 processing the input comprising the object feature representation using the localization neural network to generate an output comprising a distance value characterizing a distance of the object in the environment from the vehicle.   
     
     
         16 . A system comprising:
 one or more computers; and   one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:   obtaining sensor data captured by one or more sensors of a vehicle;   generating a sensor feature representation of the sensor data by processing an input comprising the sensor data using an encoding neural network;   obtaining data defining a spatial region in the sensor data that has been classified as including sensor data that characterizes an object in an environment in a vicinity of the vehicle;   generating an object feature representation of the object from a portion of the sensor feature representation corresponding to the spatial region; and   processing an input comprising the object feature representation of the object using a localization neural network to generate a localization output characterizing a location of the object in the environment relative to the vehicle;   wherein the encoding neural network and the localization neural network have been jointly trained to optimize a loss function by training operations comprising:
 using the encoding neural network and the localization neural network to generate a predicted localization output; 
 determining gradients of the loss function with respect to current parameter values of the encoding neural network and the localization neural network, wherein the loss function measures an error in the predicted localization output; and 
 updating the current parameter values of the encoding neural network and the localization neural network using the gradients of the loss function. 
   
     
     
         17 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
 obtaining sensor data captured by one or more sensors of a vehicle;   generating a sensor feature representation of the sensor data by processing an input comprising the sensor data using an encoding neural network;   obtaining data defining a spatial region in the sensor data that has been classified as including sensor data that characterizes an object in an environment in a vicinity of the vehicle;   generating an object feature representation of the object from a portion of the sensor feature representation corresponding to the spatial region; and   processing an input comprising the object feature representation of the object using a localization neural network to generate a localization output characterizing a location of the object in the environment relative to the vehicle;   wherein the encoding neural network and the localization neural network have been jointly trained to optimize a loss function by training operations comprising:
 using the encoding neural network and the localization neural network to generate a predicted localization output; 
 determining gradients of the loss function with respect to current parameter values of the encoding neural network and the localization neural network, wherein the loss function measures an error in the predicted localization output; and 
 updating the current parameter values of the encoding neural network and the localization neural network using the gradients of the loss function. 
   
     
     
         18 . The non-transitory computer storage media of  claim 17 , wherein the predicted localization output defines a location of a training object in a training environment relative to a training vehicle;
 generating the predicted localization output comprises processes training sensor data captured by one or more sensors of the training vehicle using the encoding neural network; and   the training operations further comprise training the encoding neural network to generate auxiliary outputs characterizing a geometry of the training environment.   
     
     
         19 . The non-transitory computer storage media of  claim 18 , wherein the auxiliary outputs characterizing the geometry of the training environment comprise one or more of: (i) an auxiliary output defining a predicted depth of the training environment, (ii) an auxiliary output that define spatial regions in the training sensor data that include sensor data characterizing objects in the training environment, or (iii) an auxiliary output defining a spatial region in the sensor data that defines a location of an area of ground beneath the training object. 
     
     
         20 . The non-transitory computer storage media of  claim 18 , wherein the training operations further comprise training the localization neural network to generate auxiliary outputs characterizing the training object.