Approaches for encoding environmental information

Abstract

Systems, methods, and non-transitory computer-readable media can access a plurality of schema-based encodings providing a structured representation of an environment captured by one or more sensors associated with a plurality of vehicles traveling through the environment. The plurality of schema-based encodings can be clustered into one or more clusters of schema-based encodings. At least one scenario associated with the environment can be determined based at least in part on the one or more clusters of schema-based encodings.

1 . A computer-implemented method comprising:
 receiving, by a computing system, sensor data captured by at least one sensor of a vehicle while the vehicle navigates an environment over a period of time comprising a plurality of time intervals;   for each time interval in the plurality of time intervals:
 determining, by the computing system, information describing one or more agents in the environment during the time interval based at least in part on the sensor data; and 
 generating, by the computing system, a schema-based encoding based on the determined information and a scenario schema, wherein the scenario schema comprises a set of elements including an agent element and an action element; 
   generating, by the computing system, a time-based representation of the environment by associating the schema-based encodings corresponding to the plurality of time intervals in temporal order with a point-in-time encoding of the environment generated for a point in time;   computing, by the computing system based on a plurality of the time-based representations of the environment, frequencies of scenarios experienced by vehicles while navigating the environment and determining exposure rates of the scenarios based on the frequencies; and   providing, by the computing system based on the frequencies or the exposure rates, vehicle operational instructions to avoid a road segment associated with a likelihood of encountering a scenario determined to be risky or modify operation to reduce risk.   
     
     
         2 . The method of  claim 1 , wherein the scenario schema further comprises a metadata element representing roadway attributes of the environment, comprising at least one of roadway type, speed limits, number of lanes, locations of intersections, merging lanes, traffic signals and states, street signs, curbs, presence of bicycle lanes, presence of crosswalks, or whether a road segment is in a residential, school, business, mixed-use, high-density, or rural zone. 
     
     
         3 . The method of  claim 1 , wherein the scenario schema further comprises a metadata element representing contextual information of the environment, comprising a calendar date, a day of week, a time of day, and weather conditions encountered while the vehicle navigates the environment. 
     
     
         4 . The method of  claim 1 , wherein determining the information describing the one or more agents in the environment during the time interval comprises:
 determining action or motion information comprising at least one of a velocity, a direction of travel, distances between the one or more agents, locations of the one or more agents relative to the vehicle, or locations of the one or more agents relative to other agents.   
     
     
         5 . The method of  claim 4 , further comprising:
 overlaying locations of the one or more agents on a semantic map of the environment to determine whether a given agent is on a sidewalk, in a bicycle lane, or in a particular lane of a road.   
     
     
         6 . The method of  claim 1 , wherein the time-based representation spans the period of time and identifies, for a given agent, different actions across sub-intervals within the period of time. 
     
     
         7 . The method of  claim 1 , further comprising:
 reconstructing a scene of the environment from the sensor data by generating the scene from a single camera capture over time as the vehicle navigates the environment.   
     
     
         8 . The method of  claim 1 , wherein providing the vehicle operational instructions comprises, by an application module executing on a vehicle:
 reducing a speed setpoint of the vehicle and increasing a minimum following distance on a road segment identified as having a higher likelihood of a scenario, or   routing the vehicle to avoid the road segment whose exposure rate for a scenario exceeds a threshold.   
     
     
         9 . The method of  claim 1 , wherein the environment comprises at least one of a geographic location, a geographic region, a city, or a state, and
 computing the frequencies of scenarios comprises:   generating respective histograms for the scenarios for each such environment and determining exposure rates of the scenarios for each such environment.   
     
     
         10 . The method of  claim 1 , wherein providing the vehicle operational instructions based on the frequencies or the exposure rates comprises:
 selecting a route that avoids the road segment whose exposure rate for a scenario exceeds a threshold and causing the vehicle to follow the selected route,   reducing a speed setpoint on a road segment identified as having a higher likelihood of a scenario, or   increasing a minimum following distance on a road segment identified as having a higher likelihood of a scenario.   
     
     
         11 . A system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
 receiving sensor data captured by at least one sensor of a vehicle while the vehicle navigates an environment over a period of time comprising a plurality of time intervals;   for each time interval in the plurality of time intervals:
 determining information describing one or more agents in the environment during the time interval based at least in part on the sensor data; and 
 generating a schema-based encoding based on the determined information and a scenario schema, wherein the scenario schema comprises a set of elements including an agent element and an action element; 
   generating a time-based representation of the environment by associating the schema-based encodings corresponding to the plurality of time intervals in temporal order with a point-in-time encoding of the environment generated for a point in time;   computing, from a plurality of the time-based representations, frequencies of scenarios experienced by vehicles while navigating the environment and determining exposure rates of the scenarios based on the frequencies; and   providing, based on the frequencies or the exposure rates, vehicle operational instructions to avoid a road segment associated with a likelihood of encountering a scenario determined to be risky or modify operation to reduce risk.   
     
     
         12 . The system of  claim 11 , wherein the scenario schema further includes roadway attributes comprising at least one of roadway type, speed limits, number of lanes, locations of intersections, merging lanes, traffic signals and states, street signs, curbs, presence of bicycle lanes, presence of crosswalks, or whether a road segment is in a residential, school, business, mixed-use, high-density, or rural zone. 
     
     
         13 . The system of  claim 11 , wherein the scenario schema further includes contextual information comprising a calendar date, a day of week, a time of day, and weather conditions encountered while the vehicle navigates the environment. 
     
     
         14 . The system of  claim 11 , wherein determining the information describing the one or more agents in the environment during the time interval comprises:
 determining action or motion information comprising at least one of a velocity, a direction of travel, distances between the one or more agents, locations of the one or more agents relative to the vehicle, or locations of the one or more agents relative to other agents.   
     
     
         15 . The system of  claim 11 , wherein providing the vehicle operational instructions based on the frequencies or the exposure rates comprises:
 selecting a route that avoids the road segment whose exposure rate for a scenario exceeds a threshold and causing the vehicle to follow the selected route,   reducing a speed setpoint on a road segment identified as having a higher likelihood of a scenario, or   increasing a minimum following distance on a road segment identified as having a higher likelihood of a scenario.   
     
     
         16 . A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform operations comprising:
 receiving sensor data captured by at least one sensor of a vehicle while the vehicle navigates an environment over a period of time comprising a plurality of time intervals;   for each time interval in the plurality of time intervals:
 determining information describing one or more agents in the environment during the time interval based at least in part on the sensor data; and 
 generating a schema-based encoding based on the determined information and a scenario schema, wherein the scenario schema comprises a set of elements including an agent element and an action element; 
   generating a time-based representation of the environment by associating the schema-based encodings corresponding to the plurality of time intervals in temporal order with a point-in-time encoding of the environment generated for a point in time;   computing, from a plurality of the time-based representations, frequencies of scenarios experienced by vehicles while navigating the environment and determining exposure rates of the scenarios based on the frequencies; and   providing, based on the frequencies or the exposure rates, vehicle operational instructions to avoid a road segment associated with a likelihood of encountering a scenario determined to be risky or modify operation to reduce risk.   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 16 , wherein the scenario schema further includes roadway attributes comprising at least one of roadway type, speed limits, number of lanes, locations of intersections, merging lanes, traffic signals and states, street signs, curbs, presence of bicycle lanes, presence of crosswalks, or whether a road segment is in a residential, school, business, mixed-use, high-density, or rural zone. 
     
     
         18 . The non-transitory computer-readable storage medium of  claim 16 , wherein determining the information describing the one or more agents in the environment during the time interval comprises:
 determining action or motion information comprising at least one of a velocity, a direction of travel, distances between the one or more agents, locations of the one or more agents relative to the vehicle, or locations of the one or more agents relative to other agents.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 16 , wherein providing the vehicle operational instructions based on the frequencies or the exposure rates comprises:
 selecting a route that avoids the road segment whose exposure rate for a scenario exceeds a threshold and causing the vehicle to follow the selected route,   reducing a speed setpoint on a road segment identified as having a higher likelihood of a scenario, or   increasing a minimum following distance on a road segment identified as having a higher likelihood of a scenario.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 16 , wherein the scenario schema further includes contextual information comprising a calendar date, a day of week, a time of day, and weather conditions encountered while the vehicle navigates the environment.