Flight Software Testing Using Actual Flight Data

Abstract

Example embodiments may include determining test flight data based on actual flight data that has been captured by a sensor of an aerial vehicle during a previous flight performed by the aerial vehicle in a physical environment. The test flight data may be processed using a software component that forms part of an aerial vehicle control system. An observed performance of the software component may be determined based on processing the test flight data using the software component. A performance metric may be determined for the software component based on comparing (i) the observed performance of the software component to (ii) an expected performance of the software component. The performance metric may be output.

What is claimed is: 
     
         1 . A system comprising:
 a processor; and   a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
 determining test flight data based on actual flight data that has been captured by a sensor of an aerial vehicle during a previous flight performed by the aerial vehicle in a physical environment; 
 processing the test flight data using a software component that forms part of an aerial vehicle control system; 
 determining an observed performance of the software component based on processing the test flight data using the software component; 
 determining a performance metric for the software component based on comparing (i) the observed performance of the software component to (ii) an expected performance of the software component; and 
 outputting the performance metric. 
   
     
     
         2 . The system of  claim 1 , wherein determining the test flight data comprises:
 determining augmented flight data by modifying the actual flight data to represent a simulated environmental condition that was not present in the physical environment during the previous flight.   
     
     
         3 . The system of  claim 2 , wherein determining the augmented flight data comprises:
 processing the actual flight data using a machine learning model configured to modify the actual flight data to represent the simulated environmental condition; and   generating the augmented flight data based on processing the actual flight data using the machine learning model.   
     
     
         4 . The system of  claim 3 , wherein the machine learning model is configured to modify the actual flight data based on a textual prompt that describes the simulated environmental condition, and wherein processing the actual flight data using the machine learning model comprises:
 processing, using the machine learning model, the actual flight data and the textual prompt that describes the simulated environmental condition.   
     
     
         5 . The system of  claim 2 , determining the augmented flight data comprises:
 adding to the actual flight data a distortion expected to be caused by the simulated environmental condition.   
     
     
         6 . The system of  claim 1 , wherein the test flight data comprises the actual flight data. 
     
     
         7 . The system of  claim 1 , wherein the actual flight data represents effects on the aerial vehicle of actual environmental conditions that were present in the physical environment during the previous flight performed by the aerial vehicle. 
     
     
         8 . The system of  claim 7 , wherein the actual environmental conditions represent at least one of: weather conditions, geolocation of the aerial vehicle, or air traffic. 
     
     
         9 . The system of  claim 1 , further comprising a physical test sensor configured for providing test sensor data to the aerial vehicle control system, wherein determining the test flight data comprises:
 providing the actual flight data as input to the physical test sensor; and   generating the test sensor data using the physical test sensor based on processing the actual flight data using the physical test sensor, wherein the test flight data comprises the test sensor data.   
     
     
         10 . The system of  claim 9 , wherein:
 the sensor comprises a camera,   the test sensor comprises a test camera,   the actual flight data comprises video data captured by the camera of the aerial vehicle during the previous flight,   providing the actual flight data as input to the physical test sensor comprises displaying the video data using a display configured for viewing by the test camera, and   generating the test sensor data comprises generating, using the test camera, test video data by capturing the video as displayed using the display.   
     
     
         11 . The system of  claim 1 , further comprising a sensor emulator configured for providing emulated sensor data to the aerial vehicle control system, wherein determining the test flight data comprises:
 providing the actual flight data as input to the sensor emulator; and   generating the emulated sensor data using the sensor emulator based on processing the actual flight data using the sensor emulator, wherein the test flight data comprises the emulated sensor data.   
     
     
         12 . The system of  claim 1 , wherein the software component forms part of an updated version of the aerial vehicle control system, and wherein the method further comprises:
 determining the expected performance of the software component by processing the test flight data using a prior version of the software component that forms part of a prior version of the aerial vehicle control system.   
     
     
         13 . The system of  claim 1 , wherein the expected performance of the software component corresponds to an actual performance of the aerial vehicle during the previous flight. 
     
     
         14 . The system of  claim 1 , wherein processing the test flight data comprises executing a software performance test of the software component, wherein determining the observed performance of the software component comprises determining an output of the software performance test based on processing the test flight data using the software component, and wherein the expected performance of the software component represents an acceptable range of outputs of the software performance test. 
     
     
         15 . The system of  claim 1 , wherein processing the test flight data using the software component comprises:
 obtaining, from a test sensor of a test aerial vehicle controlled by the aerial vehicle control system, sensor data representing a test environment, wherein the test sensor is of a same type as the sensor of the aerial vehicle;   replacing the sensor data with the test flight data; and   controlling operation of the test aerial vehicle in the test environment using the software component based on the test flight data and independently of the sensor data.   
     
     
         16 . The system of  claim 1 , wherein outputting the performance metric comprises one or more of acceptance of the software component, rejection of the software component, or flagging the software component for further evaluation. 
     
     
         17 . A computer-implemented method comprising:
 determining test flight data based on actual flight data that has been captured by a sensor of an aerial vehicle during a previous flight performed by the aerial vehicle in a physical environment;   processing the test flight data using a software component that forms part of an aerial vehicle control system;   determining an observed performance of the software component based on processing the test flight data using the software component;   determining a performance metric for the software component based on comparing (i) the observed performance of the software component to (ii) an expected performance of the software component; and   outputting the performance metric.   
     
     
         18 . The computer-implemented method  17 , wherein determining the test flight data comprises:
 determining augmented flight data by modifying the actual flight data to represent a simulated environmental condition that was not present in the physical environment during the previous flight.   
     
     
         19 . The computer-implemented method  17 , wherein determining the augmented flight data comprises:
 processing the actual flight data using a machine learning model configured to modify the actual flight data to represent the simulated environmental condition; and   generating the augmented flight data based on processing the actual flight data using the machine learning model.   
     
     
         20 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing system, cause the computing system to perform operations comprising:
 determining test flight data based on actual flight data that has been captured by a sensor of an aerial vehicle during a previous flight performed by the aerial vehicle in a physical environment;   processing the test flight data using a software component that forms part of an aerial vehicle control system;   determining an observed performance of the software component based on processing the test flight data using the software component;   determining a performance metric for the software component based on comparing (i) the observed performance of the software component to (ii) an expected performance of the software component; and   outputting the performance metric.