Systems, Methods, and Devices for Synchronization of Vehicle Data with Recorded Audio

Abstract

A method for post-processing to synchronize audio data with vehicle data includes generating an artificial sound data based on time-series vehicle data. The method includes determining an offset that maximizes cross-correlation between the artificial sound data and recorded audio data. The method also includes shifting one or more of the time-series data and the recorded audio data relative to each other in time based on the offset. The shift may be used to generate or render a synchronized set of time-series data and recorded audio data.

1 . A method comprising:
 generating proxy sound data based on timeseries vehicle data;   determining recorded audio data;   determining an offset that maximizes cross-correlation between the proxy sound data and the recorded audio data; and   shifting one or more of the timeseries vehicle data or the recorded audio data in time based on the offset to generate a synchronized set of timeseries vehicle data and recorded audio data.   
     
     
         2 . The method of  claim 1 , wherein generating the proxy sound data comprises modifying an amplitude of a sound based on a value of a portion of the timeseries vehicle data. 
     
     
         3 . The method of  claim 2 , wherein modifying the amplitude of the sound comprises modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise. 
     
     
         4 . The method of  claim 1 , wherein generating the proxy sound data comprises generating a proxy sound to simulate one or more of:
 a proxy engine sound;   a proxy wind sound; or   a proxy tire sound.   
     
     
         5 . The method of  claim 1 , wherein the timeseries vehicle data comprises data for one or more of an engine speed or a vehicle speed, wherein generating the proxy sound data comprises generating based on one or more of the engine speed or the vehicle speed. 
     
     
         6 . The method of  claim 1 , wherein shifting one or more of the vehicle time series data and the recorded audio data in time based on the offset comprises shifting during rendering of one or more of:
 the recorded audio data; or   image or video data corresponding to the recorded audio data.   
     
     
         7 . The method of  claim 1 , wherein shifting one or more of the vehicle time series data and the recorded audio data in time based on the offset comprises one or more of:
 modifying a time signature of one or more of the timeseries vehicle data and the recorded audio data;   storing the offset with one or more of the timeseries vehicle data and the recorded audio data; and   generating a file that correlates portions of the timeseries vehicle data with a corresponding audio or video data based on the offset, wherein the recorded audio data comprises audio data from the video recording.   
     
     
         8 . The method of  claim 1 , wherein the timeseries vehicle data comprises data produced by one or more vehicle sensors or control systems of a vehicle during a driving period and the recorded audio data comprises audio data recorded during the driving period. 
     
     
         9 . A system comprising:
 a microphone; and   a processor that is configurable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
 generating proxy sound data based on timeseries vehicle data; 
 receiving recoded audio data captured by the microphone; 
 determining an offset that maximizes cross-correlation between the proxy sound data and the recorded audio data; and 
 shifting one or more of the timeseries vehicle data or the recorded audio data in time based on the offset to generate a synchronized set of timeseries vehicle data and recorded audio data. 
   
     
     
         10 . The system of  claim 9 , wherein generating the proxy sound data comprises modifying an amplitude of a sound based on a value of a portion of the timeseries vehicle data. 
     
     
         11 . The system of  claim 10 , wherein modifying the amplitude of the sound comprises modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise. 
     
     
         12 . The system of  claim 9 , wherein generating the proxy sound data comprises one or more of:
 generating a proxy engine sound;   generating a proxy wind sound; or   generating a proxy tire sound.   
     
     
         13 . The system of  claim 9 , wherein the timeseries vehicle data comprises data for one or more of an engine speed and a vehicle speed, and wherein generating the proxy sound data comprises generating based on one or more of the engine speed and the vehicle speed. 
     
     
         14 . The system of  claim 9 , wherein shifting one or more of the vehicle time series data and the recorded audio data in time comprises one or more of:
 shifting during rendering of the recorded audio data; or   shifting during rendering of image or video data corresponding to the recorded audio data.   
     
     
         15 . The system of  claim 9 , wherein shifting one or more of the vehicle time series data and the recorded audio data in time comprises one or more of:
 modifying a time signature of one or more of the timeseries vehicle data and the recorded audio data;   storing the offset with one or more of the timeseries vehicle data and the recorded audio data; or   generating a file that correlates portions of the timeseries vehicle data with a corresponding audio or video data based on the offset, wherein the recorded audio data comprises audio data from the video recording.   
     
     
         16 . Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:
 generate proxy sound data based on timeseries vehicle data produced by one or more vehicle sensors or control systems of a vehicle during a driving period;   determine recorded audio data from video data recorded during the driving period;   determine an offset that maximizes cross-correlation between the proxy sound data and the recorded audio data;   render at least a portion of the video data; and   during rendering a portion of the video data, determine and display a portion of the timeseries vehicle data corresponding to the portion of the video data based on the offset.   
     
     
         17 . The non-transitory computer readable media of  claim 16 , wherein the instructions cause the one or more processors to generate the proxy sound data by modifying an amplitude of a sound over time based on a value of a portion of the timeseries vehicle data over time. 
     
     
         18 . The non-transitory computer readable media of  claim 17 , wherein the instructions cause the one or more processors to modify the amplitude of the sound by modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise. 
     
     
         19 . The non-transitory computer readable media of  claim 16 , wherein the timeseries vehicle data comprises data for one or more of an engine speed and a vehicle speed, wherein the instructions cause the one or more processors to generate the proxy sound data by generating based on one or more of the engine speed and the vehicle speed. 
     
     
         20 . The non-transitory computer readable media of  claim 16 , wherein the instructions cause the one or more processors to determine the offset that maximizes cross-correlation by matching one or more features in the proxy sound data with one or more features in the recorded audio data and determining the offset to match the one or more features in the proxy sound data with the one or more features in the recorded audio data in time.