Automated data ingestion using an autoencoder

Abstract

Systems, methods, apparatuses, and computer program products for processing data using an autoencoder. In one example, the autoencoder may receive streaming data comprising numeric values during a first time interval. The autoencoder may determine, during the first time interval, a maximum value and a minimum value of a first subset of the numeric values. The autoencoder may then process, during the first time interval, a second subset of the numeric values based on the determined maximum and minimum values.

What is claimed is: 
     
         1 . A system, comprising:
 a processor circuit; and   a memory storing instructions which when executed by the processor circuit, cause the processor circuit to:
 receive, by an autoencoder during a first time interval, streaming data comprising numeric values; 
 determine, by the autoencoder during the first time interval, a maximum value and a minimum value of a first subset of the numeric values; and 
 process, by the autoencoder during the first time interval, a second subset of the numeric values based on the determined maximum and minimum values. 
   
     
     
         2 . The system of  claim 1 , wherein processing the second subset of the numeric values comprises normalizing the second subset of the numeric values to be within the determined maximum and minimum values. 
     
     
         3 . The system of  claim 1 , wherein processing the second subset of the numeric values comprises filtering a numeric value from the second subset that is not within the determined maximum and minimum values, wherein filtering the numeric value from the second subset removes the filtered numeric value from the second subset. 
     
     
         4 . The system of  claim 1 , wherein processing the second subset of the numeric values comprises converting a numeric value from the second subset that is not within the determined maximum and minimum values from a first data type to a second data type, wherein the converted numeric value of the second data type is within the determined maximum and minimum values. 
     
     
         5 . The system of  claim 1 , wherein the autoencoder is trained based on a training dataset generated by a computing model, wherein the autoencoder is trained to process numeric values according to a predefined operation, wherein the autoencoder comprises a latent vector. 
     
     
         6 . The system of  claim 5 , wherein an accuracy of the trained autoencoder exceeds a threshold accuracy. 
     
     
         7 . The system of  claim 1 , the memory storing instructions which when executed by the processor circuit, cause the processor circuit to:
 provide, by the autoencoder, the processed second subset of the numeric values to a processing pipeline.   
     
     
         8 . A non-transitory computer-readable storage medium storing instructions that when executed by a processor cause the processor to:
 receive, by an autoencoder during a first time interval, streaming data comprising numeric values;   determine, by the autoencoder during the first time interval, a maximum value and a minimum value of a first subset of the numeric values; and   process, by the autoencoder during the first time interval, a second subset of the numeric values based on the determined maximum and minimum values.   
     
     
         9 . The medium of  claim 8 , wherein processing the second subset of the numeric values comprises normalizing the second subset of the numeric values to be within the determined maximum and minimum values. 
     
     
         10 . The medium of  claim 8 , wherein processing the second subset of the numeric values comprises filtering a numeric value from the second subset that is not within the determined maximum and minimum values, wherein filtering the numeric value from the second subset removes the filtered numeric value from the second subset. 
     
     
         11 . The medium of  claim 8 , wherein processing the second subset of the numeric values comprises converting a numeric value from the second subset that is not within the determined maximum and minimum values from a first data type to a second data type, wherein the converted numeric value of the second data type is within the determined maximum and minimum values. 
     
     
         12 . The medium of  claim 8 , wherein the autoencoder is trained based on a training dataset generated by a computing model, wherein the autoencoder is trained to process numeric values according to a predefined operation, wherein the autoencoder comprises a latent vector. 
     
     
         13 . The medium of  claim 12 , wherein an accuracy of the trained autoencoder exceeds a threshold accuracy. 
     
     
         14 . The medium of  claim 8 , storing instructions which when executed by the processor, cause the processor to:
 provide, by the autoencoder, the processed second subset of the numeric values to a processing pipeline.   
     
     
         15 . A method, comprising:
 receiving, by an autoencoder executing on a computer processor, streaming data comprising numeric values during a first time interval;   determining, by the autoencoder during the first time interval, a maximum value and a minimum value of a first subset of the numeric values; and   processing, by the autoencoder during the first time interval, a second subset of the numeric values based on the determined maximum and minimum values.   
     
     
         16 . The method of  claim 15 , wherein processing the second subset of the numeric values comprises normalizing the second subset of the numeric values to be within the determined maximum and minimum values. 
     
     
         17 . The method of  claim 15 , wherein processing the second subset of the numeric values comprises filtering a numeric value from the second subset that is not within the determined maximum and minimum values, wherein filtering the numeric value from the second subset removes the filtered numeric value from the second subset. 
     
     
         18 . The method of  claim 15 , wherein processing the second subset of the numeric values comprises converting a numeric value from the second subset that is not within the determined maximum and minimum values from a first data type to a second data type, wherein the converted numeric value of the second data type is within the determined maximum and minimum values. 
     
     
         19 . The method of  claim 15 , wherein the autoencoder is trained based on a training dataset generated by a computing model, wherein the autoencoder is trained to process numeric values according to a predefined operation, wherein the autoencoder comprises a latent vector, wherein an accuracy of the trained autoencoder exceeds a threshold accuracy. 
     
     
         20 . The method of  claim 15 , further comprising:
 providing, by the autoencoder, the processed second subset of the numeric values to a processing pipeline.