Method and system for constructing vector databases used for converting free text queries to cyber language queries

Abstract

A system and method for querying data sources for cybersecurity analysis is presented. The system and method include: receiving security logs from at least one data source, wherein the security logs lack pre-defined schema; generating a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields; embedding field vectors, wherein each field vector is a vector representation of a value of each respective field; embedding value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field; and generating a query in a cyber language query, using an AI system, for execution on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.

What is claimed is: 
     
         1 . A method for querying data sources for cybersecurity analysis, comprising:
 receiving, by a device, security logs from at least one data source, wherein the security logs lack pre-defined schema;   generating, using an artificial intelligence (AI) embedding system, a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields;   embedding, using the AI embedding system, field vectors, wherein each field vector is a vector representation of a value of each respective field;   embedding, using the AI embedding system, value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field;   generating a query in a cyber language query, using an AI system; and   executing the generated query on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.   
     
     
         2 . The method of  claim 1 , further comprising:
 generating pairs of NLQs and cyber language queries, wherein the pairs are potentially valid matches between NLQs and the cyber language queries;   validating that the generated cyber language queries are properly matched to the generated NLQs; and   embedding vectors of each of the pairs.   
     
     
         3 . The method of  claim 1 , wherein generating the cyber language query, using an AI system further comprises:
 identifying fields that are relevant to a natural language query (NLQ) based on a field vector similarity search using the embedded field vectors;   computing values for the identified fields based on actual values that are relevant to the identified fields and the NLQ based on a value vector similarity search using the embedded value vectors;   refining the generated schema based on the identified fields and the computed values;   generating, based on at least the refined schema, a prompt for a Large Language Model (LLM), wherein the LLM is executed by an AI-based query generation system; and   feeding the prompt to the LLM, wherein the LLM executes the prompt using the AI-based query generation system, to convert the NLQ into the cyber language query.   
     
     
         4 . The method of  claim 3 , wherein the field vector similarity search further comprises:
 comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded field vector;   identifying at least one field for which the embedded field vector is below a semantic distance threshold from the feature vector representing the NLQ; and   validating the identified at least one field against ground-truth pairs of natural language field descriptions and fields searchable in the cyber language for at least one target data source.   
     
     
         5 . The method of  claim 3 , wherein the value vector similarity search further comprises:
 comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded value vector;   identifying at least one value for which the embedded value vector is below a semantic distance threshold from the value vector representing the NLQ; and   validating the at least one value against ground-truth pairs of natural language value descriptions and actual values in the at least one target data source.   
     
     
         6 . The method of  claim 3 , wherein refining the schema of the logs based on the identified fields and computed values further comprises:
 narrowing the schema to the most pertinent elements of the schema based on the identified fields and computed values.   
     
     
         7 . The method of  claim 1 , further comprising:
 generating, using the AI embedding system, the natural language descriptions of the received fields, wherein the AI embedding system is configured to gather relevant information on the fields from official documentation and other available data sources to generate the natural language descriptions.   
     
     
         8 . The method of  claim 1 , further comprising:
 validating syntactical correctness of the generated cyber language query.   
     
     
         9 . The method of  claim 1 , wherein the cyber language query is a read-only request to process data stored hierarchically in databases, tables, and columns. 
     
     
         10 . The method of  claim 1 , wherein the cyber language is at least Kusto Query Language. 
     
     
         11 . A non-transitory computer-readable medium storing a set of instructions for querying data sources for cybersecurity analysis, the set of instructions comprising:
 one or more instructions that, when executed by one or more processors of a device, cause the device to:
 receive, by a device, security logs from at least one data source, wherein the security logs lack pre-defined schema; 
 generate, using an artificial intelligence (AI) embedding system, a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields; 
 embed, using the AI embedding system, field vectors, wherein each field vector is a vector representation of a value of each respective field; 
 embed, using the AI embedding system, value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field; 
 generate a query in a cyber language query, using an AI system and 
   execute the query on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.   
     
     
         12 . A system for querying data sources for cybersecurity analysis comprising:
 one or more processors; and   a memory including instructions configured to:
 receive, by a device, security logs from at least one data source, wherein the security logs lack pre-defined schema; 
 generate, using an artificial intelligence (AI) embedding system, a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields; 
 embed, using the AI embedding system, field vectors, wherein each field vector is a vector representation of a value of each respective field; 
 embed, using the AI embedding system, value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field; and 
 generate a query in a cyber language query, using an AI system; and 
 execute the query on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors. 
   
     
     
         13 . The system of  claim 12 , further comprising:
 generating pairs of NLQs and cyber language queries, wherein the pairs are potentially valid matches between NLQs and the cyber language queries   validating that the generated cyber language queries are properly matched to the generated NLQs; and   embedding vectors of each of the pairs.   
     
     
         14 . The system of  claim 12 , wherein generating the cyber language query, using an AI system further comprises:
 identifying fields that are relevant to a natural language query (NLQ) based on a field vector similarity search using the embedded field vectors   computing values for the identified fields based on actual values that are relevant to the identified fields and the NLQ based on a value vector similarity search using the embedded value vectors   refining the generated schema based on the identified fields and the computed values   generating, based on at least the refined schema, a prompt for a Large Language Model (LLM), wherein the LLM is executed by an AI-based query generation system; and   feeding the prompt to the LLM, wherein the LLM executes the prompt using the AI-based query generation system, to convert the NLQ into the cyber language query.   
     
     
         15 . The system of  claim 14 , wherein the field vector similarity search further comprises:
 comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded field vector   identifying at least one field for which the embedded field vector is below a semantic distance threshold from the feature vector representing the NLQ; and   validating the identified at least one field against ground-truth pairs of natural language field descriptions and fields searchable in the cyber language for at least one target data source.   
     
     
         16 . The system of  claim 14 , wherein the value vector similarity search further comprises:
 comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded value vector   identifying at least one value for which the embedded value vector is below a semantic distance threshold from the value vector representing the NLQ; and   validating the at least one value against ground-truth pairs of natural language value descriptions and actual values in the at least one target data source.   
     
     
         17 . The system of  claim 14 , wherein refining the schema of the logs based on the identified fields and computed values further comprises:
 narrowing the schema to the most pertinent elements of the schema based on the identified fields and computed values.   
     
     
         18 . The system of  claim 12 , further comprising:
 generating, using the AI embedding system, the natural language descriptions of the received fields, wherein the AI embedding system is configured to gather relevant information on the fields from official documentation and other available data sources to generate the natural language descriptions.   
     
     
         19 . The system of  claim 12 , further comprising:
 validating syntactical correctness of the generated cyber language query.   
     
     
         20 . The system of  claim 12 , wherein the cyber language query is a read-only request to process data stored hierarchically in databases, tables, and columns. 
     
     
         21 . The system of  claim 12 , wherein the cyber language is at least Kusto Query Language.