Systems and methods for trained embedding mappings for improved retrieval augmented generation

Abstract

Systems and methods for implementing trained embedding mappings for improved retrieval augmented generation are disclosed. A system can maintain a dataset comprising a first set of embeddings corresponding to a first embeddings space and stored in association with a set of query results for the first set of embeddings. The set of query results can correspond to a second embeddings space. The system can train a transformation data structure using the first set of embeddings and the set of query results. The transformation data structure can be used to transform the first set of embeddings to the second embeddings space. The system can execute a search operation for the second embeddings space by applying the transformation data structure to a second set of embeddings corresponding to the first embeddings space.

What is claimed is: 
     
       1. A system, comprising:
 one or more processors coupled to non-transitory memory, the one or more processors configured to:
 store a dataset comprising a first set of embeddings corresponding to a first embeddings space and stored in association with a set of query results for the first set of embeddings, the set of query results corresponding to a second embeddings space; 
 train a transformation data structure using the first set of embeddings and the set of query results, the transformation data structure to transform the first set of embeddings to the second embeddings space, wherein the first embeddings space is different from the second embeddings space; and 
 execute a search operation for the second embeddings space by applying the transformation data structure to a second set of embeddings corresponding to the first embeddings space. 
 
 
     
     
       2. The system of  claim 1 , wherein the dataset further comprises a corpus of text data corresponding to the second embeddings space. 
     
     
       3. The system of  claim 1 , wherein the one or more processors are further configured to execute the search operation over a corpus of embeddings corresponding to the second embeddings space. 
     
     
       4. The system of  claim 3 , wherein the one or more processors are further configured to:
 generate a set of transformed embeddings based on an intermediate transformation data structure and the first set of embeddings; and 
 determine an error based on the set of transformed embeddings and the set of query results. 
 
     
     
       5. The system of  claim 4 , wherein the one or more processors are further configured to update the intermediate transformation data structure according to the error to generate the transformation data structure. 
     
     
       6. The system of  claim 4 , wherein the one or more processors are further configured to determine the error based on a similarity search between the set of transformed embeddings and the set of query results. 
     
     
       7. The system of  claim 1 , wherein the one or more processors are further configured to:
 receive a first query specifying the second set of embeddings; and 
 execute the search operation in response to the first query. 
 
     
     
       8. The system of  claim 7 , wherein the one or more processors are further configured to execute the search operation during training of the transformation data structure in response to the first query. 
     
     
       9. A method, comprising:
 storing, by one or more processors coupled to non-transitory memory, a dataset comprising a first set of embeddings corresponding to a first embeddings space and stored in association with a set of query results for the first set of embeddings, the set of query results corresponding to a second embeddings space; 
 training, by the one or more processors, a transformation data structure using the first set of embeddings and the set of query results, the transformation data structure to transform the first set of embeddings to the second embeddings space, wherein the first embeddings space is different from the second embeddings space; and 
 executing, by the one or more processors, a search operation for the second embeddings space by applying the transformation data structure to a second set of embeddings corresponding to the first embeddings space. 
 
     
     
       10. The method of  claim 9 , wherein the dataset further comprises a corpus of text data corresponding to the second embeddings space. 
     
     
       11. The method of  claim 9 , further comprising executing, by the one or more processors, the search operation over a corpus of embeddings corresponding to the second embeddings space. 
     
     
       12. The method of  claim 11 , wherein training the transformation data structure comprises:
 generating, by the one or more processors, a set of transformed embeddings based on an intermediate transformation data structure and the first set of embeddings; and 
 determining, by the one or more processors, an error based on the set of transformed embeddings and the set of query results. 
 
     
     
       13. The method of  claim 12 , further comprising updating, by the one or more processors, the intermediate transformation data structure according to the error to generate the transformation data structure. 
     
     
       14. The method of  claim 12 , further comprising determining, by the one or more processors, the error based on a similarity search between the set of transformed embeddings and the set of query results. 
     
     
       15. The method of  claim 9 , further comprising:
 receiving, by the one or more processors, a first query specifying the second set of embeddings; and 
 executing, by the one or more processors, the search operation in response to the first query. 
 
     
     
       16. The method of  claim 15 , further comprising executing, by the one or more processors, the search operation during training of the transformation data structure in response to the first query. 
     
     
       17. A system, comprising:
 one or more processors coupled to non-transitory memory, the one or more processors configured to:
 store a dataset comprising a first set of embeddings corresponding to a first embeddings space and stored in association with a set of query results for the first set of embeddings, the set of query results corresponding to a second embeddings space; 
 train a transformation data structure using the first set of embeddings and the set of query results, the transformation data structure to transform the first set of embeddings to the second embeddings space, wherein the second embeddings space is a subset of the first embeddings space; and 
 execute a search operation for the second embeddings space by applying the transformation data structure to a second set of embeddings corresponding to the first embeddings space. 
 
 
     
     
       18. A method, comprising:
 storing, by one or more processors coupled to non-transitory memory, a dataset comprising a first set of embeddings corresponding to a first embeddings space and stored in association with a set of query results for the first set of embeddings, the set of query results corresponding to a second embeddings space; 
 training, by the one or more processors, a transformation data structure using the first set of embeddings and the set of query results, the transformation data structure to transform the first set of embeddings to the second embeddings space, wherein the second embeddings space is a subset of the first embeddings space; and 
 executing, by the one or more processors, a search operation for the second embeddings space by applying the transformation data structure to a second set of embeddings corresponding to the first embeddings space.