Techniques for supporting model registries and model ensemble determination

Abstract

Uploaded data including an artificial intelligence (AI) model and metadata associated with the AI model may be received. A task corresponding to the AI model may be determined. A processing device may confirm the metadata includes samples predictions generated based on benchmark data associated with the task corresponding to the AI model. The AI model and the sample predictions may be stored in an AI model registry in response to confirming the metadata includes the sample predictions generated based on the benchmark data associated with the task corresponding to the AI model.

What is claimed is: 
     
         1 . A method comprising:
 receiving uploaded data comprising an artificial intelligence (AI) model and metadata associated with the AI model;   determining a task corresponding to the AI model;   confirming, by a processing device, the metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model; and   storing the AI model and the sample predictions in an AI model registry in response to confirming the metadata includes the sample predictions generated based on the benchmark data associated with the task corresponding to the AI model.   
     
     
         2 . The method of  claim 1 , further comprising:
 identifying a plurality of AI models stored in the AI model registry, the plurality of AI models associated with the task, and the plurality of AI models including the AI model; and   analyzing the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein the suggested ensemble of tasks includes one or more of the plurality of models.   
     
     
         3 . The method of  claim 2 , further comprising analyzing the sample predictions and additional metadata associated with each of the plurality of models stored in the model registry to determine the suggested ensemble of models for performing the task. 
     
     
         4 . The method of  claim 3 , wherein the additional metadata includes at least one of compute resource requirements for each of the plurality of models, a video random access memory (VRAM) requirement for each of the plurality of models, or compute time for each of the plurality of models. 
     
     
         5 . The method of  claim 1 , wherein the AI model registry includes a set of groupings with each grouping in the set of groupings associated with one of a plurality of tasks, and the method further comprising storing the AI model and the sample predictions in the grouping associated with the task. 
     
     
         6 . The method of  claim 1 , further comprising:
 receiving second uploaded data comprising a second AI model;   determining a second task corresponding to the second AI model; and   determining the second uploaded data fails to include second sample predictions generated based on second benchmark data associated with the second task corresponding to the second AI model.   
     
     
         7 . The method of  claim 6 , further comprising:
 generating the second sample predictions based on the second benchmark data associated with the second task corresponding to the second AI model; and   storing the second AI model and the second sample predictions in the AI model registry.   
     
     
         8 . The method of  claim 6 , further comprising:
 requesting, from a client device, the second sample predictions generated based on the second benchmark data associated with the second task in response to determining the second uploaded data fails to include the second sample predictions generated based on the second benchmark data;   receiving, from the client device, third uploaded data;   confirming the third uploaded data includes the second sample predictions generated based on the second benchmark data associated with the second task; and   storing the second AI model and the second sample predictions in the AI model registry in response to confirming the third uploaded data includes metadata includes the second sample predictions generated based on the second benchmark data associated with the second task.   
     
     
         9 . The method of  claim 8 , further comprising providing the second benchmark data associated with the second task to the client device. 
     
     
         10 . A system comprising:
 a memory; and   a processing device, operatively coupled to the memory, to:
 receive uploaded data comprising an artificial intelligence (AI) model and metadata associated with the AI model; 
 determine a task corresponding to the AI model; 
 confirm, by a processing device, the metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model; and 
 store the AI model and the sample predictions in an AI model registry in response to confirming the metadata includes the sample predictions generated based on the benchmark data associated with the task corresponding to the AI model. 
   
     
     
         11 . The system of  claim 10 , wherein the processing device is further to:
 identify a plurality of AI models stored in the AI model registry, the plurality of AI models associated with the task, and the plurality of AI models including the AI model; and   analyze the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein the suggested ensemble of tasks includes one or more of the plurality of models.   
     
     
         12 . The system of  claim 11 , wherein the processing device is further to analyze the sample predictions and additional metadata associated with each of the plurality of models stored in the model registry to determine the suggested ensemble of models for performing the task. 
     
     
         13 . The system of  claim 12 , wherein the additional metadata includes at least one of compute resource requirements for each of the plurality of models, a video random access memory (VRAM) requirement for each of the plurality of models, or compute time for each of the plurality of models. 
     
     
         14 . The system of  claim 10 , wherein the AI model registry includes a set of groupings with each grouping in the set of groupings associated with one of a plurality of tasks, and the processing device is further to store the AI model and the sample predictions in the grouping associated with the task. 
     
     
         15 . The system of  claim 10 , wherein the processing device is further to:
 receive second uploaded data comprising a second AI model;   determine a second task corresponding to the second AI model; and   determine the second uploaded data fails to include second sample predictions generated based on second benchmark data associated with the second task corresponding to the second AI model.   
     
     
         16 . The system of  claim 15 , wherein the processing device is further to:
 generate the second sample predictions based on the second benchmark data associated with the second task corresponding to the second AI model; and   store the second AI model and the second sample predictions in the AI model registry.   
     
     
         17 . The system of  claim 15 , wherein the processing device is further to:
 request, from a client device, the second sample predictions generated based on the second benchmark data associated with the second task in response to determining the second uploaded data fails to include the second sample predictions generated based on the second benchmark data;   receive, from the client device, third uploaded data;   confirm the third uploaded data includes the second sample predictions generated based on the second benchmark data associated with the second task; and   store the second AI model and the second sample predictions in the AI model registry in response to confirming the third uploaded data includes metadata includes the second sample predictions generated based on the second benchmark data associated with the second task.   
     
     
         18 . The system of  claim 17 , wherein the processing device is further to provide the second benchmark data associated with the second task to the client device. 
     
     
         19 . A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:
 receive uploaded data comprising an artificial intelligence (AI) model and metadata associated with the AI model;   determine a task corresponding to the AI model;   confirm, by the processing device, the metadata includes sample predictions generated based on benchmark data associated with the task corresponding to the AI model; and   store the AI model and the sample predictions in an AI model registry in response to confirming the metadata includes the sample predictions generated based on the benchmark data associated with the task corresponding to the AI model.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 19 , wherein the processing device is further to:
 identify a plurality of AI models stored in the AI model registry, the plurality of AI models associated with the task, and the plurality of AI models including the AI model; and   analyze the sample predictions for each of the plurality of models associated with the task to determine a suggested ensemble of models for performing the task, wherein the suggested ensemble of tasks includes one or more of the plurality of models.