Defect detection in lyophilized drug products with convolutional neural networks

Abstract

In one embodiment, a method includes receiving one or more querying images associated with a container of a pharmaceutical product, each of the one or more querying images being based on a particular angle of the container of the pharmaceutical product, calculating one or more confidence scores associated with one or more defect indications, respectively for the container of the pharmaceutical product, by processing the one or more querying images using a target machine-learning model, and determining a defect indication for the container of the pharmaceutical product from the one or more defect indications based on a comparison between the one or more confidence scores and one or more predefined threshold scores, respectively.

What is claimed is: 
     
         1 . A method for determining defect indications for a container of a lyophilized pharmaceutical product, comprising:
 receiving, from a first camera station, one or more first querying images of a container of a lyophilized pharmaceutical product, wherein the first camera station is configured to capture the one or more first querying images of the container from a first set of one or more angles;   receiving, from a second camera station, one or more second querying images of the container, wherein the second camera station is configured to capture the one or more second querying images of the container from a second set of one or more angles;   calculating, for the container of the lyophilized pharmaceutical product, one or more confidence scores associated with one or more defect indications, respectively, by processing the one or more first querying images and the one or more second querying images using a target machine-learning model; and   determining, for the container of the lyophilized pharmaceutical product, a defect indication from the one or more defect indications based on a comparison between the one or more confidence scores and one or more predefined threshold scores, respectively.   
     
     
         2 . The method of  claim 1 , wherein one or more first layers of a plurality of layers of the target machine-learning model are used to process the one or more first querying images, and one or more second layers of the plurality of layers of the target machine-learning model are used to process the one or more second querying images. 
     
     
         3 . The method of  claim 1 , wherein:
 the first camera station comprises one or more first optical camera sensors configured to capture the one or more first querying images of the container; and   the second camera station comprises one or more second optical camera sensors configured to capture the one or more second querying images of the container.   
     
     
         4 . The method of  claim 1 , wherein:
 the first camera station comprises a first gear drive motor configured to rotate the container to capture the one or more first querying images; and   the second camera station comprises a second gear drive motor configured to rotate the container to capture the one or more second querying images.   
     
     
         5 . The method of  claim 1 , further comprising:
 processing, using the target machine-learning model, the one or more first querying images to obtain one or more first feature representations; and   processing, using the target machine-learning model, the one or more second querying images to obtain one or more second feature representations, wherein calculating the one or more confidence scores comprises:
 calculating the one or more confidence scores based on the one or more first feature representations and the one or more second feature representations. 
   
     
     
         6 . The method of  claim 1 , further comprising:
 training the target machine-learning model based on a plurality of training images depicting sample containers of a sample lyophilized pharmaceutical product captured at a plurality of angles, wherein the plurality of training images comprises one or more first training images captured by the first camera station and one or more second training images captured by the second camera station.   
     
     
         7 . The method of  claim 5 , wherein training comprises:
 dividing the plurality of training images into a set of groups of target training images, wherein the set of groups are associated with one or more first optical camera sensors of the first camera station and one or more second optical camera sensors of the second camera station, respectively;   generating and aggregating one or more first groups of feature representations for the one or more first training images to obtain a first aggregated feature representation; and   generating and aggregating one or more second groups of feature representations for the one or more second training images to obtain a second aggregated feature representation, wherein the target machine-learning model is trained based on the first aggregated feature representation and the second aggregated feature representation.   
     
     
         8 . The method of  claim 7 , further comprising:
 integrating, by vectorization, the first aggregated feature representation and the second aggregated feature representation to obtain vectorized features, wherein the target machine-learning model is trained based on the vectorized features.   
     
     
         9 . The method of  claim 7 , wherein the plurality of training images comprise (i) a first plurality of training images depicting sample containers for a sample lyophilized pharmaceutical product captured at a first plurality of angles by the one or more first optical camera sensors of the first camera station and (ii) a second plurality of training images depicting the sample containers for the sample lyophilized pharmaceutical product captured at a second plurality of angles by the one or more second optical camera sensors of the second camera station. 
     
     
         10 . The method of  claim 9 , wherein training the target machine-learning model comprises:
 training a first sub-model associated with the one or more first optical camera sensors of the first camera station based on the one or more first groups of feature representations; and   training a second sub-model associated with the one or more second optical camera sensors of the second camera station one or more second groups of feature representations, wherein the target machine-learning model is based on the first sub-model and the second sub-model.   
     
     
         11 . The method of  claim 10 , further comprising:
 integrating the first sub-model and the second sub-model to generate the target machine-learning model.   
     
     
         12 . The method of  claim 1 , wherein the target machine-learning model is based on a convolutional neural network comprising a plurality of layers. 
     
     
         13 . The method of  claim 1 , wherein the first set of one or more angles and the second set of one or more angles are identical. 
     
     
         14 . The method of  claim 1 , wherein the target machine-learning model comprises a binary classification model or a multi-class classification model, wherein determining the defect indication comprises:
 inputting the one or more confidence score to the binary classification model or the multi-class classification model to determine the defect indication.   
     
     
         15 . The method of  claim 1 , further comprising:
 generating one or more first feature representations of the one or more first querying images by processing the one or more first querying images using a first sub-model of the target machine-learning model; and   generating one or more second feature representations of the one or more second querying images by processing the one or more second querying images using a second sub-model of the target machine-learning model, wherein the one or more confidence scores associated with one or more defect indications are calculated based on the one or more first feature representations and the one or more second feature representations.   
     
     
         16 . The method of  claim 15 , wherein the one or more confidence scores associated with the one or more defect indications are calculated based on one or more of:
 a relationship between feature representations of a plurality of target training images and the one or more defect indications; or   at least one of the one or more first feature representations of the one or more first querying images or the one or more second feature representations of the one or more second querying images.   
     
     
         17 . The method of  claim 1 , wherein each of the one or more predefined threshold scores is determined based on an acceptance quality limit associated with the defect indication. 
     
     
         18 . The method of  claim 1 , further comprising:
 accessing a plurality of target training samples comprising a plurality of target training images associated with containers of the lyophilized pharmaceutical product visible inside the containers in at least one of the plurality of target training images, and a plurality of defect indications associated with the plurality of target training images, respectively, wherein the plurality of target training images comprise one or more first target training images captured by the first camera station and one or more second target training images captured by the second camera station;   selecting a plurality of auxiliary training samples based on the plurality of target training samples comprising a plurality of auxiliary training images and a plurality of content categories associated with the plurality of auxiliary training images, respectively, wherein the plurality of auxiliary training images comprise one or more first auxiliary training images captured by the first camera station and one or more second auxiliary training images captured by the second camera station; and   training the target machine-learning model, the training comprising:
 training an auxiliary machine-learning model based on the plurality of auxiliary training samples; 
 generating feature representations of the plurality of target training images based on the auxiliary machine-learning model; and 
 learning, based on the generated feature representations and the one or more defect indications, a relationship between the feature representations of the plurality of target training images and the one or more defect indications. 
   
     
     
         19 . The method of  claim 18 , wherein the plurality of content categories associated with the plurality of auxiliary training images do not match the one or more defect indications associated with the plurality of target training images, and wherein the plurality of content categories comprises a plurality of handwritten numeric digits. 
     
     
         20 . A system for determining defect indications for a container of a lyophilized pharmaceutical product, comprising:
 a first camera station configured to capture the one or more first querying images of the container from a first set of one or more angles;   a second camera station configured to capture the one or more second querying images of the container from a second set of one or more angles;   a non-transitory memory storing instructions; and   one or more processors configured to execute the instructions cause the one or more processors to:
 receive, from the first camera station, one or more first querying images of a container of a lyophilized pharmaceutical product; 
 receive, from the second camera station, one or more second querying images of the container; 
 calculate, for the container of the lyophilized pharmaceutical product, one or more confidence scores associated with one or more defect indications, respectively, by processing the one or more first querying images and the one or more second querying images using a target machine-learning model; and 
 determine, for the container of the lyophilized pharmaceutical product, a defect indication from the one or more defect indications based on a comparison between the one or more confidence scores and one or more predefined threshold scores, respectively. 
   
     
     
         21 . A non-transitory computer-readable medium storing computer program instructions that, when executed by one or more processors, effectuate operations comprising:
 receiving, from a first camera station, one or more first querying images of a container of a lyophilized pharmaceutical product, wherein the first camera station is configured to capture the one or more first querying images of the container from a first set of one or more angles;   receiving, from a second camera station, one or more second querying images of the container, wherein the second camera station is configured to capture the one or more second querying images of the container from a second set of one or more angles;   calculating, for the container of the lyophilized pharmaceutical product, one or more confidence scores associated with one or more defect indications, respectively, by processing the one or more first querying images and the one or more second querying images using a target machine-learning model; and   determining, for the container of the lyophilized pharmaceutical product, a defect indication from the one or more defect indications based on a comparison between the one or more confidence scores and one or more predefined threshold scores, respectively.