Cell Detection Studio: a system for the development of Deep Learning Neural Networks Algorithms for cell detection and quantification from Whole Slide Images

Abstract

The invention is made out of methods for the development of Deep Neural Networks for cell detection and quantification in Whole Slide Images (WSI): 1. Method to create generic cell detector that detects the centers and contours of all cells in a WSI. 2. Method to create algorithms to detect cells of specific categories and that can classify between various types of cells of different categories. 3. Method for efficient cell annotation with online learning. 4. Method for efficient cell annotation with active learning. 5. Method for efficient cell annotation with online learning and data balancing. 6. Method for auto annotation of cells 7. Cell Detection Studio: a method to create an AI based system that provides pathologists with a semi-automatic tool to create new algorithms aiming to find cells of specific categories in WSI digitally scanned from histological specimen

What is claimed is: 
     
         1 . A method for generic cell detection in WSI: providing a digital WSI scanned by a digital scanner from a histological specimen-stained slide; detect the centers and contours of cells of all types in the WSI using image processing and deep learning algorithms. 
     
     
         2 . A method to create a detector that can classify between various types of cells of different categories using neural network algorithms. In order to create the neural network algorithm image crops are selected from the WSI that contain cells detected using the method described in  claim 1 . The crop size is set to a constant value or is adjusted to the size of cells in the image crop. An interactive annotation scheme of the sampled image patches with a GUI (graphical user interface) application can be used by human annotators. Each time a single image crop is presented to the annotator, and using a keyboard press or mouse click, or touch screen tap, the annotator chooses one of the possible categories, each containing a specific type of cell or background. 
     
     
         3 . The method of  claim 2  where online learning methodology is used, thus the classification neural network is trained in parallel to the annotation process. The deep learning neural network is continuously trained on an evolving dataset. During the training the number of annotated cells in each cell category constantly changes and this can affect the loss function of the neural network. Thus, the weights of each cell category have to be constantly updated as the database constantly changes during online learning.
 Therefore the loss function is calculated as weighted cross entropy where the weights can be set using methods for class balancing, e.g. median frequency balancing. In addition, during the generation of the algorithm the architecture of the neural network, the number of layers, the number of parameters and the connection between layers may change. 
 
     
     
         4 . A method for selecting the next image patches to be annotated using active learning framework. An active learning algorithm is applied on the cells yet to be annotated. The active learning algorithm ranks each cell. Each rank has a life time of a pre-defined parameter that can be configured. When the life time elapses the rank is then reset to a default value. The active learning algorithm chooses the best images to be annotated in order to make the annotation process most efficient. This ranking can be based on the acquisition function (should be minimized) or using an ensemble of models. If the ensemble of algorithms is in disagreement this means that the algorithm is less confident for that image patch. The ensemble of algorithms can be generated using Bayesian deep learning, where drop out is applied at test time. 
     
     
         5 . The method of  claim 4  where we add data balancing using resampling. Unbalanced data is a common situation where the number of instances of one category is significantly smaller than the number of instances of another category. In order to obtain a robust network there should be enough examples of each category. We therefore add data balancing methodology for effective active learning. We rank the cells inversely proportional to their existence. Once we have enough examples of each category we can move to the usual approach of active learning. We duplicate image patches that belong to the least frequent category. 
     
     
         6 . The method of  claim 4  where we add data balancing using weighting. The weight is inversely proportional to the proportion of the least frequent category. 
     
     
         7 . The method of  claim 4  where we add data balancing using the following weighting: Weight=E*A−B*(N−E)*Pminority where:
 E=Entropy(class proportion) 
 A=Acquisition function as defined in Active Learning. 
 B=parameter 
 N=number of categories 
 Pminority=output of neural network of  claim 2  that gives the probability for the minority category. 
 
     
     
         8 . A method for suggested auto annotation of the image crops that contain different kind of cells. The activation of the auto annotation process can be triggered by one of the following: 1) Time elapsed from beginning of training is higher than a threshold. 2) Classification results of the algorithm are close enough (according to some metric and threshold) to that of a human annotator. 3) Accuracy on a pre-defined validation set is good enough according to a pre-defined metric and threshold. 4) There is large enough number of annotations of cells of all categories of interest. 
     
     
         9 . The method of  claim 8  where we add noise to the auto annotation process so that some of the images will be selected at random so that we do not fall in dead ends. 
     
     
         10 . Cell Detection Studio: an AI based system that provides pathologists with a semi-automatic tool to create new algorithms aiming to find cell of specific categories in WSI digitally scanned from histological specimen; a computer running a dedicated WSI viewer that contains data upload and save utilities, zoom and pan and the ability to quickly navigate to a region of interest. The WSI viewer also offers computer vision, machine learning and deep learning algorithms.
 Specifically, the system provides the methods detailed in  claims 1 - 9  that allows detection of the presence of specific types of cells.   
     
     
         11 . The method of  claim 10 , wherein the system is adapted to the transfer, storage and retrieval of the associated images, and for the generation of reports. 
     
     
         12 . The method of  claim 10  for Quality Assurance (QA) as part of the training process. The user can annotate cells of specific category in a region of interest and apply the algorithm developed on that region of interest. Then the system generates statistical report on the accuracy level achieved. Accuracy measures to be used can be but are not limited to: FA, AUC, confusion matrix, false positives and false negatives. 
     
     
         13 . The method of  claim 10  with the addition of calculating various attributes related to the specific cell categories for which the detection algorithm was created. These features are based on the location and contours of the above mentioned cells: number, density, area, location and perimeter.