Method and system for cell image segmentation using multi-stage convolutional neural networks

Abstract

An artificial neural network system for image classification, including multiple independent individual convolutional neural networks (CNNs) connected in multiple stages, each CNN configured to process an input image to calculate a pixelwise classification. The output of an earlier stage CNN, which is a class score image having identical height and width as its input image and a depth of N representing the probabilities of each pixel of the input image belonging to each of N classes, is input into the next stage CNN as input image. When training the network system, the first stage CNN is trained using first training images and corresponding label data; then second training images are forward propagated by the trained first stage CNN to generate corresponding class score images, which are used along with label data corresponding to the second training images to train the second stage CNN.

What is claimed is: 
     
         1 . An artificial neural network system implemented on a computer for image classification, comprising:
 a first stage convolutional neural network (CNN), for receiving an input image and classifying each pixel of the input image among N classes, N being a natural number greater than or equal to two, to generate a first stage class score image, the first stage class score image having a height and a width identical to those of the input image and having a depth equal to N, a pixel value of each pixel of the first stage class score image being a vector of size N representing first stage preliminary probabilities of a corresponding pixel in the input image belonging to the corresponding one of N classes; and   a second stage CNN, coupled to the first stage CNN, for receiving the first stage class score image and classifying each pixel of the first stage class score image among N classes, to generate a second stage class score image, the second stage class score image having a height and a width identical to those of the input image and having a depth equal to N, a pixel value of each pixel of the second stage class score image being a vector of size N representing second stage preliminary probabilities of a corresponding pixel in the input image belonging to the corresponding classes,   wherein each of the first stage CNN and the second stage CNN has a plurality of layers of neurons stacked sequentially, including at least a plurality of convolutional layers and a plurality of pooling layers, each convolutional layer performing convolution operations to convolve a number of filters across its previous layer, each pooling layer performing pooling operations on its previous layer.   
     
     
         2 . The artificial neural network system of  claim 1 , which comprises only two CNNs and wherein the second stage preliminary probabilities are final probabilities of the corresponding pixel in the input image belonging to the corresponding classes. 
     
     
         3 . The artificial neural network system of  claim 1 , further comprising a third stage CNN, coupled to the second stage CNN, for receiving the second stage class score image and classifying each pixel of the second stage class score image among N classes, to generate a final class score image, the final class score image having a height and a width identical to those of the input image and having a depth equal to N, a pixel value of each pixel of the final class score image being a vector of size N representing final probabilities of a corresponding pixel in the input image belonging to the corresponding classes. 
     
     
         4 . The artificial neural network system of  claim 1 , wherein the first stage CNN and the second stage CNN have identical numbers and identical arrangements of the convolutional layers and the pooling layers, wherein the corresponding layers of the CNNs have identical sizes and the filters in corresponding convolutional layers of the CNNs have identical sizes. 
     
     
         5 . The artificial neural network system of  claim 1 , wherein the first stage and second stage CNNs are independent of each other. 
     
     
         6 . The artificial neural network system of  claim 1 , wherein the filters in all of the plurality of convolutional layers in the first stage and second stage CNNs have a height of 3 and a width of 3. 
     
     
         7 . The artificial neural network system of  claim 1 , wherein N is equal to three, wherein the three classes include a background class, a foreground class and an edge class. 
     
     
         8 . An image classification method using an artificial neural network system implemented on a computer, comprising:
 providing a first stage convolutional neural network (CNN) and a second stage CNN, each of the first stage CNN and the second stage CNN having a plurality of layers of neurons stacked sequentially, including at least a plurality of convolutional layers and a plurality of pooling layers, each convolutional layer performing convolution operations to convolve a number of filters across its previous layer, each pooling layer performing pooling operations on its previous layer;   inputting an input image into the first stage CNN;   using the first stage CNN to classify each pixel of the input image among N classes, N being a natural number greater than or equal to two, to generate a first stage class score image, the first stage class score image having a height and a width identical to those of the input image and having a depth equal to N, a pixel value of each pixel of the first stage class score image being a vector of size N representing first stage preliminary probabilities of a corresponding pixel in the input image belonging to the corresponding one of N classes;   inputting the first stage class score image into the second stage CNN; and   using the second stage CNN to classify each pixel of the first stage class score image among N classes, to generate a second stage class score image, the second stage class score image having a height and a width identical to those of the input image and having a depth equal to N, a pixel value of each pixel of the second stage class score image being a vector of size N representing second stage preliminary probabilities of a corresponding pixel in the input image belonging to the corresponding classes.   
     
     
         9 . The image classification method of  claim 8 , wherein the second stage preliminary probabilities are final probabilities of the corresponding pixel in the input image belonging to the corresponding classes. 
     
     
         10 . The image classification method of  claim 8 , further comprising:
 providing a third stage CNN, the third stage CNN having a plurality of layers of neurons stacked sequentially, including at least a plurality of convolutional layers and a plurality of pooling layers, each convolutional layer performing convolution operations to convolve a number of filters across its previous layer, each pooling layer performing pooling operations on its previous layer;   inputting the second stage class score image into the third stage CNN; and   using the third stage CNN to classify each pixel of the second stage class score image among N classes, to generate a final class score image, the final class score image having a height and a width identical to those of the input image and having a depth equal to N, a pixel value of each pixel of the final class score image being a vector of size N representing final probabilities of a corresponding pixel in the input image belonging to the corresponding classes.   
     
     
         11 . The image classification method of  claim 8 , wherein the first stage CNN and the second stage CNN have identical numbers and identical arrangements of the convolutional layers and the pooling layers, wherein the corresponding layers of the CNNs have identical sizes and the filters in corresponding convolutional layers of the CNNs have identical sizes. 
     
     
         12 . The image classification method of  claim 8 , wherein the first stage and second stage CNNs are independent of each other. 
     
     
         13 . The image classification method of  claim 8 , wherein the filters in all of the plurality of convolutional layers in the first stage and second stage CNNs have a height of 3 and a width of 3. 
     
     
         14 . The image classification method of  claim 8 , wherein N is equal to three, wherein the three classes include a background class, a foreground class and an edge class. 
     
     
         15 . A method of training an artificial neural network system for image classification, the artificial neural network system being implemented on a computer and including a first stage convolutional neural network (CNN) and a second stage CNN, each of the first stage CNN and the second stage CNN having a plurality of layers of neurons stacked sequentially, including at least a plurality of convolutional layers and a plurality of pooling layers, each convolutional layer performing convolution operations to convolve a number of filters across its previous layer, each pooling layer performing pooling operations on its previous layer,
 the training method comprising:   (a) training the first stage CNN using a first plurality of training images and first corresponding label data, wherein the label data corresponding to each of the first plurality of training image has a height and a weight equal to those of the corresponding training image and a pixel value of each pixel of the label data represents a desired classification result for a corresponding pixel of the corresponding training image, the desired classification being one of N classes, the training being conducted for M1 iterations to obtain a set of parameters for the first stage CNN;   (b) using the first stage CNN with the parameters obtained in step (a), performing forward propagation on each of a second plurality of training images, to generate a corresponding plurality of first stage class score images, each first stage class score image having a height and a width identical to those of the corresponding training image and having a depth equal to N, a pixel value of each pixel of the first stage class score image being a vector of size N representing first stage preliminary probabilities of a corresponding pixel in the corresponding training image belonging to the corresponding one of N classes;   (c) training the second stage CNN using the plurality of first stage class score images and second label data that correspond to the second plurality of training images, wherein the second label data corresponding to each of the second plurality of training image has a height and a weight equal to those of the corresponding training image and a pixel value of each pixel of the label data represents a desired classification result for a corresponding pixel of the corresponding training image, the desired classification being one of N classes, the training being conducted for M2 iterations to obtain a set of parameters for the second stage CNN.   
     
     
         16 . The training method of  claim 15 , wherein M1 is less than M2. 
     
     
         17 . The training method of  claim 15 , wherein the first stage CNN and the second stage CNN have identical numbers and identical arrangements of the convolutional layers and the pooling layers, wherein the corresponding layers of the CNNs have identical sizes and the filters in corresponding convolutional layers of the CNNs have identical sizes. 
     
     
         18 . The training method of  claim 15 , wherein the first stage and second stage CNNs are independent of each other. 
     
     
         19 . The training method of  claim 15 , wherein the filters in all of the plurality of convolutional layers in the first stage and second stage CNNs have a height of 3 and a width of 3. 
     
     
         20 . The training method of  claim 15 , wherein N is equal to three, wherein the three classes include a background class, a foreground class and an edge class.