Neural network, computer readable medium, and methods including a method for training a neural network

Abstract

The present disclosure provides an artificial neural network communicatively-coupled to at least one computer having one or more processors, including a plurality of neurons arranged in layers. The artificial neural network is arranged to receive a new neuron into a layer of the artificial neural network during training; the new neuron being added to the neural network when no other neuron in that layer for a selected output can learn a relationship associated with an input vector of a data set being learnt. The new neuron is updated with both the relationship which could not be learnt by any other neuron in that layer and a modified data set from a last trained neuron in that layer that contributes to the selected output of the neural network, wherein the modified data set is formed by copying all learnt relationships from the last trained neuron into the new neuron and modifying the copied relationship based upon the relationship which could not be learnt by any other neuron in that layer; and, one or more output neurons are updated to accept input from the new neuron. Methods and computer-readable media are also disclosed.

What is claimed: 
     
         1 . A method for training an artificial neural network, said method comprising: (i) initialising the neural network by selecting an output of the neural network to be trained and connecting an output neuron of the neural network to input neuron(s) in an input layer of the neural network for the selected output; (ii) preparing a data set to be learnt by the neural network; and (iii) applying the prepared data set to the neural network to be learnt by applying an input vector of the prepared data set to a first hidden layer of the neural network, or an output layer of the neural network if the neural network has no hidden layer(s), and determining whether at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, wherein: if at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, and if there are more input vectors of the prepared data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained; if no neuron in a hidden layer for the selected output of the neural network can learn to produce the associated output for the input vector, a new neuron is added to that layer to learn the associated output which could not be learnt by any other neurons in that layer for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained; if the output neuron for the selected output of the neural network cannot learn to produce the associated output for the input vector, that output neuron becomes a neuron of a hidden layer of the neural network, a new neuron is added to this hidden layer to learn the associated output which could not be learnt by the output neuron, and a new output neuron is added to the neural network for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained. 
     
     
         2 . A method as claimed in  claim 1 , wherein (ii) preparing the data set is performed before (i) initializing the neural network. 
     
     
         3 . The method as claimed in  claim 1 , wherein the neurons of the neural network are Linear Threshold Gates (LTGs). 
     
     
         4 . The method as claimed in  claim 3 , wherein in said (iii), to determine whether an LTG can learn to produce the associated output for the input vector is to determine whether a relationship between weights and a threshold of the LTG has a solution given what the LTG has previously learnt. 
     
     
         5 . The method as claimed in  claim 4 , wherein said relationship is a constraint, and wherein the input vector and the LTG's weight vector form a relationship with the LTG's threshold based on the selected output of the neural network. 
     
     
         6 . The method as claimed in  claim 5 , wherein to learn a constraint is to be able to add the constraint to a constraint set of an LTG. 
     
     
         7 . The method as claimed in  claim 6 , wherein to be able to add the constraint to a constraint set of an LTG there must be a solution between all the constraints. 
     
     
         8 . The method as claimed in  claim 6 , wherein initialising the neural network further includes clearing the constraints set of the output LTG so that the constraints set of the output LTG is empty. 
     
     
         9 . The method as claimed in  claim 1 , wherein preparing the data set to be learnt by the neural network includes in any order: converting the data set into a predefined data format before the data set is presented to the neural network for training; determining whether there are any inconsistencies in the data set before the data set is presented to the neural network for training; sorting the data set before the data set is presented to the neural network for training; and, determining whether the 0 input vector is available in the data set before the data set is presented to the neural network for training, and if the 0 input vector is available the data set, the data set is ordered so that the 0 input vector is presented to the neural network to be trained first. 
     
     
         10 . The method as claimed in  claim 9 , wherein said predefined data format is binary or floating-point data format. 
     
     
         11 . The method as claimed in  claim 9 , wherein determining whether there are any inconsistencies in the data set before the data set is presented to the neural network includes: determining whether there are two or more identical input vectors which produce different output. 
     
     
         12 . The method as claimed in  claim 11 , wherein if it is determined that two or more identical input vectors produce a different output, only one of the input vectors is used. 
     
     
         13 . The method as claimed in  claim 9 , wherein sorting the data set before the data set is presented to the neural network for training includes: sorting the input vectors of the data set into two sets, separating those that output 1 from those that produce 0 for that output, and selecting one of the two sets to be trained first; sorting the data with a Self Organising Map (SOM); and/or sorting the data using any other suitable method. 
     
     
         14 . The method as claimed in  claim 13 , wherein a single list for each input layer is created from the sorted data before the data is presented to the neural network for training. 
     
     
         15 . The method as claimed in  claim 5 , wherein if a new LTG is added to a layer to learn a constraint that could not be learnt by any other LTG in accordance with (iii): the new LTG is connected to all LTGs in the next layer which contribute to the selected output of the neural network, and the constraints set of the LTGs in the next layer which receive input from the new LTG are updated to accept input from the new LTG; if the layer with the new LTG is not the first layer of the neural network, the new LTG is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network; and, the constraints set of the new LTG is updated to include a copy of the modified constraints set of the previous last LTG in that layer and the constraint which could not be learnt by any other LTG in that layer. 
     
     
         16 . The method as claimed in  claim 5 , wherein if a new output LTG is added to the neural network in accordance with (iii): the new output LTG is connected to and receives input from the LTGs in the hidden layer; if the hidden layer is not the first layer of the neural network, the new LTG in the hidden layer is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network; the constraints set of the new LTG added to the hidden layer is updated to include a copy of the modified constraints set of the previous output LTG in that layer and the constraint which could not be learnt by the previous output LTG; and, the new output LTG combines its inputs in a predefined logical relationship according to what could not be learnt by the previous output LTG. 
     
     
         17 . The method as claimed in  claim 16 , wherein when a new output LTG is added to the neural network in accordance with (iii), the predefined logical relationship formed between the inputs into this new output LTG is logical OR, logical AND, or any other suitable logical relationship. 
     
     
         18 . The method as claimed in  claim 17 , wherein logical OR is used if the input vector that could not be learnt by the previous output LTG produces an output 1, and logical AND is used if the input vector that could not be learnt by the previous output LTG produces an output 0. 
     
     
         19 . A method for adding a new neuron into a layer of a neural network during training, the new neuron being added to the neural network when no other neuron in that layer for the selected output can learn a relationship associated with an input vector of a data set being learnt, said method including: updating the new neuron with a copy of all the modified data from a previous last neuron that contributes to the selected output of the neural network in that layer and the relationship which could not be learnt by any other neuron in that layer; and, updating the output neuron(s) to accept input from the new neuron. 
     
     
         20 . The method as claimed in  claim 19 , wherein the neurons of the neural network are LTGs. 
     
     
         21 . The method as claimed in  claim 20 , wherein said relationship is a relationship between weights and a threshold of an LTG. 
     
     
         22 . The method as claimed in  claim 20 , wherein said relationship is a constraint, and wherein the input vector of the data set and an LTG's weight vector form a relationship with the LTG's threshold based on the output of the neural network. 
     
     
         23 - 39 . (canceled)