Systems and Methods for Debugging Neural Networks with Coverage Guided Fuzzing

Abstract

The present disclosure provides systems and methods for debugging neural networks. In one example, a computer-implemented method is provided, which includes obtaining, by one or more computing devices, one or more inputs from an input corpus. The method further includes mutating, by the one or more computing devices, the one or more inputs and providing the one or more mutated inputs to a neural network; obtaining, by the one or more computing devices as a result of the neural network processing the one or more mutated inputs, a set of coverage arrays; determining, by the one or more computing devices based at least in part on the set of coverage arrays, whether the one or more mutated inputs provide new coverage; and upon determining that the one or more mutated inputs provide new coverage, adding the one or more mutated inputs to the input corpus.

1 .- 20 . (canceled) 
     
     
         21 . A computer-implemented method for debugging a neural network, the method comprising:
 obtaining, by one or more computing devices, one or more inputs from an input corpus;   mutating, by the one or more computing devices, the one or more inputs to generate one or more mutated inputs, wherein the one or more mutated inputs are constrained to have a same class as the one or more inputs;   providing, by the one or more computing devices, the one or more mutated inputs to a neural network;   obtaining, by the one or more computing devices as a result of the neural network processing the one or more mutated inputs, a set of coverage arrays that describe whether one or more neurons of the neural network were activated during processing of the one or more mutated inputs by the neural network;   determining, by the one or more computing devices based at least in part on the set of coverage arrays, whether the one or more mutated inputs provide new coverage; and   upon determining that the one or more mutated inputs provide new coverage, adding, by the one or more computing devices, the one or more mutated inputs to the input corpus.   
     
     
         22 . The method of  claim 21 , further comprising:
 obtaining, by the one or more computing devices as a result of the neural network processing the one or more mutated inputs, a set of metadata arrays that describe metadata associated with execution of the neural network to process the one or more mutated inputs;   determining, by the one or more computing devices based at least in part on the set of metadata arrays, whether an objective function is satisfied; and   upon determining that the objective function is satisfied, adding, by the one or more computing devices, the one or more mutated inputs to a list of test cases.   
     
     
         23 . The method of  claim 21 , wherein determining, by the one or more computing devices based at least in part on the set of coverage arrays, whether the one or more mutated inputs provide new coverage comprises:
 generating, by the one or more computing devices, an activation vector based at least in part on the set of coverage arrays;   performing, by the one or more computing devices, an approximate nearest neighbors algorithm to identify a previous activation vector;   determining, by the one or more computing devices, a distance between the activation vector and the previous activation vector identified by the approximate nearest neighbors algorithm; and   comparing, by the one or more computing devices, the distance to a threshold distance;   wherein the one or more mutated inputs provide new coverage when the distance is greater than the threshold distance.   
     
     
         24 . The method of  claim 21 , wherein obtaining one or more inputs from the input corpus comprises using uniform random selection to select the one or more inputs from the input corpus. 
     
     
         25 . The method of  claim 21 , wherein obtaining one or more inputs from the input corpus comprises selecting the one or more inputs from the input corpus using a heuristic of 
       
         
           
             
               
                 
                   p 
                   ⁡ 
                   
                     ( 
                     
                       
                         c 
                         k 
                       
                       , 
                       t 
                     
                     ) 
                   
                 
                 = 
                 
                   
                     e 
                     
                       
                         t 
                         k 
                       
                       - 
                       t 
                     
                   
                   
                     ∑ 
                     
                       e 
                       
                         
                           t 
                           k 
                         
                         - 
                         t 
                       
                     
                   
                 
               
               , 
             
           
         
       
       wherein p(c k , t) gives a probability of choosing input corpus element c k  at time t where t k  is the time when element c k  was added to the input corpus. 
     
     
         26 . The method of  claim 21 , wherein mutating the one or more inputs comprises mutating the one or more inputs subject to a constraint, wherein the constraint comprises a L ∞  norm. 
     
     
         27 . The method of  claim 26 , wherein the L ∞  norm is specified by a user. 
     
     
         28 . The method of  claim 21 , wherein determining whether the one or more mutated inputs provide new coverage comprises determining whether the neural network has reached a new state that it has not previously reached. 
     
     
         29 . The method of  claim 28 , wherein determining whether the neural network has reached a new state that it has not previously reached comprises determining whether an activation vector is approximately close to a previous activation vector. 
     
     
         30 . The method of  claim 22 , wherein determining whether the objective function is satisfied comprises determining whether the neural network has reached a desired state. 
     
     
         31 . The method of  claim 30 , wherein the desired state is an erroneous state for the neural network. 
     
     
         32 . A computing device comprising:
 one or more processors; and   one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing device to:
 obtain one or more inputs from an input corpus; 
 mutate the one or more inputs to generate one or more mutated inputs, wherein the one or more mutated inputs are constrained to have a same class as the one or more inputs; 
 provide the one or more mutated inputs to a neural network; 
 obtain as a result of the neural network processing the one or more mutated inputs, a set of coverage arrays that describe whether one or more neurons of the neural network were activated during processing of the one or more mutated inputs by the neural network; 
 determine based at least in part on the set of coverage arrays, whether the one or more mutated inputs provide new coverage; and 
 upon determining that the one or more mutated inputs provide new coverage, add the one or more mutated inputs to the input corpus. 
   
     
     
         33 . The computing device of  claim 32 , further comprising instructions, that when executed, cause the computing device to:
 obtain as a result of the neural network processing the one or more mutated inputs, a set of metadata arrays that describe metadata associated with execution of the neural network to process the one or more mutated inputs;   determine based at least in part on the set of metadata arrays, whether an objective function is satisfied; and   upon determining that the objective function is satisfied, add the one or more mutated inputs to a list of test cases.   
     
     
         34 . The computing device of  claim 32 , further comprising instructions, that when executed, cause the computing device to:
 obtain the input corpus from a seed corpus, the seed corpus containing at least one set of inputs.   
     
     
         35 . The computing device of  claim 32 , wherein obtaining one or more inputs from the input corpus comprises:
 using uniform random selection to select the one or more inputs from the input corpus; or   selecting the one or more inputs from the input corpus using a heuristic of   
       
         
           
             
               
                 
                   p 
                   ⁡ 
                   
                     ( 
                     
                       
                         c 
                         k 
                       
                       , 
                       t 
                     
                     ) 
                   
                 
                 = 
                 
                   
                     e 
                     
                       
                         t 
                         k 
                       
                       - 
                       t 
                     
                   
                   
                     ∑ 
                     
                       e 
                       
                         
                           t 
                           k 
                         
                         - 
                         t 
                       
                     
                   
                 
               
               , 
             
           
         
       
       wherein p(c k , t) gives a probability of choosing input corpus element c k  at time t where t k  is the time when element c k  was added to the input corpus. 
     
     
         36 . The computing device of  claim 32 , wherein a difference between the one or more mutated inputs and the one or more inputs from which the one or more mutated inputs are descended is constrained to have a L ∞  norm. 
     
     
         37 . The computing device of  claim 32 , wherein determining whether the one or more mutated inputs provide new coverage comprises determining whether the neural network has reached a new state that it has not previously reached; and
 wherein determining whether the neural network has reached a new state that it has not previously reached comprises determining whether an activation vector is approximately close to a previous activation vector.   
     
     
         38 . The computing device of  claim 33 , wherein determining whether the objective function is satisfied comprises determining whether the neural network has reached a desired state, wherein the desired state is an erroneous state for the neural network. 
     
     
         39 . One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations, the operations comprising:
 obtaining one or more inputs from an input corpus;   mutating the one or more inputs to generate one or more mutated inputs, wherein a difference between the one or more mutated inputs and the one or more inputs from which the one or more mutated inputs are descended is constrained to have a L ∞  norm;   providing the one or more mutated inputs to a neural network;   obtaining, as a result of the neural network processing the one or more mutated inputs, a set of metadata arrays that describe metadata associated with execution of the neural network to process the one or more mutated inputs;   determining, based at least in part on the set of metadata arrays, whether an objective function is satisfied; and   upon determining that the objective function is satisfied, adding the one or more mutated inputs to a list of test cases.   
     
     
         40 . The one or more non-transitory computer-readable media of  claim 39 , wherein the operations further comprise:
 obtaining, as a result of the neural network processing the one or more mutated inputs, a set of coverage arrays that describe whether one or more neurons of the neural network were activated during processing of the one or more mutated inputs by the neural network;   determining, based at least in part on the set of coverage arrays, whether the one or more mutated inputs provide new coverage; and   upon determining that the one or more mutated inputs provide new coverage, adding the one or more mutated inputs to the input corpus.