Method and systems for respiratory sound classification

Abstract

Described embodiments relate to methods, systems, and computer-readable media for training a feature encoder for encoding sound samples, such as respiratory sounds. Some embodiments further relate to methods, systems, and computer-readable media for training an audio classifier, such as a respiratory sound classifier, using the pre-trained feature encoder. Some embodiments relate to methods, systems, and computer-readable media for classifying a sample of an audio file, such as a respiratory sound, as being a positive example or a negative example of a condition, such as a respiratory condition.

1 . A system comprising:
 one or more processors; and   memory comprising computer executable instructions, which when executed by the one or more processors, cause the system to:
 a) determine a first training set comprising a plurality of audio files, wherein each audio file comprises a respiratory sound; 
 b) determine a database comprising a plurality of samples for each of the audio files of the first training set, wherein each sample is associated with an identifier; 
 c) determine a batch of sample pairs from the database, wherein the batch comprises one positive sample pair comprising a first sample and second sample, wherein the first sample and the second sample are associated with a common identifier, and the batch comprises a plurality of negative sample pairs, each negative sample pair comprising the first sample and a respective third sample, wherein the identifier associated with each respective third sample is different to the identifier associated with the first sample; 
 d) apply a first masking matrix having a first masking rate to the first sample to mask one or more select elements of the first sample in accordance with the first masking rate; 
 e) provide the masked first sample to a feature encoder to generate a first numerical representation of the masked first sample; 
 f) apply a second masking matrix having a second masking rate to the second sample to mask one or more select elements of the second sample in accordance with the second masking rate; 
 g) provide the masked second sample of the positive candidate pair to the feature encoder to generate a second numerical representation of the masked second sample; 
 h) provide the first numerical representation and the second numerical representation to a machine learning module to generate a first similarity measure indicative of the similarity between the masked first and second samples of the positive candidate pair; 
 i) determine a plurality of second similarity measures, each second similarity measure being indicative of the similarity between the first and third samples of each of the negative sample pairs, wherein said determining comprises:
 for each negative sample pair:
 i) apply a third masking matrix having a third masking rate to the third sample to mask one or more select elements of the third sample in accordance with the third masking rate; 
 ii) provide the masked third sample to the feature encoder to generate a third numerical representation of the masked third sample; and 
 iii) provide the first numerical representation and the third numerical representation to the machine learning model to generate a second similarity measure between the first and third samples; 
 
 
 j) determine a loss function value using a loss function based on the first similarity measure, and the plurality of second similarity measures; 
 k) adjust one or more of the weights of the feature encoder based on the determined loss function value; 
 l) responsive to a iteration count being less than a threshold value:
 i) increment the iteration count; and 
 ii) repeat c) to k); 
 
 m) responsive to the iteration count reaching the threshold value, determine the feature encoder as a pre-trained feature encoder. 
   
     
     
         2 . The system of  claim 1 , wherein the computer executable instructions, which when executed by the one or more processors, further cause the system to:
 a) determine a second training set of a plurality of audio files, each audio file comprising a label indicative of whether the sound is a positive example of a condition or a negative example of the condition;   b) generate a batch of samples from the second training set, wherein each sample is generated from a respective audio file and is associated with the label of the audio file from which it was generated; and   c) for each of the plurality of samples of the second batch:
 i) select a candidate sample from the second batch; 
 ii) provide the candidate sample to the pre-trained feature encoder to generate a candidate numerical representation of the candidate sample; 
 iii) provide the candidate numerical representation to a classifier to determine a predictive score, wherein the predictive score is indicative of the likelihood of the candidate sample being a positive example of the condition; and 
 iv) adjust one or more weights of the classifier based on the label associated with the candidate sample and the predictive score. 
   
     
     
         3 . The system of  claim 2 , wherein the pre-trained feature encoder comprises a first pre-trained feature encoder and a second pre-trained feature encoder, and wherein the computer executable instructions, which when executed by the one or more processors, cause the system to provide the candidate sample to the pre-trained feature encoder further cause the system to:
 a) apply a fourth masking matrix having a fourth masking rate to the candidate sample to mask one or more select elements of the candidate sample in accordance with the fourth masking rate to provide a fourth masked sample;   b) apply a fifth masking matrix having a fifth masking rate to the candidate sample to mask one or more select elements of the candidate sample in accordance with the fifth masking rate to provide a fifth masked sample; and   c) provide the fourth masked sample to the first pre-trained encoder to generate a fourth numerical representation of the fourth masked sample and providing the fifth masked sample to the second pre-trained encoder to generate a fifth numerical representation of the fifth masked sample; and   
       wherein providing the candidate numerical representation to the classifier comprises providing both the fourth numerical representation and the fifth numerical representation as inputs to the classifier. 
     
     
         4 . A computing device comprising:
 one or more processors; and   memory comprising computer executable instructions, which when executed by the one or more processors, cause the computing device to:   determine an audio file comprising a respiratory sound;   provide a sample of the audio file to a respiratory sound classification module, wherein the respiratory sound classification model was trained using a contrastive pre-trained feature encoder; and   determine an output from the respiratory sound classification module, wherein the output in indicative of whether or not the respiratory sound is a positive or negative example of a respiratory condition.   
     
     
         5 . The device of  claim 4 , wherein providing the sample of the audio file to the respiratory sound classification module further causes the system to:
 a) mask the sample using a masking matrix with an associated masking rate; and   b) provide the masked sample to the cough classification module.   
     
     
         6 . The computing device of  claim 4 or claim 5 , wherein the computing device is a mobile phone device or a smart sensor. 
     
     
         7 . The computing device of any one of  claims 4 to 6 , wherein the respiratory sound classification module is a cough classification module. 
     
     
         8 . A method comprising:
 a) determining a first training set comprising a plurality of audio files, wherein each audio file comprises a respiratory sound;   b) determining a database comprising a plurality of samples for each of the audio files of the first training set, wherein each sample is associated with an identifier;   c) determining a batch of sample pairs from the database, wherein the batch comprises one positive sample pair comprising a first sample and second sample, wherein the first sample and the second sample are associated with a common identifier, and the batch comprises a plurality of negative sample pairs, each negative sample pair comprising the first sample and a respective third sample, wherein the identifier associated with each respective third sample is different to the identifier associated with the first sample;   d) applying a first masking matrix having a first masking rate to the first sample to mask one or more select elements of the first sample in accordance with the first masking rate;   e) providing the masked first sample to a feature encoder to generate a first numerical representation of the masked first sample;   f) applying a second masking matrix having a second masking rate to the second sample to mask one or more select elements of the second sample in accordance with the second masking rate;   g) providing the masked second sample of the positive candidate pair to the feature encoder to generate a second numerical representation of the masked second sample;   h) providing the first numerical representation and the second numerical representation to a machine learning module to generate a first similarity measure indicative of the similarity between the masked first and second samples of the positive candidate pair;   i) determining a plurality of second similarity measures, each second similarity measure being indicative of the similarity between the first and third samples of each of the negative sample pairs, wherein said determining comprises:
 for each negative sample pair:
 i) applying a third masking matrix having a third masking rate to the third sample to mask one or more select elements of the third sample in accordance with the third masking rate; 
 ii) providing the masked third sample to the feature encoder to generate a third numerical representation of the masked third sample; and 
 iii) providing the first numerical representation and the third numerical representation to the machine learning model to generate a second similarity measure between the first and third samples; 
 
   j) determining a loss function value using a loss function based on the first similarity measure, and the plurality of second similarity measures;   k) adjusting one or more of the weights of the feature encoder based on the determined loss function value;   l) responsive to a iteration count being less than a threshold value:
 i) incrementing the iteration count; and 
 ii) repeating steps c) to k); 
   m) responsive to the iteration count reaching the threshold value, determining the feature encoder as a pre-trained feature encoder.   
     
     
         9 . The method of  claim 8 , further comprising:
 a) determining a second training set of a plurality of audio files, each audio file comprising a label indicative of whether the sound is a positive example of a condition or a negative example of the condition;   b) generating a batch of samples from the second training set, wherein each sample is generated from a respective audio file and is associated with the label of the audio file from which it was generated; and   c) for each of the plurality of samples of the second batch:
 i) selecting a candidate sample from the second batch; 
 ii) providing the candidate sample to the pre-trained feature encoder to generate a candidate numerical representation of the candidate sample; 
 iii) providing the candidate numerical representation to a classifier to determine a predictive score, wherein the predictive score is indicative of the likelihood of the candidate sample being a positive example of the condition; and 
 iv) adjusting one or more weights of the classifier based on the label associated with the candidate sample and the predictive score. 
   
     
     
         10 . The method of  claim 9 , wherein the pre-trained feature encoder comprises a first pre-trained feature encoder and a second pre-trained feature encoder, and wherein providing the candidate sample to the pre-trained feature encoder comprises:
 a) applying a fourth masking matrix having a fourth masking rate to the candidate sample to mask one or more select elements of the candidate sample in accordance with the fourth masking rate to provide a fourth masked sample;   b) applying a fifth masking matrix having a fifth masking rate to the candidate sample to mask one or more select elements of the candidate sample in accordance with the fifth masking rate to provide a fifth masked sample; and   c) providing the fourth masked sample to the first pre-trained encoder to generate a fourth numerical representation of the fourth masked sample and providing the fifth masked sample to the second pre-trained encoder to generate a fifth numerical representation of the fifth masked sample; and   
       wherein providing the candidate numerical representation to the classifier comprises providing both the fourth numerical representation and the fifth numerical representation as inputs to the classifier. 
     
     
         11 . The method of  claim 10 , wherein the first and second pre-trained feature encoders are initialized using: (i) the same pre-trained weights; or (ii) different pre-trained weights. 
     
     
         12 . The method of  claim 10 or claim 11 , wherein the pre-trained feature encoder comprises a third pre-trained feature encoder and wherein providing the candidate sample to the pre-trained feature encoder comprises:
 a) applying a sixth masking matrix having a sixth masking rate to the candidate sample to mask one or more select elements of the candidate sample in accordance with the sixth masking rate to provide a sixth masked sample;   b) providing the sixth masked sample to the first pre-trained encoder to generate a sixth numerical representation of the sixth masked sample; and   
       wherein providing the candidate numerical representation to the classifier further comprises providing the sixth numerical representation as an input to the classifier. 
     
     
         13 . The method of any one of  claims 10 to 12 , wherein the fourth and/or fifth sampling matrices are generated using a pseudo-random number generator. 
     
     
         14 . The method of any one of  claims 8 to 13 , wherein the first and/or second and/or third sampling matrices are generated using a pseudo-random number generator. 
     
     
         15 . The method of any one of  claims 8 to 14 , wherein determining the database comprising the plurality of samples comprises:
 i) transforming each audio file into a feature matrix, each feature matrix comprising a first dimension corresponding to a number of frequency bins, and a second dimension corresponding to a number of time frames in the audio file;   ii) for each feature matrix, generating at least one sample, each sample comprising the first dimension corresponding to a number of frequency bins, and a third dimension corresponding to a predefined number of time frames, wherein the third dimension is a subset of the second dimension.   
     
     
         16 . The method of any one of  claims 8 to 15 , wherein each of the plurality of samples comprises at least a portion of a sound of a respective audio file. 
     
     
         17 . The method of any one of  claims 8 to 16 , wherein the feature encoder is a Transformer based feature encoder. 
     
     
         18 . The method of any one of  claims 8 to 17 , wherein the feature encoder is a Convolutional Neural Network (CNN) based feature encoder. 
     
     
         19 . The method of any one of  claims 8 to 17 , wherein the feature encoder is a Recurrent Neural Network (RNN) based feature encoder. 
     
     
         20 . The method of any one of  claims 8 to 19 , wherein the machine learning model generates a similarity measure using one of: (i) a cosine similarity metric; and (ii) a bilinear similarity metric. 
     
     
         21 . The method of any one of  claims 8 to 20 , wherein each audio file of the first training set comprises a respiratory sound. 
     
     
         22 . The method of  claim 9 , or any one of  claims 10 to 21  when directly or indirectly dependent on  claim 9 , wherein each audio file of the second training set comprises a respiratory sound. 
     
     
         23 . The method of  claim 21 or claim 22 , wherein the respiratory sound is a cough. 
     
     
         24 . The method of any one of  claims 21 to 23 , wherein the condition is a respiratory condition. 
     
     
         25 . The method of any one of  claims 21 to 24 , wherein the respiratory condition is a classification of a cough. 
     
     
         26 . The method of any one of  claims 21 to 25 , wherein the respiratory condition is COVID-19. 
     
     
         27 . The method of any one of  claims 8 to 26 , wherein the audio files of the first training set do not include labels indicative of whether the sound of the audio file is a positive example of a condition or a negative example of the condition. 
     
     
         28 . The method of  claim 9  or any one of  claims 10 to 27 , when directly or indirectly dependent on  claim 9 , further comprising deploying the trained feature encoder and the trained classifier on a computing device for use. 
     
     
         29 . A method comprising:
 determining an audio file comprising a respiratory sound;   providing a sample of the audio file to a respiratory sound classification module, wherein the respiratory sound classification model was trained using a contrastive pre-trained feature encoder; and   determining an output from the respiratory sound classification module, wherein the output in indicative of whether or not the respiratory sound is a positive or negative example of a respiratory condition.   
     
     
         30 . The method of  claim 29 , wherein providing the sample of the audio file to the respiratory sound classification module further comprises:
 a) masking the sample using a masking matrix with an associated masking rate; and   b) providing the masked sample to the cough classification module.   
     
     
         31 . The method of  claim 29 or 30 , wherein the output is provided to a user. 
     
     
         32 . The method of any one of  claims 29 to 31 , wherein the cough classification module is trained according to the method of  claim 9  or any one of  claims 9 to 28 , when directly or indirectly dependent on  claim 9 . 
     
     
         33 . A system comprising:
 one or more processors; and   memory comprising computer executable instructions, which when executed by the one or more processors, cause the system to perform the method of any one of  claims 8 to 28 .   
     
     
         34 . A computing device comprising:
 one or more processors; and   memory comprising computer executable instructions, which when executed by the one or more processors, cause the computing device to perform the method of any one of claims  29  to  32 .   
     
     
         35 . The computing device of  claim 34 , wherein memory comprises a trained cough classification module. 
     
     
         36 . The computing device of  claim 35 , wherein the cough classification module has been trained according to the method of  claim 9 , or any one of  claims 10 to 28 , when directly or indirectly dependent on  claim 9 . 
     
     
         37 . The computing device of any one of claims  34  to  37 , wherein the computing device is a mobile phone device or a smart sensor. 
     
     
         38 . A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method of any one of  claims 8 to 32 .