Cardiopulmonary health monitoring using thermal camera and audio sensor

Abstract

System and method for monitoring vital signs of a subject, such as a sleeping patient. A health monitoring device includes a thermal camera such as an uncooled microbolometer array, to monitor breathing, pulse, core temperature, and other vital signs. An audio sensor, e.g., microphone, may be used for monitoring patient respiratory sounds and other sounds. Further information such as pulse rate, PRV, blood pressure, breathing rate and oxygenation level are derived from these signals. The health monitoring device utilizes acquired signals and higher order data in analyzing patient conditions and behaviors. Higher order data include visual data based upon thermal camera signals and audio data based upon audio sensor signals. A processor is configured to output a health determination relating to one or more health parameters of the patient by inputting one or both of the visual data and the audio data into one or more machine learning models.

1 . A monitoring device, comprising:
 a set of sensors configured to receive signals pertaining to one or more health parameters of a patient through non-physical contact with the patient, wherein the set of sensors comprise a thermal camera and an audio sensor, wherein the monitoring device is configured to monitor the one or more health parameters of the patient;   a signal processing unit configured to generate thermal imaging spatial data and thermal imaging temperature data based upon signals output by the thermal camera and to generate audio data based upon signals output by the audio sensor; and   a processor configured to output a health determination relating to the one or more health parameters of the patient by inputting at least two of the thermal imaging spatial data, thermal imaging temperature data, and the audio data into one or more machine learning models.   
     
     
         2 . The device of  claim 1 , wherein the health determination comprises a value of the one or more health parameters, a binary classification of the one or more health parameters, a multiclass classification of the one or more health parameters, an event relating to the one or more health parameters, or a health anomaly relating to the one or more health parameters. 
     
     
         3 . The device of  claim 1 ; wherein the thermal camera comprises an uncooled microbolometer array. 
     
     
         4 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data representative of movement of the patient's chest or throat. 
     
     
         5 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data comprising pulse data derived from thermal camera images of one or more of carotid artery in neck of the patient, temple of the patient, an arm of the patient, and a hand of the patient. 
     
     
         6 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data comprising pulse data representative of pulse waveform and energy of the patient. 
     
     
         7 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data comprising corrected tear ducts coordinate temperature values of the patient, wherein the one or more machine learning models comprise a core temperature model derived from the tear duct coordinate temperature values of the patient. 
     
     
         8 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise one or both thermal imaging spatial data and thermal imaging temperature data representative of movement of the patient's chest or throat, and further comprise audio data comprising audio respiratory data representative of one or more of patient breathing rate and adventitious lung sounds of the patient, wherein the one or more machine learning models comprise a respiratory sounds recognition model. 
     
     
         9 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data, and further comprise audio data comprising spectrograms of audio clips output by the audio sensor, wherein the one or more machine learning models comprise a model for classifying the spectrograms. 
     
     
         10 . The device of  claim 1 , wherein the one or more health parameters of the patient identify a stage of sleep, rapid eye movement, or an apnea event, further comprising a health anomaly module configured to trigger one or more alerts displaying the identified stage of sleep, rapid eye movement, or apnea event. 
     
     
         11 . The device of  claim 1 , wherein the one or more machine learning models comprise a supervised learning model including a factorization machine. 
     
     
         12 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data and the one or more machine learning models comprise a blood pressure model that inputs the thermal imaging spatial data and thermal imaging temperature data to analyze blood pressure of the patient. 
     
     
         13 . The device of  claim 1 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data representative of movement of the patient's chest and audio data representative of patient breathing rate, wherein the one or more machine learning models comprise a tidal volume model that analyzes tidal volume of the patient based on the movement of the patient's chest and the patient breathing rate. 
     
     
         14 . The device of  claim 1 , wherein the one or more machine learning models execute multimodal machine learning in which two or more data types from the list thermal imaging spatial data, thermal imaging temperature data, and audio data are combined in a plurality of model algorithms. 
     
     
         15 . A method, comprising:
 receiving, by a set of sensors, signals pertaining to one or more health parameters of a patient through non-physical contact with the patient, wherein the sensing unit comprises a thermal camera and an audio sensor;   generating, by a processor coupled to the set of sensors, thermal imaging spatial data and thermal imaging temperature data based upon signals output by the thermal camera and audio data based upon signals output by the audio sensor; and   outputting, by the processor, a health determination relating to the one or more health parameters of the patient by inputting at least two of the thermal imaging spatial data, thermal imaging temperature data, and the audio data into one or more machine learning models.   
     
     
         16 . The method of  claim 15 , wherein the health determination comprises a value of the one or more health parameters, a binary classification of the one or more health parameters, a multiclass classification of the one or more health parameters, an event relating to the one or more health parameters, or a health anomaly relating to the one or more health parameters. 
     
     
         17 . The method of  claim 15 , wherein the one or more health parameters of the patient identify a stage of sleep, rapid eye movement, or an apnea event, further comprising the step of triggering one or more alerts displaying the identified stage of sleep, rapid eye movement, or apnea event. 
     
     
         18 . The method of  claim 15 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data comprising pulse data derived from thermal camera images of one or more of carotid artery in neck of the patient, temple of the patient, an arm of the patient, and a hand of the patient. 
     
     
         19 . The method of  claim 15 , wherein the data inputted into the one or more machine learning models comprise one or both thermal imaging spatial data and thermal imaging temperature data representative of movement of the patient's chest or throat, and further comprise audio data comprising audio respiratory data representative of one or both patient breathing rate and one or more adventitious lung sounds of the patient, wherein the analyzing step inputs the audio respiratory data into a respiratory sounds recognition model. 
     
     
         20 . The method of  claim 15 , wherein the data inputted into the one or more machine learning models comprise thermal imaging spatial data and thermal imaging temperature data, and further comprise audio data comprising spectrograms of audio clips output by the audio sensor, wherein the analyzing step inputs the spectrograms of audio clips into a model for classifying the spectrograms.