Driving state monitoring methods and apparatuses, driver monitoring systems, and vehicles

Abstract

Embodiments of the present application disclose driving state monitoring methods and apparatuses, driver monitoring systems, and vehicles. The driving state monitoring method includes: performing driver state detection on a driver image; and performing at least one of: outputting a driving state monitoring result of a driver or performing intelligent driving control based on a result of the driver state detection. The embodiments of the present application can implement real-time monitoring of the driving state of a driver, so as to take corresponding measures in time when the driving state of the driver is poor, to ensure safe driving and avoid road traffic accidents.

1 . A driving state monitoring method, comprising:
 detecting a state of a driver on a driver image; and   outputting a driving state monitoring result of the driver and/or performing an intelligent driving control, based on the detected state;   wherein the detecting comprises at least one of: driver fatigue state detection, driver distraction state detection, or driver scheduled distraction action detection,   wherein, in response to the scheduled distraction action being an eating action, a drinking action, a phone call action, or an entertainment action,   performing a preset target object detection corresponding to the scheduled distraction action on the driver image via a neural network to obtain a detection frame for a preset target object; and   determining a detection result of the scheduled distraction action based on the detection frame for the preset target object.   
     
     
         2 . The method according to  claim 1 , wherein the driver fatigue state detection comprises:
 detecting at least part of a face region of the driver in the driver image to obtain state information of the at least part of the face region, the state information of the at least part of the face region comprising at least one of: eye open/closed state information or mouth open/closed state information;   obtaining a parameter value of an index for representing a driver fatigue state based on the state information of the at least part of the face region within a period of time; and   determining a result of the driver fatigue state detection based on the parameter value of the index for representing the driver fatigue state.   
     
     
         3 . The method according to  claim 2 , wherein the index for representing the driver fatigue state comprises at least one of: an eye closure degree or a yawning degree; the parameter value of the eye closure degree comprises at least one of: a number of eye closures, an eye closure frequency, eye closure duration, eye closure amplitude, a number of eye semi-closures, or an eye semi-closure frequency; and the parameter value of the yawning degree comprises at least one of: a yawning state, a number of yawns, yawning duration, or a yawning frequency. 
     
     
         4 . The method according to  claim 1 , wherein the detecting comprises:
 performing at least one of face orientation or gaze direction detection on the driver in the driver image to obtain at least one of face orientation information or gaze direction information;   determining a parameter value of an index for representing a driver distraction state based on at least one of the face orientation information or the gaze direction information within a period of time, wherein the index comprises at least one of: a face orientation deviation degree or a gaze deviation degree; and   determining a result of the driver distraction state detection based on the parameter value of the index for representing the driver distraction state;   wherein the parameter value of the face orientation deviation degree comprises at least one of: a number of head turns, head turning duration, or a head turning frequency; or the parameter value of the gaze deviation degree comprises at least one of: a gaze direction deviation angle, gaze direction deviation duration, or a gaze direction deviation frequency.   
     
     
         5 . The method according to  claim 1 , the driver scheduled distraction action detection comprises:
 performing a target object detection corresponding to the scheduled distraction action on the driver image to obtain a detection frame for a target object; and   determining whether the scheduled distraction action occurs based on the detection frame for the target object,   wherein the scheduled distraction action comprises at least one of: a smoking action, a drinking action, an eating action, a phone call action, or an entertainment action.   
     
     
         6 . The method according to  claim 5 , further comprising:
 in response to detecting the scheduled distraction action, obtaining a determination result indicating whether the scheduled distraction action occurs within a period of time to obtain a parameter value of an index for representing a distraction degree; and   determining the result of the driver scheduled distraction action detection based on the parameter value of the index for representing the distraction degree;   wherein the parameter value of the distraction degree comprises at least one of: a number of occurrences of the scheduled distraction action, duration of the scheduled distraction action, or a frequency of the scheduled distraction action.   
     
     
         7 . The method according to  claim 1 , wherein the preset target object comprising: hands, mouth, eyes, or a target item; and the target item comprising at least one of following types: containers, foods, or electronic devices. 
     
     
         8 . The method according to  claim 1 , wherein the detection result of the scheduled distraction action comprising one of: no eating action/drinking action/phone call action/entertainment action occurs, the eating action occurs, the drinking action occurs, the phone call action occurs, or the entertainment action occurs. 
     
     
         9 . The method according to  claim 1 , wherein the determining a detection result of the scheduled distraction action based on the detection frame for the preset target object comprises:
 determining the scheduled distraction action based on whether a detection frame for the hands, a detection frame for the mouth, a detection frame for the eyes, or a detection frame for the target item are detected, whether the detection frame for the hands overlaps the detection frame for the target item, a type of the target item, and whether a distance between the detection frame for the target item and the detection frame for the mouth or the detection frame for the eyes satisfies preset conditions.   
     
     
         10 . The method according to  claim 9 , wherein the determining the scheduled distraction action comprises:
 determining that the eating action or the drinking action occurs, in response to the detection frame for the hands overlapping with the detection frame for the target item, the type of the target item being a container or food, and the detection frame for the target item overlapping with the detection frame for the mouth.   
     
     
         11 . The method according to  claim 9 , wherein the determining the scheduled distraction action comprises:
 determining that the entertainment action or the phone call action occurs, in response to the detection frame for the hands overlapping with the detection frame for the target item, the type of the target item being an electronic device, and the minimum distance between the detection frame for the target item and the detection frame for the mouth being less than a first preset distance, or the minimum distance between the detection frame for the target item and the detection frame for the eyes being less than a second preset distance.   
     
     
         12 . The method according to  claim 9 , further comprising:
 in response to the detection frame for the hands, the detection frame for the mouth, and the detection frame for any one target item being not detected simultaneously, and the detection frame for the hands, the detection frame for the eyes, and the detection frame for any one target item being not detected simultaneously, determining that the detection result of the distraction action is that no eating action, drinking action, phone call action and entertainment action is detected; or   in response to the detection frame for the hands not overlapping with the detection frame for the target item, determining that the detection result of the distraction action is that no eating action, drinking action, phone call action, and entertainment action is detected; or   in response to the type of the target item being a container or food and the detection frame for the target item not overlapping with the detection frame for the mouth, or the type of the target item being an electronic device and the minimum distance between the detection frame for the target item and the detection frame for the mouth being not less than the first preset distance, or the minimum distance between the detection frame for the target item and the detection frame for the eyes being not less than the second preset distance, determining that the detection result of the distraction action is that no eating action, drinking action, phone call action, and entertainment action is detected.   
     
     
         13 . The method according to  claim 9 , further comprising:
 prompting the detected distraction action or outputting distraction prompt information based on at least one of the result of the driver distraction state detection or the result of the driver scheduled distraction action detection, in response to the scheduled distraction action being detected.   
     
     
         14 . The method according to  claim 1 , wherein the outputting comprises:
 determining a driving state level according to a preset condition that a result of the driver fatigue state detection, a result of the driver distraction state detection, and a result of the driver scheduled distraction action detection satisfy; and   using the determined driving state level as the driving state monitoring result.   
     
     
         15 . The method according to  claim 1 , further comprising:
 performing a control operation corresponding to the driving state monitoring result by:
 in response to determining that the determined driving state monitoring result satisfies a predetermined prompting/warning condition, outputting one ore more prompting/warning information corresponding to the predetermined prompting/warning condition; and/or 
 in response to determining that the determined driving state monitoring result satisfies a predetermined driving mode switching condition, switching a driving mode to an automatic driving mode. 
   
     
     
         16 . The method according to  claim 1 , further comprising:
 performing facial recognition on the driver image; and   performing authentication control based on a result of the facial recognition,   wherein the performing facial recognition on the driver image comprises:   performing face detection on the driver image via a neural network, and performing feature extraction on the detected face to obtain a face feature; and   performing face matching between the face feature and face feature templates in a database.   
     
     
         17 . The method according to  claim 16 , further comprising:
 in response to a face feature template matching the face feature existing in the database, outputting identity information corresponding to the face feature template matching the face feature, or   in response to no face feature template matching the face feature existing in the database, prompting the driver to register.   
     
     
         18 . The method according to  claim 17 , further comprising:
 in response to receiving a registration request from the driver, performing face detection on the collected driver image via the neural network, and performing feature extraction on the detected face to obtain a face feature;   establishing user information of the driver in the database by using the face feature as the face feature template of the driver, the user information comprising the face feature template of the driver and the identity information inputted by the driver; and   storing the driving state monitoring result in the user information of the driver in the database.   
     
     
         19 . A driving state monitoring apparatus, comprising:
 a processor; and   a memory storing instructions, the instructions when executed by the processor, cause the processor to perform operations, the operations comprising:   detecting a state of a driver on a driver image; and   outputting a driving state monitoring result of the driver and/or performing an intelligent driving control, based on the detected state;   wherein the detecting comprises at least one of: driver fatigue state detection, driver distraction state detection, or driver scheduled distraction action detection,   wherein, in response to the scheduled distraction action being an eating action, a drinking action, a phone call action, or an entertainment action,   performing a preset target object detection corresponding to the scheduled distraction action on the driver image via a neural network to obtain a detection frame for a preset target object; and   determining a detection result of the scheduled distraction action based on the detection frame for the preset target object.   
     
     
         20 . A non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program when executed by a processor, causes the processor to perform operations, the operations comprising:
 detecting a state of a driver on a driver image; and   outputting a driving state monitoring result of the driver and/or performing an intelligent driving control, based on the detected state;   wherein the detecting comprises at least one of: driver fatigue state detection, driver distraction state detection, or driver scheduled distraction action detection,   wherein, in response to the scheduled distraction action being an eating action, a drinking action, a phone call action, or an entertainment action,   performing a preset target object detection corresponding to the scheduled distraction action on the driver image via a neural network to obtain a detection frame for a preset target object; and   determining a detection result of the scheduled distraction action based on the detection frame for the preset target object.