Driving state monitoring methods and apparatuses, driver monitoring systems, and vehicles

Abstract

Embodiments of the present application disclose driving state monitoring methods and apparatuses, driver monitoring systems, and vehicles. The driving state monitoring method includes: performing driver state detection on a driver image; and performing at least one of: outputting a driving state monitoring result of a driver or performing intelligent driving control based on a result of the driver state detection. The embodiments of the present application can implement real-time monitoring of the driving state of a driver, so as to take corresponding measures in time when the driving state of the driver is poor, to ensure safe driving and avoid road traffic accidents.

1 . A driving state monitoring method, comprising:
 detecting a driver state on a driver image; and   outputting a driving state monitoring result of a driver and/or performing an intelligent driving control based on the detected driver state;   wherein the detecting comprises at least one of: driver fatigue state detection, driver distraction state detection, or driver scheduled distraction action detection,   wherein,   the driver distraction state detection comprises:
 performing gaze direction detection on the driver in the driver image to obtain gaze direction information, and 
   the gaze direction detection comprises:
 determining a pupil edge location based on an eye image positioned by an eye key point among face key points, and computing a pupil center location based on the pupil edge location; and 
 computing gaze direction information based on the computed pupil center location and an eye center location. 
   
     
     
         2 . The method according to  claim 1 , wherein the driver fatigue state detection comprises:
 detecting at least part of a face region of the driver in the driver image to obtain state information of the at least part of the face region, the obtained state information comprising at least one of: eye open/closed state information or mouth open/closed state information;   obtaining a parameter value of an index for representing a driver fatigue state based on the state information of the at least part of the face region within a period of time; and   determining the driver fatigue state based on the obtained parameter value;   wherein the index for representing the driver fatigue state comprises at least one of: an eye closure degree or a yawning degree;   wherein the parameter value of the eye closure degree comprises at least one of: a number of eye closures, an eye closure frequency, eye closure duration, eye closure amplitude, a number of eye semi-closures, or an eye semi-closure frequency;   wherein the parameter value of the yawning degree comprises at least one of: a yawning state, a number of yawns, yawning duration, or a yawning frequency.   
     
     
         3 . The method according to  claim 1 , wherein the driver distraction state detection comprises:
 performing at least one of face orientation detection on the driver in the driver image to obtain at least one of face orientation information.   
     
     
         4 . The method according to  claim 3 , further comprising:
 determining a parameter value of an index for representing a driver distraction state based on at least one of the face orientation information or the gaze direction information within a period of time,   determining a result of the driver distraction state detection based on the determined parameter value;   wherein the index for representing the driver distraction state comprises at least one of: a face orientation deviation degree or a gaze deviation degree.   wherein the parameter value of the face orientation deviation degree comprises at least one of: a number of head turns, head turning duration, or a head turning frequency;   wherein the parameter value of the gaze deviation degree comprises at least one of: a gaze direction deviation angle, gaze direction deviation duration, or a gaze direction deviation frequency.   
     
     
         5 . The method according to  claim 3 , wherein the face orientation detection comprises:
 detecting face key points of the driver image;   obtaining feature information of head pose based on the face key points; and   determining the face orientation information based on the feature information of the head pose.   
     
     
         6 . The method according to  claim 5 , wherein the obtaining feature information comprise:
 extracting the feature information of the head pose via a first neural network based on the face key points; and   wherein determining the face orientation information based on the feature information of the head pose comprise:   performing face orientation estimation via a second neural network based on the feature information of the head pose to obtain the face orientation information.   
     
     
         7 . The method according to  claim 1 , wherein the determining a pupil edge location based on an eye image positioned by an eye key point among the face key points comprises:
 detecting, based on a third neural network, a pupil edge location of an eye region image among images divided based on the face key points, and obtaining the pupil edge location based on information outputted by the third neural network.   
     
     
         8 . The method according to  claim 1 , wherein the driver scheduled distraction action detection comprises:
 performing a target object detection corresponding to the scheduled distraction action on the driver image to obtain a detection frame for a target object; and   determining whether the scheduled distraction action occurs based on the detection frame for the target object.   
     
     
         9 . The method according to  claim 8 , further comprising:
 in response to detecting the scheduled distraction action, obtaining a determination result indicating whether the scheduled distraction action occurs within a period of time to obtain a parameter value of an index for representing a distraction degree; and   determining the result of the driver scheduled distraction action detection based on the parameter value of the index for representing the distraction degree;   wherein the parameter value of the distraction degree comprises at least one of: a number of occurrences of the scheduled distraction action, duration of the scheduled distraction action, or a frequency of the scheduled distraction action.   
     
     
         10 . The method according to  claim 1 , wherein the driver scheduled distraction action detection further comprises:
 performing a preset target object detection to obtain a detection frame for a preset target object; the preset target object comprising: hands, mouth, eyes, or a target item; and the target item comprising at least one of following types: containers, foods, or electronic devices; and   determining the detection result of the scheduled distraction action based on whether a detection frame for the hands, a detection frame for the mouth, a detection frame for the eyes, or a detection frame for the target item are detected, whether the detection frame for the hands overlaps the detection frame for the target item, a type of the target item, and whether a distance between the detection frame for the target item and the detection frame for the mouth or the detection frame for the eyes satisfies preset conditions.   
     
     
         11 . The method according to  claim 9  or  claim 10 , further comprising:
 prompting the detected distraction action or outputting distraction prompt information based on at least one of a result of the driver distraction state detection or the result of the driver scheduled distraction action detection, in response to the scheduled distraction action being detected. 
 
     
     
         12 . The method according to  claim 1 , wherein the outputting comprises:
 determining a driving state level according to a preset condition that the result of the driver fatigue state detection, the result of the driver distraction state detection, and the result of the driver scheduled distraction action detection satisfy; and   using the determined driving state level as the driving state monitoring result.   
     
     
         13 . The method according to  claim 1 , further comprising:
 performing a control operation corresponding to the driving state monitoring result comprising at least one of:   in response to determining that the determined driving state monitoring result satisfies a predetermined prompting/warning condition, outputting one or more prompting/warning information corresponding to the predetermined prompting/warning condition; or   in response to determining that the determined driving state monitoring result satisfies a predetermined driving mode switching condition, switching a driving mode to an automatic driving mode.   
     
     
         14 . The method according to  claim 1 , further comprising:
 performing facial recognition on the driver image; and   performing authentication control based on a result of the facial recognition.   
     
     
         15 . The method according to  claim 14 , wherein the performing facial recognition on the driver image comprises:
 performing face detection on the driver image via a neural network, and performing feature extraction on the detected face to obtain a face feature;   performing face matching between the face feature and face feature templates in a database; and   in response to a face feature template matching the face feature existing in the database, outputting identity information corresponding to the face feature template matching the face feature.   
     
     
         16 . The method according to  claim 15 , further comprising:
 in response to no face feature template matching the face feature existing in the database, prompting the driver to register;   in response to receiving a registration request from the driver, performing face detection on the collected driver image via a neural network, and performing feature extraction on the detected face to obtain a face feature;   establishing user information of the driver in the database by using the face feature as the face feature template of the driver, the user information comprising the face feature template of the driver and the identity information inputted by the driver; and   storing the driving state monitoring result in the user information of the driver in the database.   
     
     
         17 . The method according to  claim 1 , further comprising:
 performing image collection via an infrared camera to obtain the driver image by:
 performing image collection using the infrared camera deployed in at least one location within a vehicle; 
   wherein the at least one location comprises at least one of the following locations: a location above or near a dashboard, a location above or near a center console, an A-pillar or nearby location, or a rear-view mirror or nearby location.   
     
     
         18 . The method according to  claim 1 , further comprising:
 performing driver gesture detection based on the driver image;   generating a control instruction based on a result of the driver gesture detection;   wherein the performing driver gesture detection based on the driver image comprises:   detecting a hand key point in a driver image of a current frame; and   using a static gesture determined based on the detected hand key point as the result of the driver gesture detection; or   wherein the performing driver gesture detection based on the driver image comprises:   detecting hand key points of a plurality of driver image frames in a driver video; and   using a dynamic gesture determined based on the detected hand key points of the plurality of driver image frames as the result of the driver gesture detection.   
     
     
         19 . A driving state monitoring apparatus, comprising:
 a processor; and   a memory storing instructions, the instructions when executed by the processor, cause the processor to perform operations, the operations comprising:   detecting a driver state on a driver image; and   outputting a driving state monitoring result of a driver and/or performing an intelligent driving control based on the detected driver state;   wherein the detecting comprises at least one of: driver fatigue state detection, driver distraction state detection, or driver scheduled distraction action detection,   wherein,   the driver distraction state detection comprises:
 performing gaze direction detection on the driver in the driver image to obtain gaze direction information, and 
   the gaze direction detection comprises:
 determining a pupil edge location based on an eye image positioned by an eye key point among face key points, and computing a pupil center location based on the pupil edge location; and 
 computing gaze direction information based on the computed pupil center location and an eye center location. 
   
     
     
         20 . A non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program when executed by a processor, causes the processor to perform operations, the operations comprising:
 detecting a driver state on a driver image; and   outputting a driving state monitoring result of a driver and/or performing an intelligent driving control based on the detected driver state;   wherein the detecting comprises at least one of: driver fatigue state detection, driver distraction state detection, or driver scheduled distraction action detection,   wherein,   the driver distraction state detection comprises:
 performing gaze direction detection on the driver in the driver image to obtain gaze direction information, and 
   the gaze direction detection comprises:
 determining a pupil edge location based on an eye image positioned by an eye key point among face key points, and computing a pupil center location based on the pupil edge location; and 
   computing gaze direction information based on the computed pupil center location and an eye center location.