Monitoring of interactions using eye tracking

Abstract

An electronic interaction device may include an image sensor, a display, a storage device, a memory storing instructions, and a processing unit including one or more processors. The one or more processors may be configured to: receive at least one image of a predetermined space adjacent the electronic interaction device, identify a plurality of persons in the predetermined space adjacent the electronic interaction device, generate a positional mapping of the predetermined space adjacent the electronic interaction device, identify a user and one or more onlookers, extract eye tracking information corresponding to the one or more onlookers, calculate one or more points of focus representing a point at which one of the one or more onlookers is looking, determine whether any of the one or more points of focus is within a predetermined zone of interest of the electronic interaction device, and alert the user.

1 - 20 . (canceled) 
     
     
         21 . A computer-implemented method comprising:
 collecting a first set of image data;   generating, based on the first set of image data, a positional mapping of a predetermined space adjacent an electronic interaction device, the positional mapping including a position for each of a plurality of persons;   identifying, among the plurality of persons and based on the positional mapping, at least one onlooker;   extracting, from the first set of image data, eye tracking information for the at least one onlooker;   determining, based on the eye tracking information, whether the at least one onlooker is looking at a predetermined zone of interest of the electronic interaction device; and   in response to a determination that the at least one onlooker is looking at the predetermined zone of interest, alerting a user via the electronic interaction device.   
     
     
         22 . The computer-implemented method of  claim 21 , wherein identifying the at least one onlooker comprises:
 calculating, for one of the plurality of persons, a distance between the position for the one of the plurality of persons and the electronic interaction device;   determining whether the distance is greater than a predetermined threshold value; and   selecting, in response to a determination that the distance is greater than the predetermined threshold value, the one of the plurality of persons as the at least one onlooker.   
     
     
         23 . The computer-implemented method of  claim 21 , wherein identifying the at least one onlooker comprises:
 selecting as the user a first person from the plurality of persons that is closest to the electronic interaction device; and   selecting as the at least one onlooker a second person from the plurality of persons other than the user.   
     
     
         24 . The computer-implemented method of  claim 21 , wherein the positional mapping is a three-dimensional mapping. 
     
     
         25 . The computer-implemented method of  claim 24 , wherein the three-dimensional mapping includes position information for a keypad and a display of the electronic interaction device. 
     
     
         26 . The computer-implemented method of  claim 25 , wherein the predetermined zone of interest is defined based on the position information for the keypad and the display 
     
     
         27 . The computer-implemented method of  claim 21 , wherein the image data includes at least one photograph and depth information. 
     
     
         28 . The computer-implemented method of  claim 21 , further comprising:
 identifying, using a trained machine learning engine, the plurality of persons using the first set of image data.   
     
     
         29 . The computer-implemented method of  claim 21 , wherein the positional mapping is generated by the electronic interaction device. 
     
     
         30 . The computer-implemented method of  claim 21 , wherein the positional mapping is generated by a server that is remote relative to the electronic interaction device. 
     
     
         31 . A non-transitory computer-readable storage medium storing program instructions that are computer-executable to implement operations comprising:
 retrieving a first set of image data;   generating, based on the first set of image data, a positional mapping of a predetermined space adjacent an electronic interaction device, the positional mapping including a position for each of a plurality of persons;   identifying, among the plurality of persons and based on the positional mapping, at least one onlooker;   extracting, from the first set of image data, eye tracking information for the at least one onlooker;   determining, based on the eye tracking information, whether the at least one onlooker is looking at a predetermined zone of interest of the electronic interaction device; and   in response to a determination that the at least one onlooker is looking at the predetermined zone of interest, causing an alert to be provided to a user via the electronic interaction device.   
     
     
         32 . The non-transitory computer-readable storage medium of  claim 31 , wherein identifying the at least one onlooker comprises:
 calculating, for one of the plurality of persons, a distance between the position for the one of the plurality of persons and the electronic interaction device;   determining whether the distance is greater than a predetermined threshold value; and   selecting, in response to a determination that the distance is greater than the predetermined threshold value, the one of the plurality of persons as the at least one onlooker.   
     
     
         33 . The non-transitory computer-readable storage medium of  claim 31 , wherein identifying the at least one onlooker comprises:
 selecting as the user a first person from the plurality of persons that is closest to the electronic interaction device; and   selecting as the at least one onlooker a second person from the plurality of persons other than the user.   
     
     
         34 . The non-transitory computer-readable storage medium of  claim 31 , wherein the predetermined zone of interest includes at least one of a keypad and a display of the electronic interaction device. 
     
     
         35 . The non-transitory computer-readable storage medium of  claim 34 , wherein the predetermined zone of interest is defined based on a three-dimensional model of the electronic interaction device. 
     
     
         36 . A non-transitory computer-readable storage medium storing program instructions that are computer-executable to implement operations comprising:
 retrieving a first set of image data corresponding to a predetermined space adjacent an electronic interaction device, wherein the predetermined space is occupied by a plurality of persons;   identifying, from the plurality of persons, at least one onlooker;   extracting, from the first set of image data, eye tracking information for the at least one onlooker;   determining, based on the eye tracking information, whether the at least one onlooker is looking at a predetermined zone of interest of the electronic interaction device; and   in response to a determination that the at least one onlooker is looking at the predetermined zone of interest, causing an alert to be provided to a user.   
     
     
         37 . The non-transitory computer-readable storage medium of  claim 36 , wherein the predetermined zone of interest includes at least one of a display and a keypad of the electronic interaction device. 
     
     
         38 . The non-transitory computer-readable storage medium of  claim 36 , wherein the alert includes at least one visual indication on a display of the electronic interaction device. 
     
     
         39 . The non-transitory computer-readable storage medium of  claim 36 , wherein the alert is provided via a user device separate from the electronic interaction device. 
     
     
         40 . The non-transitory computer-readable storage medium of  claim 36 , wherein the eye tracking information for the at least one onlooker is extracted using a trained machine learning engine.