Sensor recalibration in performance capture

Abstract

The present description relates relate to recalibration of a sensor device for performance capture, by detecting a miscalibration problem with sensor device and assessment of the problem. A recalibration system includes sensor devices initially calibrated at a recording site. A recording site change occurs and afterwards, a failure to match virtual rays projected from one sensor device with virtual rays projected from the active marker is detected. In response to determining the failure, the active marker is signaled to emit a unique emission of electromagnetic radiation. The failure of the rays to match is assessed based on whether sensor devices capture the unique emission of electromagnetic radiation. Three-dimensional (3-D) coordinates of an active marker is reconstructed from marker data of the calibrated sensor devices. A problematic sensor device is recalibrated based on the assessment, using the 3-D coordinates of the active marker from marker data of the remaining calibrated sensor devices, without stopping the recording.

1 . A method for recalibrating a sensor device to capture electromagnetic radiation from an active marker in performance capture, the method comprising:
 providing a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker;   determining a failure of the first sensor device;   in response to determining the failure, signaling for the active marker to emit a unique emission of electromagnetic radiation;   assessing the failure based on at least one of the first sensor device, the second sensor device, or the third sensor device capturing the unique emission of electromagnetic radiation; and   recalibrating the first sensor device based on the assessment.   
     
     
         2 . The method of  claim 1 , wherein assessing the failure includes:
 detecting that the first sensor device, the second sensor device, and the third sensor device each capture the unique emission of electromagnetic radiation; and   based on the detecting, determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device.   
     
     
         3 . The method of  claim 2 , wherein the recalibrating includes determining a changed physical position of the first sensor device by data from the first sensor device and at least one of the second sensor device and third sensor device capturing electromagnetic radiation emitted from other active markers. 
     
     
         4 . The method of  claim 2 , wherein determining a changed physical position of the first sensor device by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula. 
     
     
         5 . The method of  claim 1 , wherein assessing the failure includes:
 detecting that at least one the second sensor device and the third sensor device captures the unique emission of electromagnetic radiation and the first sensor device fails to capture the unique emission of electromagnetic radiation; and   repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated.   
     
     
         6 . The method of  claim 1 , wherein the failure of the first sensor device is based on a change of at least one intrinsic parameter including at least one of a change in focal length, optical center, principal point, skew of axis, and lens distortion. 
     
     
         7 . The method of  claim 6 , wherein recalibration includes generating a projection matrix based on data of the active marker from at least one of the second sensor device and the third sensor device. 
     
     
         8 . The method of  claim 1 , wherein assessing the failure further includes:
 identifying the active marker by the marker data representing the unique emission of electromagnetic radiation emitted by the active marker and captured in particular frames by the first sensor device, the second sensor device, and the third sensor device, when each sensor device is initially calibrated.   
     
     
         9 . A system for recalibrating a sensor device to capture electromagnetic radiation from an active marker in performance capture, the system comprising:
 a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker;   a plurality of active markers;   at least one computing device comprising one or more processors and a tangible processor-readable storage device including instructions for:   providing a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker;   after a recording site change, determining a failure of the first sensor device;   in response to determining the failure, signaling for the active marker to emit a unique emission of electromagnetic radiation;   assessing the failure based on at least one of the first sensor device, the second sensor device, or the third sensor device capturing the unique emission of electromagnetic radiation; and   recalibrating the first sensor device based on the assessment.   
     
     
         10 . The system of  claim 9 , wherein assessing the failure includes:
 detecting that the first sensor device, the second sensor device, and the third sensor device each capture the unique emission of electromagnetic radiation; and   based on the detecting, determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device.   
     
     
         11 . The system of  claim 10 , wherein the recalibrating includes determining a changed physical position of the first sensor device by data from the first sensor device and at least one of the second sensor device and third sensor device capturing electromagnetic radiation emitted from other active markers. 
     
     
         12 . The system of  claim 10 , wherein determining a changed physical position of the first sensor device by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula. 
     
     
         13 . The system of  claim 9 , wherein assessing the failure includes:
 detecting that at least one the second sensor device and the third sensor device captures the unique emission of electromagnetic radiation and the first sensor device fails to capture the unique emission of electromagnetic radiation; and   repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated.   
     
     
         14 . The system of  claim 9 , wherein the failure of the first sensor device is based on a change of at least one intrinsic parameter including at least one of a change in focal length, optical center, principal point, skew of axis, and lens distortion. 
     
     
         15 . A non-transitory computer-readable storage medium carrying program instructions thereon for recalibrating a sensor device to capture electromagnetic radiation from an active marker in performance capture, the program instructions when executed by one or more processors cause the one or more processors to perform operations comprising:
 providing a first sensor device, a second sensor device, and a third sensor device, each of the sensor devices being initially calibrated at a recording site to capture marker data for three-dimensional (3-D) coordinates of an active marker;   determining a failure of the first sensor device;   in response to determining the failure, signaling for the active marker to emit a unique emission of electromagnetic radiation;   assessing the failure based on at least one of the first sensor device, the second sensor device, or the third sensor device capturing the unique emission of electromagnetic radiation; and   recalibrating the first sensor device based on the assessment.   
     
     
         16 . The computer-readable storage medium of  claim 15 , wherein assessing the failure includes:
 detecting that the first sensor device, the second sensor device, and the third sensor device each capture the unique emission of electromagnetic radiation; and   based on the detecting, determining that the first sensor device is miscalibrated and the active marker is in a field of view of the first sensor device.   
     
     
         17 . The computer-readable storage medium of  claim 16 , wherein the recalibrating includes determining a changed physical position of the first sensor device by data from the first sensor device and at least one of the second sensor device and third sensor device capturing electromagnetic radiation emitted from other active markers. 
     
     
         18 . The computer-readable storage medium of  claim 16 , wherein determining a changed physical position of the first sensor device by applying the 3-D coordinates of the active marker and corresponding two-dimensional (2-D) coordinates of the first sensor device to a perspective-n-point pose problem formula. 
     
     
         19 . The computer-readable storage medium of  claim 15 , wherein assessing the failure includes:
 detecting that at least one the second sensor device and the third sensor device captures the unique emission of electromagnetic radiation and the first sensor device fails to capture the unique emission of electromagnetic radiation; and   repositioning the first sensor device to a changed physical position to have a field of view region including the active marker and overlapping with a field of view of the second sensor device or the third sensor device, thereby causing the first sensor device to be miscalibrated.   
     
     
         20 . The computer-readable storage medium of  claim 15 , the failure of the first sensor device is based on a change of at least one intrinsic parameter including at least one of a change in focal length, optical center, principal point, skew of axis, and lens distortion.