Imaging device control via multiple input modalities

Abstract

Techniques are disclosed for imaging device control in a computer-assisted device that includes a repositionable structure configured to support an imaging device, and a control system configured to determine a position of a first reference point associated with a first input modality; in a first mode, determine a position of a target reference point for the imaging device based on the first reference point; in a second mode, determine a position of a second reference point associated with a second input modality, and determine the position of the target reference point based on the position of the first reference point and the position of the second reference point; determine a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and cause actuation of the repositionable structure based on the movement.

1 . A computer-assisted device comprising:
 a first repositionable structure configured to support an imaging device; and   a control system coupled to the first repositionable structure;   wherein the control system is configured to:
 determine a position of a first reference point associated with a first input modality; 
 in a first mode, determine a position of a target reference point for the imaging device based on at least the first reference point; 
 in a second mode, determine a position of a second reference point associated with a second input modality, and determine the position of the target reference point based on at least the position of the first reference point and the position of the second reference point; 
 determine a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and 
 cause actuation of the first repositionable structure based on the determined movement. 
   
     
     
         2 . (canceled) 
     
     
         3 . The computer-assisted device of  claim 1 , wherein the first input modality comprises manipulation of one of a hand-operated controller or a display unit, and the second input modality comprises manipulation of the other of the hand-operated controller or the display unit. 
     
     
         4 . The computer-assisted device of  claim 3 , wherein the first input modality comprises the manipulation of the hand-operated controller, and wherein the second input modality comprises the manipulation of the display unit. 
     
     
         5 . The computer-assisted device of  claim 3 , wherein the first reference point is offset from the display unit by being in front of the display unit in a direction away from an operator, wherein the hand-operated controller comprises a first input control and a second input control, wherein the second reference point is at a midpoint between the first and second input controls, and wherein the third reference point is offset from the imaging device by being in front of the imaging device in a direction of a field of view of the imaging device. 
     
     
         6 . The computer-assisted device of  claim 3 , wherein the imaging device and the first repositionable structure configured to support the imaging device together have a first plurality of degrees of freedom, wherein the display unit and a second repositionable structure configured to support the display unit together have a second plurality of degrees of freedom different from the first plurality of degrees of freedom, and wherein an input control of the hand-operated controller and a third repositionable structure configured to support the input control together have a third plurality of degrees of freedom different from the first plurality of degrees of freedom. 
     
     
         7 . The computer-assisted device of  claim 6 , wherein the first plurality of degrees of freedom differ in number from the second plurality of degrees of freedom and the third plurality of degrees of freedom, or wherein the first plurality of degrees of freedom includes one or more degrees of freedom that is not matched by the second plurality of degrees of freedom or the third plurality of degrees of freedom. 
     
     
         8 . The computer-assisted device of  claim 3 , further comprising: another repositionable structure configured to support a tool, wherein the control system is further configured to:
 in a third mode, cause actuation of the another repositionable structure to move the tool in response to input received via the second input modality.   
     
     
         9 . The computer-assisted device of  claim 1 , wherein the control system is further configured to: in a third mode, determine the position of the second reference point associated with the second input modality, determine a position of a fourth reference point associated with a third input modality, and determine the position of the target reference point based on at least the position of the first reference point, the position of the second reference point, and the position of the fourth reference point. 
     
     
         10 . The computer-assisted device of  claim 1 , wherein, to determine the position of the target reference point based on at least the position of the first reference point and the position of the second reference point, the control system is configured to add a first scaled difference to a second scaled difference, the first scaled difference being a difference between the position of the first reference point and a baseline reference point of the first input modality scaled by a first scaling factor, and the second scaled difference being a difference between the position of the second reference point and a baseline reference point of the second input modality scaled by a second scaling factor. 
     
     
         11 . (canceled) 
     
     
         12 . The computer-assisted device of  claim 10 , wherein the control system is further configured to:
 determine the baseline reference point of the first input modality based on the position of the first reference point at an entry to the first mode;   determine the baseline reference point of the second input modality based on the position of the second reference point at an entry to a third mode; and   determine the baseline reference point of the imaging device based on a configuration of the imaging device at the entry to the first mode or the entry to the third mode.   
     
     
         13 . The computer-assisted device of  claim 12 , wherein the control system is further configured to:
 in the third mode, determine the position of the second reference point associated with the second input modality, and determine the position of the target reference point based on at least the position of the second reference point.   
     
     
         14 . The computer-assisted device of  claim 1 , wherein the control system is further configured to:
 determine whether the target reference point can be achieved using a virtual follower device; and   in response to determining that the target reference point cannot be achieved:
 determine haptic feedback based on a difference between an achievable position of the third reference point and the target reference point, 
 determine joint forces or torques for joints in a second repositionable structure configured to support the first input modality or a third repositionable structure configured to support the second input modality based on the haptic feedback, and 
 cause actuation of the joints of the second repositionable structure or the third repositionable structure based on the determined joint forces or torques. 
   
     
     
         15 . The computer-assisted device of  claim 1 , wherein the control system is further configured to, in response to determining that the first repositionable structure cannot achieve the determined movement:
 determine haptic feedback based on a difference between an achievable movement of the first repositionable structure and the determined movement;   determine joint forces or torques for joints in a second repositionable structure configured to support the first input modality based on the haptic feedback; and   cause actuation of the joints of the second repositionable structure based on the determined joint forces or torques.   
     
     
         16 . The computer-assisted device of  claim 1 , wherein the control system is further configured to:
 in response to identifying a manual adjustment to the first input modality:
 set a first baseline reference point of the first input modality to the position of the first reference point at a completion of the manual adjustment to the first input modality, 
 set a first correction based on a difference between the second reference point and a first baseline reference point of the second input modality and on a difference between the third reference point and a first baseline reference point of the imaging device, and 
 reduce the first correction over one or more movement cycles; or 
   in response to identifying a manual adjustment to the second input modality:
 set a second baseline reference point of the second input modality to the position of the second reference point at a completion of the manual adjustment to the second input modality, 
 set a second correction based on a difference between the first reference point and a second baseline reference point of the first input modality, and on a difference between the third reference point and a second baseline reference point of the imaging device, and 
 reduce the second correction over one or more movement cycles; or 
   in response to identifying a manual adjustment to the imaging device:
 set a third baseline reference point of the imaging device to the position of the third reference point at a completion of the manual adjustment to the imaging device, 
 set a third correction based on a difference between the first reference point and a third baseline reference point of the first input modality, and on a difference between the second reference point and a third baseline reference point of the second input modality, and 
   reduce the third correction over one or more movement cycles.   
     
     
         17 - 27 . (canceled) 
     
     
         28 . A method of operating a computer-assisted device comprising a first repositionable structure and one or more processors, the one or more processors communicatively coupled to the first repositionable structure, the method comprising:
 determining, by the one or more processors, a position of a first reference point associated with a first input modality;   in a first mode, determining, by the one or more processors, a position of a target reference point for an imaging device supported by a first repositionable structure of a computer-assisted device based on at least the first reference point;   in a second mode, determining, by the one or more processors, a position of a second reference point associated with a second input modality, and determining the position of the target reference point based on at least the position of the first reference point and the position of the second reference point;   determining, by the one or more processors, a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and   causing, by the one or more processors, actuation of the first repositionable structure based on the determined movement.   
     
     
         29 . The method of  claim 28 , wherein the first input modality comprises manipulation of one of a hand-operated controller or a display unit, and the second input modality comprises manipulation of the other of the hand-operated controller or the display unit. 
     
     
         30 . The method of  claim 29 , wherein the first reference point is offset from the display unit by being in front of the display unit in a direction away from an operator, wherein the hand-operated controller comprises a first input control and a second input control, wherein the second reference point is at a midpoint between the first and second input controls, and wherein the third reference point is offset from the imaging device by being in front of the imaging device in a direction of a field of view of the imaging device. 
     
     
         31 . The method of  claim 29 , further comprising:
 in a third mode, causing actuation of another repositionable structure to move a tool in response to input received via the second input modality, wherein the another repositionable structure is configured to support the tool.   
     
     
         32 . The method of  claim 28 , wherein determining the position of the target reference point based on at least the position of the first reference point and the position of the second reference point comprises:
 adding a first scaled difference to a second scaled difference, the first scaled difference being a difference between the position of the first reference point and a baseline reference point of the first input modality scaled by a first scaling factor, and the second scaled difference being a difference between the position of the second reference point and a baseline reference point of the second input modality scaled by a second scaling factor.   
     
     
         33 - 36 . (canceled) 
     
     
         37 . The method of  claim 28 , further comprising,
 in response to identifying a manual adjustment to the first input modality:
 setting a first baseline reference point of the first input modality to the position of the first reference point at a completion of the manual adjustment to the first input modality, 
 setting a first correction based on a difference between the second reference point and a first baseline reference point of the second input modality, and on a difference between the third reference point and a first baseline reference point of the imaging device, and 
   reducing the first correction over one or more movement cycles; or   in response to identifying a manual adjustment to the second input modality:
 setting a second baseline reference point of the second input modality to the position of the second reference point at a completion of the manual adjustment to the second input modality, 
 setting a second correction based on a difference between the first reference point and a second baseline reference point of the first input modality, and on a difference between the third reference point and a second baseline reference point of the imaging device and 
 reducing the second correction over one or more movement cycles; or 
   in response to identifying a manual adjustment to the imaging device:
 setting a third baseline reference point of the imaging device to the position of the third reference point at a completion of the manual adjustment to the imaging device, 
 setting a third correction based on a difference between the first reference point and a third baseline reference point of the first input modality, and on a difference between the second reference point and a third baseline reference point of the second input modality, and 
 reducing the third correction over one or more movement cycles. 
   
     
     
         38 - 44 . (canceled) 
     
     
         45 . One or more non-transitory machine-readable media comprising a plurality of machine-readable instructions which when executed by one or more processors of a computer-assisted device comprising a first repositionable structure, are adapted to cause the one or more processors to perform a method comprising:
 determining a position of a first reference point associated with a first input modality;   in a first mode, determining a position of a target reference point for an imaging device supported by a first repositionable structure of a computer-assisted device based on at least the first reference point;   in a second mode, determining a position of a second reference point associated with a second input modality, and determining the position of the target reference point based on at least the position of the first reference point and the position of the second reference point;   determining a movement of the first repositionable structure that moves the imaging device such that a third reference point associated with the imaging device moves toward the target reference point; and   causing actuation of the first repositionable structure based on the determined movement.   
     
     
         46 . The one or more non-transitory machine-readable media of  claim 45 , wherein:
 the first input modality comprises manipulation of one of a hand-operated controller or a display unit, and the second input modality comprises manipulation of the other of the hand-operated controller or the display unit; and   the first reference point is offset from the display unit by being in front of the display unit in a direction away from an operator, wherein the hand-operated controller comprises a first input control and a second input control, wherein the second reference point is at a midpoint between the first and second input controls, and wherein the third reference point is offset from the imaging device by being in front of the imaging device in a direction of a field of view of the imaging device.   
     
     
         47 . The one or more non-transitory machine-readable media of  claim 45 , wherein determining the position of the target reference point based on at least the position of the first reference point and the position of the second reference point comprises:
 adding a first scaled difference to a second scaled difference, the first scaled difference being a difference between the position of the first reference point and a baseline reference point of the first input modality scaled by a first scaling factor, and the second scaled difference being a difference between the position of the second reference point and a baseline reference point of the second input modality scaled by a second scaling factor.