Systems and methods of interacting with a robotic tool using free-form gestures

Abstract

The technology disclosed relates to motion capture and gesture recognition. In particular, it calculates the exerted force implied by a human hand motion and applies the equivalent through a robotic arm to a target object. In one implementation, this is achieved by tracking the motion and contact of the human hand and generating corresponding robotic commands that replicate the motion and contact of the human hand on a workpiece through a robotic tool.

What is claimed is: 
     
         1 . A method of using gestures to control a robotic tool to manipulate a workpiece, the method including:
 capturing sequential images of a hand in a three-dimensional (3D) sensory space while the hand is (i) empty and (ii) making motions directed to commanding the robotic tool;   recognizing, from the captured images of the hand while empty, a gesture segment of the hand that represents a manipulation of a workpiece by the robotic tool, wherein there is no physical contact between the hand and any object during the recognized gesture segment of the hand;   determining, according to the recognized gesture segment, a command to cause the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on a motion of the hand during the recognized gesture segment; and   issuing the determined command to the robotic tool to apply the force to the workpiece.   
     
     
         2 . The method of  claim 1 , further including capturing edge information for fingers of the hand that performs the gesture segment and computing finger positions of a 3D solid hand model for the hand during the gesture segment. 
     
     
         3 . The method of  claim 2 , further including:
 using the 3D solid hand model to capture a curling of the hand during the gesture segment; and   interpreting the curling as a parameter of robotic tool translation.   
     
     
         4 . The method of  claim 3 , further including:
 using the 3D solid hand model to detect the curling as an extreme degree of motion of the hand during the gesture segment; and   responsive to detecting the curling as an extreme degree of motion of the hand, interpreting a maximum value of a parameter of robotic tool actuation.   
     
     
         5 . The method of  claim 4 , wherein the maximum value of the parameter is an amplification function of the extreme degree of motion. 
     
     
         6 . The method of  claim 4 , wherein the maximum value of the parameter is a polynomial function of the extreme degree of motion. 
     
     
         7 . The method of  claim 4 , wherein the maximum value of the parameter is a transcendental function of the extreme degree of motion. 
     
     
         8 . The method of  claim 4 , wherein the maximum value of the parameter is a step function of the extreme degree of motion. 
     
     
         9 . The method of  claim 2 , further including:
 using the 3D solid hand model to capture a torsion of the hand during the gesture segment; and   interpreting the torsion as a parameter of robotic tool actuation.   
     
     
         10 . The method of  claim 9 , further including:
 using the 3D solid hand model to detect the torsion as an extreme degree of motion of the hand during the gesture segment; and   responsive to detecting the torsion as the extreme degree of motion, interpreting a maximum value of a parameter of robotic tool actuation.   
     
     
         11 . The method of  claim 10 , wherein the maximum value of the parameter is an amplification function of the extreme degree of motion. 
     
     
         12 . The method of  claim 11 , wherein the maximum value of the parameter is a polynomial function of the extreme degree of motion. 
     
     
         13 . A system comprising a processor and a memory storing computer instructions that, when executed on the processor, cause the processor to perform the method of  claim 1 . 
     
     
         14 . A non-transitory computer-readable recording medium having computer instructions recorded thereon, the computer instructions, when executed on a processor, causing the processor to perform the method of  claim 1 . 
     
     
         15 . A method of using gestures to control a robotic tool to manipulate a workpiece, the method including:
 capturing sequential images of a hand in a three-dimensional (3D) sensory space while the hand is (i) interacting with a manipulable object and (ii) making motions directed to commanding the robotic tool;   recognizing, from the captured images of the hand, a gesture segment of the hand that represents physical a manipulation of a workpiece by the robotic tool, wherein there is neither physical nor electrical connection facilitating passage of a signal between the manipulable object and the robotic tool during the recognized gesture segment;   determining, according to the recognized gesture segment, a command to cause the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force to be applied to the workpiece is determined based on motion of the hand during the recognized gesture segment; and   issuing the determined command to the robotic tool to apply the force to the workpiece.   
     
     
         16 . A system comprising a processor and a memory storing computer instructions that, when executed on the processor, cause the processor to perform the method of  claim 15 . 
     
     
         17 . A non-transitory computer-readable recording medium having computer instructions recorded thereon, the computer instructions, when executed on a processor, causing the processor to perform the method of  claim 15 . 
     
     
         18 . A method of a robotic tool manipulating a workpiece, the method including:
 receiving, by the robotic tool, a command to cause the robotic tool to apply a force to the workpiece without actual physical contact between the robotic tool and a hand of a user, the received command being determined by:
 capturing sequential images of a hand in a three-dimensional (3D) sensory space while the hand is (i) empty and (ii) making motions directed to commanding the robotic tool; and 
 recognizing, from the captured images of the hand while empty, a gesture segment of the hand that represents a manipulation of a workpiece by the robotic tool, wherein there is no physical contact between the hand and any object during the recognized gesture segment; and 
   applying the force to the workpiece, according to the received command, without actual physical contact between the robotic tool and the hand, wherein a magnitude of the force applied to the workpiece is determined based on a motion of the hand during the recognized gesture segment.   
     
     
         19 . A system comprising a processor and a memory storing computer instructions that, when executed on the processor, cause the processor to perform the method of  claim 18 . 
     
     
         20 . A non-transitory computer-readable recording medium having computer instructions recorded thereon, the computer instructions, when executed on a processor, causing the processor to perform the method of  claim 18 .