Virtual reality system for simulating a robotic surgical environment

Abstract

A virtual reality system may generate a virtual robotic surgical environment using a client application, where the virtual robotic surgical environment includes at least one virtual robotic component. In response to a user input to move the at least one virtual robotic component in the virtual robotic surgical environment, the system may pass status information regarding the at least one virtual robotic component from the client application to a server application, generate an actuation command based on the user input and the status information using the server application, pass the actuation command from the server application to the client application, and move the at least one virtual robotic component based on the actuation command. The client application and the server application may be run on a shared processor device, or on separate processor devices

1 . A virtual reality system for simulating a robotic surgical environment, the virtual reality system comprising one or more processors configured to:
 render a computer-generated virtual surgical environment comprising a virtual robotic arm which is generated based on a kinematic model of a real robotic arm;   receive a user input generated from a handheld device to move the virtual robotic arm, the handheld device having one or more sensors and an interactive feature to sense the user input;   in response to the user input, compute a motion of the virtual robotic arm based on the user input and a position of the virtual robotic arm; and   render the motion of the virtual robotic arm to a display.   
     
     
         2 . The virtual reality system of  claim 1 , wherein the one or more processors are further configured to
 detect a movement of a user based on the one or more sensors of the handheld device or from a sensor of the display; and   position the user within the computer-generated virtual surgical environment based on the movement.   
     
     
         3 . The virtual reality system of  claim 1 , wherein the one or more processors are further configured rotate a view of the computer-generated virtual surgical environment around a current vantage point of the user, in response to the user dragging selecting and dragging a pint in the computer-generated virtual surgical environment with the handheld device. 
     
     
         4 . The virtual reality system of  claim 1 , wherein the one or more processors are further configured to
 generate a virtual camera at a specified location within the computer-generated virtual surgical environment in response to a second user input through the handheld device; and   render, to the display, a camera view which is associated with the virtual camera.   
     
     
         5 . The virtual reality system of  claim 4 , wherein the virtual camera includes an endoscopic camera which is attached to a virtual endoscope in a virtual patient. 
     
     
         6 . The virtual reality system of  claim 4 , wherein the virtual camera includes a wide-angle camera. 
     
     
         7 . The virtual reality system of  claim 4 , wherein the one or more processors are further configured to highlight, in the camera view, a tumor or a perfusion of a virtual patient. 
     
     
         8 . The virtual reality system of  claim 1 , wherein the one or more processors are further configured to navigate the user in the computer-generated virtual surgical environment in a flight mode, based on inputs sensed by an interactive feature or one or more sensors of the handheld device. 
     
     
         9 . The virtual reality system of  claim 8 , wherein, in the flight mode, the interactive feature of the handheld device includes a directional pad or touchpad that translates to a flight of the user in the computer-generated virtual surgical environment. 
     
     
         10 . A computer-implemented method for operating a computer-generated virtual operating room, comprising:
 rendering a computer-generated virtual surgical environment comprising a virtual robotic arm which is generated based on kinematic models of a real robotic arm;   receiving a user input generated from two handheld devices to move the virtual robotic arm, the two handheld devices each having one or more sensors and an interactive feature to sense the user input;   in response to the user input, computing a motion of the virtual robotic arm based on the user input and a position of the virtual robotic arm; and   rendering the motion of the virtual robotic arm to a display.   
     
     
         11 . The computer-implemented method of  claim 10 , further comprising:
 detecting a movement of a user based on the one or more sensors of the two handheld devices or from a sensor of the display; and   positioning the user within the computer-generated virtual surgical environment based on the movement.   
     
     
         12 . The computer-implemented method of  claim 11 , further comprising rotating a view of the computer-generated virtual surgical environment around a current vantage point of the user, in response to the user dragging selecting and dragging a pint in the computer-generated virtual surgical environment with the two handheld devices. 
     
     
         13 . The computer-implemented method of  claim 11 , further comprising adjusting a scale of the computer-generated virtual surgical environment as-displayed to the user based on a difference in distance between the two handheld devices. 
     
     
         14 . The computer-implemented method of  claim 11 , further comprising:
 generating a virtual camera at a specified location within the computer-generated virtual surgical environment in response to a second user input through the two handheld devices; and   rendering, to the display, a camera view which is associated with the virtual camera.   
     
     
         15 . The computer-implemented method of  claim 14 , wherein the virtual camera includes an endoscopic camera which is attached to a virtual endoscope in a virtual patient. 
     
     
         16 . The computer-implemented method of  claim 14 , wherein the virtual camera includes a wide-angle camera. 
     
     
         17 . The computer-implemented method of  claim 14 , further comprising highlighting, in the camera view, a tumor or a perfusion of a virtual patient. 
     
     
         18 . The computer-implemented method of  claim 11 , further comprising navigating the user in the computer-generated virtual surgical environment in a flight mode, based on inputs sensed by the interactive feature or the one or more sensors of the two handheld devices. 
     
     
         19 . The computer-implemented method of  claim 18 , wherein, in the flight mode, the interactive feature of the two handheld devices includes a directional pad or touchpad that translates to a flight of the user in the computer-generated virtual surgical environment. 
     
     
         20 . A virtual reality system for simulating a robotic surgical environment, the virtual reality system comprising one or more processors configured to:
 render a computer-generated virtual surgical environment comprising a virtual robotic arm which is generated based on a kinematic model of a real robotic arm;   receive a user input generated from two handheld devices to move the virtual robotic arm, the two handheld devices each having one or more sensors and an interactive feature to sense the user input;   in response to the user input, compute a motion of the virtual robotic arm based on the user input and a position of the virtual robotic arm; and   render the motion of the virtual robotic arm to a display.