System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom)

Abstract

The invention relates generally to a medical apparatus and method of using the same for receiving and transmitting streaming medical imagery and audio signals in real time, and allowing remote operators to annotate and telestrate with same. The invention acquires streaming medical imagery and audio signals through a telestreamer input device, allowing users to electronically collaborate, generally by telestrating, annotating, and sketching image overlays on streaming medical imagery. Video images of streaming imagery data displayed on a monitor are superimposed onto a virtual mesh projected via computer graphics. The vertices of the mesh move according to equations of motion based on a computational physics engine. Virtual tools are projected above the mesh via computer graphics. These virtual tools interact with the virtual mesh according to the physics engine. The superposition of the video images onto the virtual mesh makes it appear that points within the video image are moving in a realistic manner and reacting to the virtual tools with a realistic response. The invention allows for recursive superposition of mesh layers, also known as ‘surgi-skins’, and creation of a multi-layered virtual mesh. Multi-layered surgi-skins synthesized from multi-modal streaming medical imagery and saved together with multi-dimensional [4-D] virtual mesh and multi-sensory annotation in single file format as DICOM files are also known as ‘DICOM mesh’ or ‘haptic holograms’.

1 . A method to perform video annotation using an augmented reality telestrator. 
     
     
         2 . The method of  claim 1 , wherein the video images are projected onto a virtual mesh which moves based on a physically-realistic computational model of the actual object being displayed in the video images. In the current embodiment, the virtual mesh is constructed in computer graphics as a rectangle made from equilateral triangles whose vertices are interconnected. Movement between vertices of the virtual mesh is calculated via physics-based calculations, including but not limited to Hooke's spring law and Newton's laws of motion. A UV-map is constructed to superimpose the video images onto the virtual mesh. As the vertices of the virtual mesh move based on the physics-based calculations, the superimposed video images are transformed in the corresponding positions. Physical parameters of the mesh and tools, including multi-sensory (e.g. haptic) feedback may be saved using standardized file formats (e.g. COLLADA) to be used in conjunction with the DICOM medical imagery. 
     
     
         3 . The method of  claim 1 , wherein computer-generated, virtual tools are overlaid on the video images and can manipulate the images in a realistic manner based on the physically-realistic model mesh of  claim 2 . In the current embodiment, these virtual tools are three-dimensional rendering of scissors, sutures, and forceps which can be used to cut, stitch, and push/pull/twist points within the video images. In effect, these video images appear to react in a realistic manner to the virtual tools based on the physics-based calculations of the virtual mesh of  claim 2 . 
     
     
         4 . A network system apparatus allowing users to capture, retrieve and view both real time and archived medical images for synchronous or asynchronous communication, collaboration and consultation by one or more users using illustrations over the medical images, comprising: a telenetwork server including at least one associated database having the capability to communicate with a local area network; and at least one telestreamer in network communication with a telenetwork server via a local area network wherein the telestreamers capture one or more medical images and provide the medical images via the network communication to the telenetwork server via the local area network as it receives medical images from at least one source 
     
     
         5 . A network system apparatus system allowing users to capture, retrieve and view both real time and archived medical imagery streams for synchronous or asynchronous communication, collaboration and consultation by one or more users using illustrations over the medical images as in  claim 4  wherein; one or more users retrieve and view the medical images, creating illustrations over the medical images such as, but not limited to drawing, annotating, telestrating; and storing the medical images with the annotations, and saving all the medical images together with annotations and metadata on the telenetwork server, on a picture archiving and communications system server, in a known digital imaging and communications in medicine format. 
     
     
         6 . A method for allowing one or more users to capture, retrieve and view both real time and archived medical images for synchronous or asynchronous communication, collaboration and consultation by one or more users using illustrations over the medical images comprising; running a computer program, storing the program on each of user's computers, displaying the graphical user interface output of that program on a computer display; linking each user's computer to a telenetwork server using a local area network, each user communicating with the telenetwork server, the telemedicine image management system server providing permission to each user wherein linking the user to a digital imaging and communications in medicine modality worklist utility, a medical image archive server, and telestreamers for capturing, retrieving and viewing medical images, user's illustrating over the medical images, telenetwork server managing all illustration file sharing wherein new user's illustrations are appending periodically to the telenetwork server, maintaining the file on the user's computer, the telenetwork server linking to the internet and other users, wherein the images remain on the telenetwork server and the user illustrations are appended to that telenetwork server; streaming images into a local area network wherein the telenetwork server having associated database in communication with telestreamers streamers connected directly to medical imaging modalities for acquiring one or more medical images, streaming those medical images to the local area network; 
     
     
         7 . A method for allowing users to capture, retrieve and view both real time and archived medical imagery streams for synchronous or asynchronous communication, collaboration and consultation by one or more users using illustrations over the medical images as in  claim 6  wherein; one or more users retrieve and view the medical images, creating illustrations over the medical images such as, but not limited to drawing, annotating, telestrating and storing the medical images with the illustrations, and viewing all of the user's illustrations as they happen and can save all the illustrations from all participant clients on their local respective computer storage devices, on the telenetwork server, on a picture archiving and communications system, in a known digital imaging and communications in medicine format.