Systems and methods for dynamic identification of a surgical tray and the items contained thereon

Abstract

The invention provides artificial intelligence-enabled image recognition methods and systems for continuously training a computer system to accurately identify a surgical item in a tray using at least 100 randomly created 2-dimensional images of a 3-dimensional synthetic item having unique identifiers assigned to the images or item. The invention also provides an artificial intelligence-enabled image recognition method and system for use to determine whether surgical instruments are present or missing on a surgical tray, and, if applicable, identifying those missing. In one aspect, a server receives an image and analyzes the image with a deep convolutional neural network to classify the type of tray and then compares a list of items that should be on the tray to that which the computer recognizes on the tray to generate an output displayed to a user identifying the items present and/or missing.

What is claimed is: 
     
         1 . A method for training a computer system to dynamically identify a surgical tray and items contained thereon, the method comprising:
 a. scanning a surgical instrument with a scanner device at least two times to create a preliminary 3-dimensional model of the surgical instrument;   b. revising the preliminary 3-dimensional model of the surgical instrument to create a final 3-dimensional synthetic item by defining at least one element selected from the group consisting of: geometry, position of each vertex, UV position of each texture coordinate vertex, vertex normals, faces that make each polygon defined as a list of vertices, and texture coordinates for the item;   c. assigning a unique identification to the final 3-dimensional synthetic item, wherein the final 3-dimensional synthetic item with the unique identification is stored in a database;   d. creating at least one hundred unique training synthetic images of the final 3-dimensional synthetic item, each of the unique training synthetic images differs from the final 3-dimensional synthetic item by randomly varying at least one element of the final 3-dimensional synthetic item selected from the group consisting of: orientation of the final 3-dimensional synthetic item, synthetic light color or intensity illuminating the final 3-dimensional synthetic item, and elevation of the final 3-dimensional synthetic item above an identified surface, wherein each of the at least one hundred unique training synthetic images is linked to the unique identification of the final 3-dimensional synthetic item;   e. creating at least one unique test synthetic image of the final 3-dimensional synthetic item, the unique test synthetic image differs from the final 3-dimensional synthetic item by randomly varying at least one element of the final 3-dimensional synthetic item selected from the group consisting of: orientation of the final 3-D synthetic item, synthetic light color or intensity illuminating the final 3-D synthetic item, and elevation of the final 3-D synthetic item above an identified surface, wherein the unique test synthetic image is not linked to the unique identification of the final 3-dimensional synthetic item;   f. repeatedly processing the training synthetic images with the system so the system identifies one or more patterns in the training synthetic images, the patterns are used to create and update an identification model linked to the unique identification of the 3-dimensional synthetic item;   g. processing the synthetic test images with the system so the system identifies the 3-dimensional synthetic item from the synthetic test images based on the identification model, and the system provides a numeric confidence factor representing confidence that the system has correctly identified the 3-dimensional synthetic item; and   h. determining if the numeric confidence factor is equal to or greater than a pre-set confidence factor uploaded into the system and repeating steps d through h if the identification is not correct or the numeric confidence factor is less than the pre-set confidence factor.   
     
     
         2 . The method of  claim 1 , at least one of steps b through h are accomplished automatically without user input. 
     
     
         3 . The method of  claim 2 , wherein steps d through h are accomplished automatically using a computer vision-driven artificial intelligence network. 
     
     
         4 . The method of  claim 2 , wherein the computer vision-driven artificial intelligence network is a convolutional neural network. 
     
     
         5 . The method of  claim 1  further comprising:
 i. when the system correctly identifies the 3-dimensional synthetic item and the numeric confidence factor is equal to or greater than a pre-set confidence factor, uploading the identification model attributable to the unique identification to a server for deployment. 
 
     
     
         6 . The method of  claim 1 , wherein step d continuously generates new unique training synthetic images of the final 3-dimensional synthetic item by varying at least one element selected from the group consisting of: orientation of the final 3-D synthetic model, synthetic light color or intensity illuminating the final 3-D synthetic model, and elevation of the final 3-D synthetic model above an identified surface, wherein each new unique training synthetic image is linked to the unique identification of the final 3-dimensional synthetic item. 
     
     
         7 . The method of  claim 1 , wherein step e continuously generates new unique test synthetic images of the final 3-dimensional synthetic item by varying at least one element selected from the group consisting of: orientation of the final 3-D synthetic model, synthetic light color or intensity illuminating the final 3-D synthetic model, and elevation of the final 3-D synthetic model above an identified surface, wherein the new unique test synthetic images are not linked to the unique identification of the final 3-dimensional synthetic item. 
     
     
         8 . The method of  claim 1 , wherein the numeric confidence factor is greater than 95 percent. 
     
     
         9 . A system for dynamically identifying a surgical tray and items contained thereon, the system comprising:
 a software application, the software application operating on a mobile computer device or a computer device in communication with at least one image data collection device configured to produce an image of the surgical tray, the software application is configured to receive the image of the surgical tray from the image data collection device and then communicate the image through a wired and/or wireless communication network to a server located at a site where the surgical tray is located or at a location remote from the site; and   a processor in communication through the wired and/or wireless communication network with the software application, as well as the server, the processer is configured to call up from a library database of the system, upon communication of the image to the server:
 a plurality of tray identification models comprised of tensors, the tray identification models previously uploaded by an administrator of the system or an employee, contractor, or agent of the administrator; 
 whereby the processor is configured to:
 analyze the image and classify the type of tray in the image based on the tray identification models as applied to the image, 
 call up from the library database:
 a list of items linked to the classification of the type of tray, and 
 a plurality of instrument identification models comprised of tensors, the instrument identification models linked to the items and were previously uploaded by an administrator of the system or an employee, contractor, or agent of the administrator; 
 
 analyze the image and identify the type of items in the image based on the instrument identification models, 
 compare the classified items to the list of items linked to the classified tray to determine any missing items, and 
 notify the software application of the classified items and any missing items. 
 
   
     
     
         10 . The system of  claim 9  wherein the image data collection device is a camera. 
     
     
         11 . The system of  claim 9  wherein the image data collection device is mounted on a wearable device. 
     
     
         12 . The system of  claim 9 , wherein the tray identification models comprised of tensors and the instrument identification models comprised of tensors are generated using an computer vision-driven artificial intelligence network trained using 2-dimensional views of a 3-dimensional synthetic item, as rendered by a view generation module. 
     
     
         13 . The system of  claim 12 , wherein the artificial intelligence network is a convolutional neural network. 
     
     
         14 . The system of  claim 12 , wherein the computer vision-driven artificial intelligence network is continuously trained using 2-dimensional views of a 3-dimensional synthetic item, as rendered by a view generation module. 
     
     
         15 . A method for identifying a surgical tray and items contained thereon, the method comprising:
 receiving an image of the surgical tray and items contained thereon from an image data collector connected to a server or a remote server using a software application operating on a mobile computer device or a computer device that may be synced with the mobile computer device, and wherein the mobile computer device or the computer device communicate through a wired and/or wireless communication network with the server at a site the surgical tray is located at or with a remote server in a location that is remote to the site and in communication with the server;   upon receiving the information, calling up from a database using a processor: a plurality of tray identification models comprised of tensors, wherein the tray identification models have been previously uploaded by a professional with knowledge of tray names and items intended to be contained by those trays;   analyzing the image and classifying the type of tray in the image based on the tray identification models;   upon classifying the tray, calling up from the database a plurality of instrument identification models linked to the classification of the tray and comprised of tensors, the instrument identification models including: (a) surface texture, (b) item material composition, and (c) a size tolerance; and a list of items linked to the tray classification; wherein the instrument identification model have been previously uploaded by the professional;   analyzing the image and classify the type of items in the image based on the instrument identification models;   comparing the classified items to the list of items linked to the classified tray to determine any missing items, and   notifying the software application of the classified items and any missing items.   
     
     
         16 . The method of  claim 15  wherein the image data collection device is a camera. 
     
     
         17 . The method of  claim 15  wherein the image data collection device is mounted on a wearable device. 
     
     
         18 . The method of  claim 15 , wherein the tray identification models comprised of tensors and the instrument identification models comprised of tensors are generated using an computer vision-driven artificial intelligence network trained using 2-dimensional views of a a 3-dimensional synthetic item, as rendered by a view generation module. 
     
     
         19 . The method of  claim 18 , wherein the artificial intelligence network is a convolutional neural network. 
     
     
         20 . The method of  claim 18 , wherein the computer vision-driven artificial intelligence network is continuously trained using 2-dimensional views of a 3-dimensional synthetic item, as rendered by a view generation module.