Navigated surgical system with eye to xr headset display calibration

Abstract

A camera tracking system for computer assisted navigation during surgery operatively determines a first pose of a second extended-reality (XR) headset relative to stereo tracking cameras located on a first XR headset based on first tracking information from the stereo tracking cameras. The camera tracking system determines a second pose of eyes of a user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset based on second tracking information from the stereo tracking cameras. The camera tracking system also calibrates an eye-to-display relationship defining pose of the eyes of the user wearing the second XR headset to a display device of the second XR headset based on the determined first and second poses. The camera tracking system also controls where symbols are displayed on the display device of the second XR headset based on the eye-to-display relationship.

What is claimed is: 
     
         1 . A method of computer assisted navigation during surgery comprising:
 receiving, from a reflective surface, a reflection of an extended-reality (XR) headset by stereo cameras of the XR headset, the XR headset having a see-through screen for displaying images for viewing by a user wearing the XR headset;   determining a pose of the user eyes relative to the XR headset based on the received reflection;   calibrating an eye-to-display relationship based on the determined pose of the eyes; and   controlling where symbols are displayed on the screen of the XR headset based on the eye-to-display relationship.   
     
     
         2 . The method of  claim 1 , wherein the step of determining includes determining the pose based on a tracking reference array attached to the XR headset and viewable by sensors of a navigation system. 
     
     
         3 . The method of  claim 1 , wherein the step of controlling includes adjusting an image displayed on the see-through screen of the XR headset based on the calibrated eye-to-display relationship. 
     
     
         4 . The method of  claim 3 , further comprising:
 obtaining a display-to-eye distortion transform relating optical distortion of real-world images passing through the see-through screen to where user eyes are posed relative to the see-through screen; and   further controlling where symbols are displayed on the see-through screen based on the eye-to-display relationship and the display-to-eye distortion transform.   
     
     
         5 . The method of  claim 1 , wherein determining a pose of the user eyes includes determining a pose of pupils of the eyes. 
     
     
         6 . The method of  claim 1 , wherein the step of receiving a reflection include receiving the reflection from a planar mirror. 
     
     
         7 . The method of  claim 1 , wherein determining a pose includes determining the pose based on the shape of the XR headset. 
     
     
         8 . The method of  claim 1 , wherein the step of determining a pose includes determining how far away the user is from the reflective surface and how far the user eyes are from the stereo cameras. 
     
     
         9 . The method of  claim 1 , wherein the step of controlling includes controlling where the symbols are overlaid on tracked real-world objects. 
     
     
         10 . The method of  claim 1 , wherein:
 the XR headset includes a tracking reference array viewable by sensors of a navigation system, and an image projector that projects images to be reflected by the see-through screen toward the user eyes;   the step of controlling includes projecting the symbols on the see-through screen to be reflected toward the user eyes.   
     
     
         11 . The method of  claim 1 , wherein the see-through screen is a semi-transparent screen that acts to combine real world image with the symbols. 
     
     
         12 . A method of computer assisted navigation during surgery comprising:
 providing an extended-reality (XR) headset having stereo cameras, an image projector and a see-through screen for reflecting images created by the image projector for viewing by a user wearing the XR headset and for transmitting real world images to the user;   receiving, from a reflective surface, a reflection of the XR headset by the stereo cameras of the XR headset worn by the user;   determining a pose of the user eyes relative to the XR headset based on the received reflection;   calibrating an eye-to-display relationship based on the determined pose of the eyes; and   controlling where symbols created by the image projector are displayed on the screen of the XR headset based on the eye-to-display relationship.   
     
     
         13 . The method of  claim 12 , wherein the step of determining includes determining the pose based on a tracking reference array attached to the XR headset and viewable by sensors of a navigation system. 
     
     
         14 . The method of  claim 12 , wherein the step of controlling includes adjusting an image projected onto the see-through screen of the XR headset based on the calibrated eye-to-display relationship. 
     
     
         15 . The method of  claim 14 , further comprising:
 obtaining a display-to-eye distortion transform relating optical distortion of real-world images passing through the see-through screen to where user eyes are posed relative to the see-through screen; and   further controlling where symbols are projected onto the see-through screen based on the eye-to-display relationship and the display-to-eye distortion transform.   
     
     
         16 . The method of  claim 12 , wherein determining a pose of the user eyes includes determining a pose of pupils of the eyes. 
     
     
         17 . The method of  claim 12 , wherein the step of receiving a reflection include receiving the reflection from a planar mirror. 
     
     
         18 . The method of  claim 12 , wherein determining a pose includes determining the pose based on the shape of the XR headset. 
     
     
         19 . The method of  claim 12 , wherein the step of determining a pose includes determining how far away the user is from the reflective surface and how far the user eyes are from the stereo cameras. 
     
     
         20 . The method of  claim 12 , wherein the step of controlling includes controlling where the symbols are overlaid on tracked real-world objects.