Automatic selection of structures-of-interest for lamella sample preparation

Abstract

Disclosed herein are scientific instrument support systems, as well as related methods, computing devices, and computer-readable media. For example, in some embodiments, a support apparatus is provided for a charged particle microscope. The support apparatus is configured to apply automated image processing to an image representing a lamella sample to segment the image into a plurality of segmented classes. The support apparatus is also configured to identify, based on the plurality of segmented classes, a subset of candidate structures-of-interest in the lamella sample and to select, from the subset of candidate structures-of-interest in the lamella sample, a selected structure-of-interest for milling. The support apparatus is also configured to set, based on the selected structure-of-interest for milling, at least one milling parameter for the scientific instrument. An automated method performed via a computing device for providing such scientific instrument support is also provided.

The invention claimed is: 
     
         1 . A support apparatus for a charged particle microscope, the support apparatus comprising:
 a first logic configured to apply automated image processing to an image representing a lamella sample to segment the image into a plurality of segmented classes;   a second logic configured to:
 identify, based on the plurality of segmented classes, a subset of candidate structures-of-interest in the lamella sample; and 
 select, from the subset of structures-of-interest in the lamella sample, a selected structure-of-interest for milling by eliminating, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest; 
   a third logic configured to set, based on the selected structure-of-interest for milling, at least one milling parameter for the charged particle microscope; and   wherein the plurality of segmented classes includes at least one selected from the group consisting of a plurality of structures-of-interest, a plurality of grid squares, a plurality of cracks, a plurality of contaminants, and a background.   
     
     
         2 . The support apparatus of  claim 1 , wherein the first logic, the second logic, and the third logic are implemented by a common computing device. 
     
     
         3 . The support apparatus of  claim 1 , wherein at least one of the first logic, the second logic, and the third logic are implemented in the charged particle microscope. 
     
     
         4 . The support apparatus of  claim 1 , wherein the first logic is further configured to:
 acquire the image representing the lamella sample by causing the charged particle microscope to scan the lamella sample with at least one selected from the group consisting of an electron beam of the charged particle microscope and an ion beam of the charged particle microscope.   
     
     
         5 . The support apparatus of  claim 1 , wherein the second logic is further configured to:
 determine, based on the plurality of grid squares, the plurality of cracks, and the plurality of contaminants, a subset of secondary grid squares, wherein each of the subset of secondary grid squares contains one or both of a crack or a contaminant;   determine, for each of the plurality of structures-of-interest, a position of the structure-of-interest and a size of the structure-of-interest;   determine, for each of the plurality of structures-of-interest, based on the position of the structure-of-interest and the size of the structure-of-interest, whether the structure-of-interest is contained within one of the plurality of grid squares; and   eliminate, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest by, for each of the plurality of structures-of-interest that is contained within one of the plurality of grid squares, selecting the structure-of-interest for inclusion in the subset of candidate structures-of-interest if the structure-of-interest is not located within one of the subset of secondary grid squares.   
     
     
         6 . The support apparatus of  claim 5 , wherein the second logic is further configured to:
 for each of the plurality of structures-of-interest that is contained within one of the plurality of grid squares:   determine a distance between the structure-of-interest and another of the plurality of structures-of-interest contained with the same grid square as the structure-of-interest; and   eliminate, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest by excluding the structure-of-interest from the subset of candidate structures-of-interest if the distance between the structure-of-interest and another of the plurality of structures-of-interest contained with the same grid square is below a structure-of-interest distance threshold.   
     
     
         7 . The support apparatus of  claim 5 , wherein the second logic is further configured to:
 select a selected structure-of-interest for milling by selecting the selected structure-of-interest based on which of the plurality of structures-of-interest that is contained within one of the plurality of grid squares is located closest to a center of the grid square containing the structure-of-interest.   
     
     
         8 . The support apparatus of  claim 1 , wherein the third logic is further configured to:
 a third logic configured to set at least one milling parameter for the charged particle microscope by setting at least one selecting from the group consisting of a milling angle, a milling time, and a milling pattern.   
     
     
         9 . The support apparatus of  claim 1 , wherein the third logic is further configured to:
 cause an ion beam instrument to mill the selected structure-of-interest in accordance with the at least one milling parameter.   
     
     
         10 . An automated method performed via a computing device for providing scientific instrument support, the method comprising:
 applying automated image processing to an image representing a lamella sample to segment the image into a plurality of segmented classes,   identifying, based on the plurality of segmented classes, a subset of candidate structures-of-interest in the lamella sample, wherein subset of candidate structures-of-interest includes a subset of candidate cells;   selecting, from the subset of candidate structures-of-interest in the lamella sample, a selected structure-of-interest for milling by eliminating, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest;   setting, based on the selected structure-of-interest for milling, at least one milling parameter for a charged particle microscope; and   wherein the plurality of segmented classes including at least two selected from the group consisting of a plurality of structures-of-interest, a plurality of grid squares, a plurality of cracks, a plurality of contaminants, and a background.   
     
     
         11 . The method of  claim 10 , further comprising:
 for each of the subset of candidate structures-of-interest:   determining a distance between the structure-of-interest and a grid bar of the grid square containing the structure-of-interest; and   wherein eliminating, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest includes excluding the structure-of-interest from the subset of candidate structures-of-interest if the distance between the structure-of-interest and the grid bar is below a grid bar distance threshold.   
     
     
         12 . The method of  claim 10 , wherein the plurality of segmented classes further includes a grid edge, and the method further comprises:
 for each of the subset of candidate structures-of-interest:   determining a distance between the structure-of-interest and the grid edge; and   wherein eliminating, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest includes excluding the structure-of-interest from the subset of candidate structures-of-interest if the distance between the structure-of-interest and the grid edge is below a grid edge distance threshold.   
     
     
         13 . The method of  claim 10 , further comprising:
 for each of the subset of candidate structures-of-interest:   determining a milling zone for the structure-of-interest based on the position of the position of the structure-of-interest and a location of a milling beam emitter;   determining whether any of the plurality of structures-of-interest is present in the milling zone;   determining whether any of the plurality of contaminants is present in the milling zone; and   wherein eliminating, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest includes, responsive to determining that one or both of a structure-of-interest or a contaminant is present in the milling zone, excluding the structure-of-interest from the subset of candidate structures-of-interest.   
     
     
         14 . The method of  claim 10 , wherein selecting a selected structure-of-interest for milling comprises:
 for each of the subset of candidate structures-of-interest:   acquiring a fluorescence light microscope image stack;   determining, based on the fluorescence light microscope image stack, whether a region of interest is present in the structure-of-interest; and   responsive to determining that the region of interest is present in the structure-of-interest, selecting the structure-of-interest as the selected structure-of-interest.   
     
     
         15 . The method of  claim 10 , further comprising:
 acquiring the image representing the lamella sample by acquiring the image from an optical microscope.   
     
     
         16 . The method of  claim 10 , wherein applying automated image processing to segment the image includes processing the image with a convolutional neural network trained using a plurality of annotated images of lamella samples. 
     
     
         17 . One or more non-transitory computer readable media having instructions thereon that, when executed by one or more processing devices of a scientific instrument support apparatus, cause the scientific instrument support apparatus to perform the method of  claim 10 . 
     
     
         18 . A charged particle microscope, comprising:
 an electron-beam instrument;   an ion-beam instrument; and   a computing device configured to:
 control the electron-beam instrument to acquire a data set representing a lamella sample, 
 segment the data set into a plurality of segmented classes, 
 identify, from the plurality of segmented classes, a subset of candidate structures-of-interest in the lamella sample, 
 select, from the subset of candidate structures-of-interest in the lamella sample, a selected structure-of-interest for milling by eliminating, based on the plurality of segmented classes, structures-of-interest from the subset of structures-of-interest, 
 control the ion-beam instrument to mill the selected structure-of-interest; and 
 wherein the plurality of segmented classes includes at least one selected from the group consisting of a plurality of structures-of-interest, a plurality of grid squares, a plurality of cracks, a plurality of contaminants, and a background.