Crystallographic variant mapping using precession electron diffraction data

Conventional PED characterization of the model systems

Figure 1 shows the virtual bright-field (VBF), index, reliability, and orientation maps of the SMA and VO₂ on sapphire, which are the model systems of this study. The VBF images are formed using the intensity of the direct beam in diffraction patterns and are similar to the conventional bright-field TEM micrographs but with less dynamical effect due to the beam precession. The index map is similar to the band contrast map in EBSD, in which brighter pixels indicate better matches between the experimentally acquired diffraction pattern and the simulated pattern. The reliability map is obtained by calculating the ratio of similarities between the best and second-best matches obtained through template matching. Brighter pixels in the reliability map indicate a higher level of confidence and reduced ambiguity between the best and second-best matches in orientation indexing. The concept of “reliability” will serve as a metric to assess the quality of our crystallographic mapping in subsequent sections of this work.

The SMA sample (B19’, monoclinic crystal structure) shows a typical martensitic microstructure where the martensite grains are plate-like [Figure 1A]. The martensite plate thickness varies from tens to hundreds of nanometers. The corresponding index, reliability, and orientation maps [Figure 1B-D] show that multiple martensite variants exist in the sample, but the result is noisy. A VO₂ (monoclinic crystal structure at room temperature) thin film grows epitaxially on c-cut sapphire^[37], which exhibits three crystallographic variants^[38]. The VBF in Figure 1E shows that the VO₂ film is approximately 50-100 nm thick. The corresponding index, reliability, and orientation maps [Figure 1F-H] are extremely noisy, and the variants of VO₂ cannot be identified at all. The poor indexing of the martensitic SMA and VO₂ may be caused by the low symmetry nature of their monoclinic crystal structure, where the spacing and angles of different lattice planes are close to each other, leading to confusion when the software does the diffraction pattern indexing. Consequently, there is a demand for new methods to better illustrate the distribution of crystallographic variants in this wide group of materials.

Method 1: user-selecting-reference-pattern approach

In this method, the user will first peruse the dataset to identify how many crystallographic variants are present in the dataset and then select a reference pattern from each variant. We observed no apparent variation in the diffraction patterns among pixels within each crystallographic variant. This allows high flexibility in selecting the reference pattern for each variant by the user. Next, the diffraction pattern from each pixel will be compared to all reference patterns, and the one with the highest similarity value will be used for the variant assignment of pixels. Each variant is represented by a color in the map. Before going into the detailed mapping results using the first method, we will briefly discuss similarity quantification using Euclidian distance, Cosine, and SSIM algorithms.

Figure 2 shows the VBF of the SMA sample (same as Figure 1A but with some brightness and contrast adjustment) and diffraction patterns from various martensite grains. One of the diffraction patterns was selected as the reference, whereas the others were compared to the reference. By visual inspection, it can be seen that the Ref and B patterns are from the same crystallographic variant, while each of the other patterns represents unique variants. Hence, we expect Ref and B patterns to display the highest similarity. The similarity results, calculated by Euclidian distance, Cosine, and SSIM algorithms, are shown in Table 1. Each comparison method returns a normalized similarity value ranging from 0 to 1. A 0 means the compared images are completely dissimilar, and a 1 means that they are exactly the same. As expected, diffraction pattern B has the highest similarity with the reference pattern for all three similarity methods. A major difference between the different methods is the overall spread in the similarity values between the patterns. Euclidean has a similarity value range of 0.081, Cosine has a small range of 0.026, and SSIM has the largest range of 0.137. We further examine the similarity comparison quality by calculating the reliability values. The reliability is calculated by dividing the highest similarity by the second highest similarity. Larger values indicate higher reliability. The reliability of the Euclidean, Cosine, and SSIM methods are 1.044, 1.006, and 1.083, respectively. The small range and low reliability for the Cosine algorithm may lead to inconclusive results when the two diffraction patterns are similar, which will be demonstrated shortly.

Figure 2. VBF image of the SMA sample (left) and the selected diffraction patterns (right) for similarity quantification.

Figure 3 shows the crystallographic variant maps generated via the Euclidean distance [Figure 3A], Cosine [Figure 3B], and SSIM [Figure 3C] algorithms, along with a manually drawn map [Figure 3D]. The manually drawn map is created by a person inspecting the dataset to determine the variant regions and is treated as the baseline. Both the Euclidean and SSIM maps are exceptionally close to the manually drawn map. The four predominant variants (red, yellow, green, and cyan in color) and two minor variants (blue and magenta in color) are clearly revealed. Note that the Euclidean map is a bit noisy in the upper right region, while the SSIM maps provide a cleaner map. Unfortunately, the Cosine algorithm produced poor results. The map is noisy, and the large cyan variant in the center is split into three separate variants. This may be due to the varying diffraction pattern intensity along the variant. When generating the crystallographic maps using our methods [Figure 3A-C], the colors are selected on a hue saturation value (HSV) color wheel and spaced equally apart based on the number of reference points to distinguish them from each other. The colors were adjusted manually for clarity as needed. Generating the above maps [Figure 3A-C] only took tens of seconds or a few minutes using a computer with an Intel i7 13700K CPU and Nvidia RTX 3080 GPU, but it took a student several hours to generate the manually drawn map [Figure 3D].

Figure 3. Crystallographic variant maps of the SMA sample generated using the (A) Euclidean distance, (B) Cosine, and (C) SSIM algorithms. (D) A mannually drawn crystallographic variant map of the same area serving as a baseline.

To investigate the effect of noise of the diffraction patterns on the similarity quantification and the resultant crystallographic orientation maps, we used the VO₂ thin film deposited on a c-cut monocrystalline sapphire. The diffraction patterns were acquired with 580 × 580 resolution. The 144 × 144 diffraction pattern resolution in the previous SMA example was a result of binning the 580 × 580 original data by the NanoMEGAS commercial software during the data acquisition. Hence, the 580 × 580 resolution diffraction patterns in this case study are much noisier. Figure 4 shows the VBF of the VO₂ thin film on the sapphire substrate with diffraction patterns taken from different pixels. A diffraction pattern from the sapphire substrate was selected as the reference pattern. The diffraction patterns from pixels A, B, C, and D are from sapphire, VO₂ variant 1, VO₂ variant 2, and vacuum, respectively. Since diffraction pattern A was also taken from the sapphire substrate, it is expected to have the highest similarity with the reference pattern. Diffraction pattern D was taken from the vacuum, in which there is only a direct beam without a diffracted beam; it is expected to have the least similarity with the reference pattern. Table 2 summarizes similarity values calculated between the selected diffraction patterns compared to the reference pattern using the Euclidean distance, Cosine, and SSIM algorithms. The Euclidean similarity shows the highest value for A and the lowest for D, as expected. The Cosine algorithm produced a similar distribution as the highest value for A and lowest for D while switching the order for B and C. In contrast, the SSIM method comparison had low overall similarity values and ranked pattern B higher than A. We also note that all the similarity values calculated here are generally lower than the binned case (the SMA example). This observation has two important implications. First, the presence of noise decreases the similarity values, although the two diffraction patterns are visually similar. Second, the presence of noise, though not significant to human eyes, can assign higher similarity values even to the visually dissimilar diffraction patterns.

Figure 4. VBF image of VO₂ thin film on sapphire (left) and the selected diffraction patterns (right) for similarity quantification.

As expected, the generated similarity maps generally display poor quality [Figure 5]. The Euclidean distance map [Figure 5A] is the best out of the three algorithms. The vacuum (black), sapphire substrate (red), and two variants of VO₂ (blue and green) were identified, but small parts VO₂ thin film were wrongly identified as the sapphire substrate. Note that there should be three variants of VO₂ on c-cut sapphire. Only two were captured in the figure due to the small mapping area. In the Cosine map [Figure 5B], large parts of the VO₂ thin film were mistakably identified as the sapphire substrate. The SSIM algorithm [Figure 5C] performed the worst, in which the film and substrate were not distinguished, albeit the two different variants in the VO₂ thin film. These observations suggest that the Euclidean distance method is most noise-tolerant when computing the similarity between two images. The accuracy of the SSIM method can be dramatically affected by noise in the diffraction patterns.

Figure 5. Crystallographic variant maps of the VO₂ thin film on sapphire sample generated using the (A) Euclidean distance, (B) Cosine, and (C) SSIM algorithms.

Removing the noise can dramatically improve the accuracy of diffraction pattern similarity quantification, which subsequently helps generate much improved crystallographic variant maps. One common way to denoise is by binning. For example, the NanoMEGAS commercial software can do binning that reduces the diffraction patterns from 580 × 580 to 144 × 144 resolution (an example see the SMA example in this work). Another common method is to apply a Gaussian filter^[39]. Here, we explore the effect of denoise on the similarity quantification and crystallographic variant mapping accuracy using a Gaussian filter. Figure 6 shows the data similar to Figure 4, but a Gaussian filter (radius = 5) was applied to all diffraction patterns in Figure 6. With the Gaussian filter, the overall noise in each diffraction pattern is reduced. The Gaussian filter improved the similarity values for all methods by increasing the range and more accurately identifying the most similar diffraction patterns [Table 3]. The Euclidean similarity between patterns A and Ref is 0.910 after filtering compared to 0.707 using the raw data. The filtered diffraction patterns from the two VO₂ variants are very different from the Ref pattern (similarity values are 0.600 and 0.723). The difference is the largest when comparing the vacuum diffraction pattern with the Ref pattern, which yields a similarity value of only 0.412. The Cosine method correctly identified that pattern A has the highest similarity value (0.919), but the ones from the two VO₂ variants also showed high similarity values (0.907 and 0.903), which may lead to crystallographic variant misidentification. Among the three algorithms, the Cosine method yields the lowest reliability value. This shortcoming of the Cosine method is consistent with the observation made in the SMA samples [Table 1]. The SSIM algorithm yields a similar trend to the Euclidean distance approach but shows low similarity values in absolute numbers. Taken together, reducing noise with a Gaussian filter significantly improved the similarity quantification across the three algorithms.

Figure 6. VBF image of VO₂ thin film on sapphire (left) and the selected diffraction patterns (right) after Gaussian filtering (radius = 5) for similarity quantification.

Reducing noise also improved the crystallographic variant mapping. Figure 7 shows the crystallographic variant maps generated using the Euclidean distance, Cosine, and SSIM algorithms. All methods produced good results where the sapphire substrate (red), VO₂ variants (blue and green), and vacuum (black) are clearly resolved. A manually drawn baseline map is not provided here because the interface between the two VO₂ variants is curved and difficult to draw. A careful examination of the diffraction patterns in this dataset confirmed that the maps shown in Figure 7 are indeed correct.

Figure 7. Crystallographic variant maps of the VO₂ thin film on sapphire sample generated using the (A) Euclidean distance, (B) Cosine, and (C) SSIM algorithms. The diffraction patterns in the PED data were filtered with a Gaussian filter (radius = 5) before generating the maps.

Method 2: algorithm-selecting-reference-pattern approach

The previous method, though powerful and efficient in generating the crystallographic variant maps, requires user input to select all known crystallographic variants. In some cases, the number and location of the crystallographic variants are unknown. Here, we describe a new method that is built upon the user-selecting-reference-pattern approach (Method 1) but uses the algorithm to select reference patterns to generate crystallographic variant maps.

This method starts at a pixel in the PED data (0, 0) by default or a pixel position chosen by the user. The diffraction pattern corresponding to the pixel is treated as the first reference and is added to a list that will contain reference patterns in the dataset. Each reference pattern in the list represents a unique crystallographic variant. The diffraction patterns from the following pixels will be compared to the reference pattern, and the similarity values (using Euclidean distance, Cosine, or SSIM) will be calculated. The starting pixel will be compared to all other pixels, and the minimum similarity value will be used as the initial cut-off value. If the calculated similarity value of the pixel is greater than the cut-off value, the algorithm treats the pixel the same as the reference pattern variant. If the calculated similarity value is less than the cut-off value, the algorithm will save the diffraction pattern of the pixel as a new reference. The diffraction patterns from the following pixels will be compared to the two references to determine which variant they belong to. This process iterates. If a pixel has one or more similarity values greater than the cut-off, the variant is determined by the one with the highest similarity. If a pixel has all similarity values less than the cut-off, the pixel itself will be added to the reference list and serve as a new variant. This process repeats for all pixels. Once all the pixels have been compared, the cut-off value increases by a step; by default, there are 20 steps between the initial minimum similarity and the maximum similarity values. The process repeats using the increased cut-off value and starting with the list of unique crystallographic variants from the previous iteration.

The above-mentioned mapping method was first used on the SMA dataset to produce a series of similarity maps at varying cut-off values. In the SMA dataset, the (0, 0) point is located at a boundary between two variants that may skew the variant identification. A user-selected point in the middle of a large variant, (116, 168), was used, and the corresponding diffraction pattern serves as the first reference pattern. Figure shows the SSIM similarity maps with various cut-off values ranging from 0.705 to 0.787. The optional initial point and comparison method are the only user inputs using this method. Note the small cut-off range generates very different crystallographic maps. With low cut-off values, such as 0.705, the four predominant variants merged into only two “variants”. With increasing cut-off values, new variants emerge [Figure 8A]. With the cut-off value at 0.769 [Figure 8D], the crystallographic variant map agrees very well with the baseline map in Figure 3D, where both maps contain four predominant and two minor variants [Figure 8D]. Further Increasing the similarity threshold splits variants into two or more pseudo-variants that are differentiated by small differences. For example, with the cut-off value of 0.787, a new dark green variant was generated from the yellow variant, and the magenta variant split from the cyan variant [Figure 8F]. The analysis and crystallographic variant mapping of this dataset took approximately 1 hour to complete.

Figure 8. Crystallographic variant maps generated using the SMA dataset and the SSIM comparison method with (A) 0.705, (B) 0.732, (C) 0.751, (D) 0.769, (E) 0.778, and (F) 0.787 cut-off values.

This algorithm was also applied to the Gaussian-filtered VO₂ using the Euclidean similarity method. For this sample, the point (22, 13) in one of the VO₂ variants was selected as the initial point. Figure 9 shows the generated crystallographic variant maps with the cut-off increasing from 0.627 to 0.816. Again, such a relatively narrow range of cut-offs led to drastically different crystallographic variant maps. With a low cut-off value of 0.627, only the sample and vacuum were identified, but with low accuracy [Figure 9A]. Part of the VO₂ thin film was mistakably identified as vacuum. The sapphire substrate and two VO₂ variants were not resolved. The crystallographic variant identification improves as the cut-off value increases. At the cut-off value of 0.735, the most accurate map was generated [Figure 9C]. Further increasing the cut-off results in the variants splitting into pseudo-variants, as shown in Figure 9D and E. The analysis and crystallographic variant mapping of this dataset took approximately 20 min to complete.

Figure 9. Crystallographic variant maps generated using the Gaussian-filtered VO₂ on sapphire dataset and the Euclidean comparison method with (A) 0.627, (B) 0.707, (C) 0.735, (D) 0.762, (E) 0.789, and (F) 0.816 cut-off values.

The algorithm-selecting-reference-pattern approach can be a useful tool to accurately identify and map similar crystallographic variants in a dataset with minimal user input. Some user input is still needed to select the optional first point and to review the generated similarity maps to determine the ideal similarity cut-off threshold. However, this method is primarily automated and takes much less human effort than the user-selecting-reference-pattern approach (Method 1). Based on the observations made in the above two case studies, it can be seen that small changes in the cut-off value can result in large differences in the final crystallographic variant map output. Lower similarity cut-off values underestimate crystallographic variants, and higher similarity cut-off values overestimate them. User analysis is required to review the generated similarity maps to determine the optimal cut-off value for variant mapping. The computational time varies depending on the size of the dataset, the comparison method used, and the number of variants in the dataset. Typically, it is expected to take between 0.5 and 2 h to generate the similarity maps for a dataset.

Method 3: k-means approach

Unsupervised learning is widely used in image segmentation to group different regions in an image based on the pixel intensities and other features^{[21,22,40,41]}. Martineau et al. and Bergh et al. successfully applied unsupervised learning techniques (e.g., non-negative matrix factorization and fuzzy clustering) to learn significant microstructural features in the PED data to achieve dimensionality reduction and overlapping pattern unmixing^[21,22]. Here, we employed k-means (a common unsupervised learning algorithm) in an approach that is different from the previous two reference pattern-based methods. The advantage of this unsupervised clustering algorithm lies in its ability to partition unlabeled data. After loading the data with HyperSpy, each diffraction pattern is processed into a feature vector using the VGG16 model from Keras^[34]. The dimensionality of each feature vector is then reduced with kernel PCA. In the user interface, the “elbow graph” is generated, allowing the user to determine an optimal k-value (the number of crystallographic variants in the data set) and input it into the k-means algorithm to create a crystallographic orientation map.

The elbow graph and the crystallographic variant maps for the SMA and Gaussian-filtered VO₂ on sapphire datasets with different k-values are shown in Figure 10. The SMA elbow graph [Figure 10A] has a rapidly dropping inertia from k = 1 to 3, after which the slope begins to flatten out. This indicates an optimal k-value of 3 or 4, as both the inertia and k at those points are minimized. From the previous discussion, the mapped region shows four predominant and two minor variants. Hence, the optimal k-value from k-means underestimates the variant numbers. Figure 10B shows crystallographic variant maps for k = 3, 4, and 5. In k = 4, the four predominant martensite variants were identified, but the two minor variants were lost. When k increased to 5, the minor variants were not revealed. Instead, a high level of noise was observed in the martensite plates. The k-means analysis took roughly 7 h on the SMA dataset. Similar observations were made in the Gaussian-filtered VO₂ on sapphire datasets. The elbow graph in Figure 10C indicates an optimal k-value of 2 or 3. From the previous discussion, the dataset should contain four crystallographic variants (sapphire, two VO₂ variants, and vacuum). Again, the optimal k-value tends to underestimate the variant numbers. Figure 10D shows crystallographic variant maps for k = 3, 4, and 5. In the map with k = 3, the sapphire substrate, VO₂ film, and vacuum were identified. However, part of the VO₂ film was erroneously marked as vacuum, as indicated by the arrows. In k = 4, the vacuum and thin film regions remain the same, but the sapphire substrate split into two pseudo-variants. In k = 5, the substrate region remains split, and the vacuum and thin film regions are accurately revealed, with the second VO₂ variant now showing. The k-means analysis took roughly 40 min on the cropped VO₂ on the sapphire dataset (100 × 100 pixels). Based on the above observations, the k-means method can provide general information on the crystallographic variants in the materials with minimal user input but can fail to capture minor variants and may falsely split variants into pseudo-variants.

Figure 10. (A) Inertia vs.k-value elbow graph and (B) crystallographic variant maps with k = 3, k = 4, and k = 5 generated using the k-means clustering method on the SMA dataset. (C) Inertia vs. k-value elbow graph and (D) crystallographic variant maps with k = 3, k = 4, and k = 5 generated using the k-means clustering method on a portion of the Gaussian-filtered VO₂ on the sapphire dataset.

It should be noted that while the k-means algorithm is well-suited for identifying similar-sized spherical clusters, it may not be the ideal choice for the SMA and VO₂ model systems. For example, in the case of the VO₂ on the sapphire dataset, sapphire, two variants of VO₂, and vacuum exhibit significant differences in data sizes, which can potentially lead to misidentifications when using k = 3 or 4. To address this challenge, alternative unsupervised machine learning techniques, such as density-based spatial clustering of applications with noise (DBSCAN), mean shift, and Gaussian mixture models (GMM), could be considered in future studies for more effective identification of crystallographic variants using the PED data^[42]. Nevertheless, the use of k-means serves as a baseline for comparing the performance of other unsupervised learning techniques in future applications.

Method comparison

Each approach described above has ideal use cases along with advantages and disadvantages. The user-selecting-reference-pattern approach (Method 1) is best suited when the number and location of crystallographic variants in the sample are known. A region can be mislabeled if a reference point for the diffraction pattern is not selected. Inversely, if more than one reference point is selected for a pattern, a variant region can be divided between the selected points. Hence, if the user is not familiar with the material, this method can be prone to human errors. When all the variants are selected correctly, this approach can produce the most accurate similarity maps with the lowest computational cost (generally a few minutes). Among the diffraction pattern similarity quantification algorithms, both the Euclidean and SSIM methods outperform the Cosine method in generating more accurate crystallographic variant maps.

The algorithmic-selecting-reference pattern approach (Method 2) omits the need for the user to select the reference pattern of each variant. This approach can generate similarity maps automatically with only user input to select the ideal cut-off threshold based on the generated maps. The cut-off values in each map govern the new variant generation. Too high of a value will capture the small differences of diffraction patterns within the same variant, thus splitting one variant into two or more pseudo-variants. Too low of a value will merge variants with different but similar diffraction patterns. Another challenge is that a small change of the cut-off value can lead to drastically different crystallographic variant maps. Generating the similarity maps with this method usually takes up to an hour on average.

The k-means approach (Method 3) can process the dataset and produce similarity maps automatically at various k-values. The only user input is the k-value (the crystallographic variant number of the mapped area). This number is more intuitive than the cut-off value in the previous method. Moreover, only one round of computation is required to generate the crystallographic orientation maps with various k-values. Changing the k-value and corresponding similarity map for a dataset is near instantaneous, allowing the user to quickly compare maps with varying numbers of diffraction pattern clusters. Unfortunately, the k-means approach tends to generate variant maps with the lowest quality. The generated similarity maps lack fine details, and noise and the number of pseudo-variants increase as the k-value increases. Additionally, the k-means approach is the slowest to compute, as it generally takes several hours to complete the analysis. This approach has room for improvement by experimenting with other unsupervised machine learning models.

These approaches can also be combined to help mitigate the shortcomings of a particular method. For example, in an unknown sample, the algorithm-assisted (Method 2) or the k-means method (Method 3) can be used to determine the approximate variant number and location, which can then be input into the user-assisted algorithm (Method 1), with a manual review, to create an accurate similarity map. It is also worth mentioning that although the methods developed in this work were designed to analyze PED datasets, the same approach can also be applied to the four-dimensional scanning transmission electron microscopy (4D-STEM) datasets, in which the diffraction patterns of each pixel are formed via convergent-beam electron diffraction^[43], to determine the crystallographic variants.

It is also worth noting that creating TEM specimens with generally uniform thickness is imperative for all algorithms to correctly identify the crystallographic variants. Uneven specimen thickness can significantly impact the diffraction patterns of individual pixels. For example, within a single variant, regions with greater thickness will exhibit decreased diffraction spot intensity and increased background noise^[44]. Thick regions also limit the reduction of the dynamical effect, even with the application of precession^[45]. These factors ultimately affect similarity comparisons and can result in poor crystallographic variant mapping.