Error when indexing .ebsp file produced by AZtec 6.0

[christiankwalters]

Hello,

I am encountering an issue when trying to index a .ebsp pattern file that was produced by Oxford AZtec, version 6.0.x. When I try to run IndexEBSD through the command line, I get the following message:

• • • • • • warning: some namelist parameters weren't used: patdset * * * * * *

couldn't determine pixel type for patterns in cc90b00e-a139-462d-8515-02d667ad026e.ebsp

When I try to use the EMSphInxEBSD graphical wizard instead of the command line, the program crashes as soon as I enter the .ebsp filename into the "Pattern File" field. The corresponding terminal output is

Unhandled unknown exception; terminating the application.

These issues occur on both a high-performance compute cluster running CentOS 7 and a laptop running Ubuntu Linux. In both cases, I am using the Debian precompiled binaries for EMSphInx version 0.2. Is the program having difficulty determining the bit depth from the pattern file? I am very interested in getting EMSphInx working with our Oxford EBSD system, so I would welcome any comments or suggestions.

I should note that I have had issues with these AZtec-generated files in EMsoft 5.0.2 as well. In that case, I was able to find a workaround by converting the AZtec .H5OINA file to a NORDIF binary file with kikuchipy. Are there any plans to add NORDIF .dat support to EMSphInx in the future, or any suggestions for getting EMSphInx working with this version of AZtec?

Thanks for your help.

[marcdegraef]

Hi Christian, first thing to check is whether or not the patterns in the .ebsp file are compressed... if they are compressed (Oxford's proprietary compression algorithm) then EMSphInx won't be able to read the file (and neither will EMsoft 5.0.2).  You can always re-save the file with compression turned off. I don't know if Oxford made changes to the .ebsp format in Aztec 6.0; if they did, then that could be another reason for the crash. EMSphInx is no longer being maintained; Will Lenthe, who wrote the code, now works for EDAX/TSL and they are working on a version of SphInx that will be part of their OIM analysis software. Personally, I don't know enough C++ to be able to update the code. It is open source, so you are more than welcome to try this yourself or have someone at your institution help you with it... I have been able to read the .H5OINA format data files in EMsoftOO (the object oriented EMsoft 6.0 beta version); this requires installing the HDF5 plugins which can be tricky sometimes.  For some reason, Oxford decided to compress noisy EBSD patterns... the compression ratio is around 1, as would be expected for noisy data... Regards, Marc.

…

On 2/16/23 11:45 AM, Christian Walters wrote: Hello, I am encountering an issue when trying to index a .ebsp pattern file that was produced by Oxford AZtec, version 6.0.x. When I try to run IndexEBSD through the command line, I get the following message: * o + # * o warning: some namelist parameters weren't used: patdset * * * * * * couldn't determine pixel type for patterns in cc90b00e-a139-462d-8515-02d667ad026e.ebsp When I try to use the EMSphInxEBSD graphical wizard instead of the command line, the program crashes as soon as I enter the .ebsp filename into the "Pattern File" field. The corresponding terminal output is Unhandled unknown exception; terminating the application. These issues occur on both a high-performance compute cluster running CentOS 7 and a laptop running Ubuntu Linux. In both cases, I am using the Debian precompiled binaries for EMSphInx version 0.2. Is the program having difficulty determining the bit depth from the pattern file? I am very interested in getting EMSphInx working with our Oxford EBSD system, so I would welcome any comments or suggestions. I should note that I have had issues with these AZtec-generated files in EMsoft 5.0.2 as well. In that case, I was able to find a workaround by converting the AZtec .H5OINA file to a NORDIF binary file with kikuchipy. Are there any plans to add NORDIF .dat support to EMSphInx in the future, or any suggestions for getting EMSphInx working with this version of AZtec? Thanks for your help. — Reply to this email directly, view it on GitHub <#16>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26VWHD2TFK56WOCUNCNQTWXZKR3ANCNFSM6AAAAAAU6NB7SE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hakonanes]

Hi Christian and Marc,

the issue here might be the same I encountered with an Oxford binary .ebsp file which was of version 4 of that file format, which could not be read by kikuchipy (pyxem/kikuchipy#591). In our case, @drowenhorst-nrl figured out that there is one extra byte (1 uint8) between the file version and the pattern byte offsets in the particular file I struggled with. After skipping that byte, the file could be read as previous files (I have acces to) could. The relevant changes to our file reader are here.

To find out if this is actually the case, could you try importing your file with kikuchipy v0.8.0, Christian? After you install/update kikuchipy, you can try the following

import kikuchipy as kp
assert kp.__version__ == "0.8.0"
s = kp.load("cc90b00e-a139-462d-8515-02d667ad026e.ebsp", lazy=True)
s.plot()

If you can run this without an error occuring it is likely that only a small update to EMsoft's reader is necessary to fix this.

[hakonanes]

In that case, I was able to find a workaround by converting the AZtec .H5OINA file to a NORDIF binary file with kikuchipy. Are there any plans to add NORDIF .dat support to EMSphInx in the future, or any suggestions for getting EMSphInx working with this version of AZtec?

kikuchipy's h5ebsd file format can be read in EMsoft using its "EMEBSD" format. If you indeed can read your file with kikuchipy v0.8, you can write it to the h5ebsd format which seems to me to be readable by EMSphinx, based on the docs.

[christiankwalters]

Marc and Håkon, thanks for your quick replies!

I think Marc's suggestion about the compressed .ebsp file was exactly right. When I followed Håkon's kikuchipy method, I received the following output:

NotImplementedError: Cannot read compressed EBSD patterns from <filename>

indicating that the file had been compressed by AZtec. When I try with an older uncompressed file instead, I get the expected plot from kikuchipy, and EMSphInxEBSD no longer crashes when inputting the filename.

With the uncompressed file, I am now able to make it through the process of setting up the namelist file with the graphical wizard. Unfortunately, I am now getting a segfault when I try to run the indexing program. If I manage to find a solution, I will post it here so others might possibly benefit from it.

Many thanks,

Christian

[hakonanes]

I think Marc's suggestion about the compressed .ebsp file was exactly right.

Indeed!

With the uncompressed file, I am now able to make it through the process of setting up the namelist file with the graphical wizard. Unfortunately, I am now getting a segfault when I try to run the indexing program. If I manage to find a solution, I will post it here so others might possibly benefit from it.

If you allow kikuchipy to print the log (executing kp.set_log_level("DEBUG") before attempting to loading the data), you should see which version of the .ebsp file format your file is in. If it is version 4, I believe the EMsoft reader needs the fix I mentioned above. You could try reading it in kikuchipy and writing to the h5ebsd format, and then read it into EMSphinx using the HDF5 reader. That might work.

[marcdegraef]

Hi Håkon, when I look at a "regular" .ebsp file, the first 8 bytes are "FEFFFFFFFFFFFFFF"; I'm not sure where the version number is located in such a file... In version 4, what does the extra byte look like ?  I'll make the change to both EMsoft 5.0.2 and EMsoftOO... Thanks! Marc.

…

On 2/17/23 4:51 AM, Håkon Wiik Ånes wrote: I think Marc's suggestion about the compressed .ebsp file was exactly right. Indeed! With the uncompressed file, I am now able to make it through the process of setting up the namelist file with the graphical wizard. Unfortunately, I am now getting a segfault when I try to run the indexing program. If I manage to find a solution, I will post it here so others might possibly benefit from it. If you allow kikuchipy to print the log (executing |kp.set_log_level("DEBUG")| before attempting to loading the data), you should see which version of the .ebsp file format your file is in. If it is version 4, I believe the EMsoft reader needs the fix I mentioned above. You could try reading it in kikuchipy and writing to the h5ebsd format, and then read it into EMSphinx using the HDF5 reader. That might work. — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26VWGASO23Q6ZHBX3IOB3WX5C3HANCNFSM6AAAAAAU6NB7SE>. You are receiving this because you commented.Message ID: ***@***.***>

[hakonanes]

I'm not sure where the version number is located in
such a file...

In files of version > 0 it should be the first 8 bytes ("long long", "int64" in Python). In all files of version > 0 I have access to, the version is a negative number in the range 1-4.

In version 4, what does the extra byte look like ?

In the two files of version 4 I have access to (I sent these to you via email), the extra byte is zero. I do not know what it means...

[drowenhorst-nrl]

As a point of clarification - the 8 byte version number, as a signed int is negative, unless the version number is 0! Thus (and I think FORTRAN does not support unsigned ints?)
Forgive my attempt at FORTRAN,

INTEGER (Int64) :: version
INTEGER(Int8)::mysterybyte
* ...  file open statements yada yada 
READ(fileid) version
IF (version GT 0) THEN 
    version = 0
ELSE
    version = -1 * version

IF (version GE 4) THEN 
    READ(fileid) mysterybyte
* continue on reading as though it is a version 2 EBSP file (near as we can tell). 

[marcdegraef]

Both EMsoft 5.0.2 and EMsoftOO (6.0 beta) have been updated to read the .ebsp version 4 file format. it should not be difficult to update EMSphInx as well... this should be done around line 760 in the include/modality/ebsd/pattern.hpp file [I don't know enough C++ to do this myself...] The code should read the first byte B; then 256-B corresponds to the file version.  If this value equals 4, then the header is 9 bytes long, otherwise it is 8 bytes long... Regards, Marc.

…

On 2/17/23 8:02 AM, Håkon Wiik Ånes wrote: I'm not sure where the version number is located in such a file... In files of version > 0 it should be the first 8 bytes ("long long", "int64" in Python). In all files of version > 0 I have access to, the version is a negative number in the range 1-4. In version 4, what does the extra byte look like ? In the two files of version 4 I have access to (I sent these to you via email), the extra byte is zero. I do not know what it means... — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26VWHERI45P33XR3LHKSTWX5ZHZANCNFSM6AAAAAAU6NB7SE>. You are receiving this because you commented.Message ID: ***@***.***>

[christiankwalters]

Håkon, in answer to your questions: When I load the newer .ebsp file, I receive the following debug output from kikuchipy:

DEBUG:kikuchipy.io.plugins.oxford_binary:Reading Oxford binary file of version 4
DEBUG:kikuchipy.io.plugins.oxford_binary:Unknown byte (uint8) in file of version 4: 1

In this case, it appears that the file is version 4, and the extra byte is present. The older, uncompressed .ebsp file seems to be version 2, so the extra byte can't be causing the error in that case. When I try to use the version 2 file with IndexEBSD, I get the following message:

• • • • • • warning: some namelist parameters weren't used: patdset,scanname * * * * * *

malloc(): mismatching next->prev_size (unsorted)
Aborted (core dumped)

A quick Google search suggests this is a C memory error. I also tried your suggestion of importing the older .ebsp file with kikuchipy, saving it as an h5ebsd file, and using that as the input to IndexEBSD. I received the following error message after doing so:

• • • • • • warning: some namelist parameters weren't used: scanname * * * * * *

/home/christianwalters/EMsoft/EMSphInx/patterns.h5 doesn't have a Manufacturer string

Here, it seems EMSphInx is expecting a dataset named 'Manufacturer' which contains the name of the instrument vendor, for example "EDAX". I tried getting around this by adding a new dataset to the file with h5py:

mfg = f.create_dataset('/Manufacturer',(1,),data="Oxford", dtype="|S5")

and after doing this, I receive a different error message from IndexEBSD.

HDF5-DIAG: Error detected in HDF5 (1.8.20) thread 0:
#000: /home/will/Documents/emsphinxpublic/build/hdf5/src/hdf5/src/H5D.c line 363 in H5Dopen2(): not a dataset
major: Dataset
minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.20) thread 0:
#000: /home/will/Documents/emsphinxpublic/build/hdf5/src/hdf5/src/H5D.c line 363 in H5Dopen2(): not a dataset
major: Dataset
minor: Inappropriate type
H5 error attempting to read EBSD patterns:
file - /home/christianwalters/EMsoft/EMSphInx/patterns.h5
dset - /Scan 1/EBSD/Data/patterns
func - H5File::openDataSet
detailed message:
H5Dopen2 failed

Still, this is much closer than I have been able to get in the past. Thanks again for your help.

Regards, Christian

[hakonanes]

Thank you for doing this test, Christian.

Here, it seems EMSphInx is expecting a dataset named 'Manufacturer' which contains the name of the instrument vendor, for example "EDAX". I tried getting around this by adding a new dataset to the file with h5py:

The kikuchipy h5ebsd format (still evolving) has a manufacturer dataset named "manufacturer" in the top group, not "Manufacturer". It seems like EMSphinx is case sensitive when trying to read HDF datasets. The other error message you get is not possible to explain based on the message itself.

A final thought: EMSphinx also reads the EMsoft binary format (.data file). @marcdegraef, can you confirm that the format is just one long byte string starting with the first pixel in the first pattern and ending with the last pixel in the last pattern? If so, it's identical to the NORDIF binary .dat file format and converting any file with kikuchipy to a NORDIF .dat file and changing the file ending to .data should make it readable by EMSphinx. I haven't tested this myself, but will see if I can get to it.

[marcdegraef]

I can confirm that the EMsoft Binary format simply has all the patterns in order without any header information and with one single byte per pattern pixel.  I remember that in the early days of EMsoft, there was some confusion about the patterns needing to be upside down in this file format; according to my most recent notes, this is *not* necessary anymore...(but, if you get weird results, that would be the first thing to check...) Regards, Marc.

…

On 2/19/23 3:49 PM, Håkon Wiik Ånes wrote: Thank you for doing this test, Christian. Here, it seems EMSphInx is expecting a dataset named 'Manufacturer' which contains the name of the instrument vendor, for example "EDAX". I tried getting around this by adding a new dataset to the file with h5py: The kikuchipy h5ebsd format (still evolving) has a manufacturer dataset named "manufacturer" in the top group, not "Manufacturer". It seems like EMSphinx is case sensitive when trying to read HDF datasets. The other error message you get is not possible to explain. A final thought: EMSphinx also reads the EMsoft binary format (.data file). @marcdegraef <https://github.com/marcdegraef>, can you confirm that the format is just one long byte string starting with the first pixel in the first pattern and ending with the last pixel in the last pattern? If so, it's identical to the NORDIF binary .dat file format and converting any file with kikuchipy to a NORDIF .dat file and changing the file ending to .data should make it readable by EMSphinx. I haven't tested this myself, but will see if I can get to it. — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26VWFSGTLHRPFQWXZS2JTWYKBOLANCNFSM6AAAAAAU6NB7SE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Marc De Graef Professor Department of Materials Science and Engineering 130 Roberts Engineering Hall Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213-3890 USA Ph: (412) 268-8527 Fx: (412) 268-7596 ***@***.*** web:http://materials.cmu.edu/degraef Admin. Asst.: Marygrace Antkowski Phone: (412) 268-7240

[marcdegraef]

Hi Christian, I have modified the EMSphInx .ebsp file handling routine to include version 4 files... I checked it out with the EBSPDims program and that seems to work.  Maybe you could pull the new code, rebuild EMSphInx and check whether or not the wizard works as well ? [I'm having an issue with the wizard not building on Apple ARM processors, but that's separate from the .ebsp version 4 issue] Thanks, Marc.

…

On 2/19/23 3:56 PM, Marc De Graef wrote: I can confirm that the EMsoft Binary format simply has all the patterns in order without any header information and with one single byte per pattern pixel.  I remember that in the early days of EMsoft, there was some confusion about the patterns needing to be upside down in this file format; according to my most recent notes, this is *not* necessary anymore...(but, if you get weird results, that would be the first thing to check...) Regards, Marc. On 2/19/23 3:49 PM, Håkon Wiik Ånes wrote: > > Thank you for doing this test, Christian. > > Here, it seems EMSphInx is expecting a dataset named > 'Manufacturer' which contains the name of the instrument vendor, > for example "EDAX". I tried getting around this by adding a new > dataset to the file with h5py: > > The kikuchipy h5ebsd format (still evolving) has a manufacturer > dataset named "manufacturer" in the top group, not "Manufacturer". It > seems like EMSphinx is case sensitive when trying to read HDF > datasets. The other error message you get is not possible to explain. > > A final thought: EMSphinx also reads the EMsoft binary format (.data > file). @marcdegraef <https://github.com/marcdegraef>, can you confirm > that the format is just one long byte string starting with the first > pixel in the first pattern and ending with the last pixel in the last > pattern? If so, it's identical to the NORDIF binary .dat file format > and converting any file with kikuchipy to a NORDIF .dat file and > changing the file ending to .data should make it readable by EMSphinx. > I haven't tested this myself, but will see if I can get to it. > > — > Reply to this email directly, view it on GitHub > <#16 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AB26VWFSGTLHRPFQWXZS2JTWYKBOLANCNFSM6AAAAAAU6NB7SE>. > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- Marc De Graef Professor Department of Materials Science and Engineering 130 Roberts Engineering Hall Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213-3890 USA Ph: (412) 268-8527 Fx: (412) 268-7596 ***@***.*** web:http://materials.cmu.edu/degraef Admin. Asst.: Marygrace Antkowski Phone: (412) 268-7240 — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26VWGRELVVSHQBUTREUJ3WYKCGDANCNFSM6AAAAAAU6NB7SE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Marc De Graef Professor Department of Materials Science and Engineering 130 Roberts Engineering Hall Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA 15213-3890 USA Ph: (412) 268-8527 Fx: (412) 268-7596 ***@***.*** web:http://materials.cmu.edu/degraef Admin. Asst.: Marygrace Antkowski Phone: (412) 268-7240

[christiankwalters]

Marc,

Thanks very much for taking the time to do this. I am able to compile the updated code on Ubuntu, and the EMSphInxEBSD program is now able to handle my older (version 2) uncompressed .ebsp files with no issues. These weren't working previously, so this will be a big help to my research group. One of the EBSD systems at my institution is still on an older version of AZtec.

So far, I have not been successful at converting my compressed version 4 .ebsp file to an uncompressed file. It seems AZtec simply copies the compressed .ebsp files when using "Save As..." to make a copy of the project, even after turning off the compression option. I'll collect some fresh data when I have a chance, and I will post an update here once I am able to try the latest code with an uncompressed version 4 .ebsp. Hopefully, this will only take a day or two.

Many thanks,
Christian

[christiankwalters]

Hello everyone,

My apologies for the very long delay in responding to this thread.

Since my last post, I've gone back and collected EBSD data from an electropolished cubic material (point group m-3m), saving it as an uncompressed version 4 .ebsp file with patterns of 622x512 pixels. I'm happy to report that the updated code accepts the version 4 .ebsp file and runs to completion with no crashes. Marc, thanks again for your help!

Unfortunately, it seems that there is a new problem. If I input the .ebsp file directly into EMSphInx, the following IPF-Z map is the result:

For reference, here is the correct map. My first thought was that this is some sort of sample-detector geometry issue.

Something I discovered while experimenting with the tutorial Ni dataset is that EMSphInx expects the patterns in a binary .data file to be flipped relative to how they are stored in the EDAX .h5 file format. If I flip the binary patterns before indexing them (for example, with numpy.flip() in Python), EMSphInx recovers exactly the same result as the .h5 file.

I tried to do something similar with my dataset, i.e., converting the patterns to 32-bit floats, inverting them with numpy.flip(), and storing them in row-major form as a .data file. The results are dramatically improved, but there seems to be a persistent mis-indexing issue: possibly still a sample-detector geometry problem?

This persistent issue is what I am currently stuck on. I initially suspected that it could be a pattern center problem. Starting with the pattern center (x*, y*, z*) = (0.520, 0.551, 0.659) outputted by AZtec, I arrived at coordinates of (xpc, ypc, L) = (12.44, 86.722, 15552) for EMsoft version 4. My binned CCD pixel size is 37.94 μm. To further check my work, I used the EBSDDetector.pc_bruker() conversion tool in kikuchipy to convert these coordinates to Bruker convention, arriving at (0.52, 0.3306, 0.8006). The result was the same regardless of the convention I used in EMSphInx.

I know this code is not really maintained anymore, and I appreciate the help you've offered me so far. Are there any strategies I could try next to troubleshoot this issue?

Many thanks,
Christian