Jeiss FIBSEM DAT file specs

[mkitti]

The Janelia-Zeiss (Jeiss) FIBSEB DAT files have a 1 kilobyte metadata header.

@clbarnes has compiled a tabular list of the format of that header here:
https://github.com/clbarnes/jeiss-specs

For example, the latest header specification is at the following URL.
https://github.com/clbarnes/jeiss-specs/blob/main/specs/v9.tsv

[virginiascarlett]

Hey Mark,

Apologies for my delayed response here. This is the part where I reveal my technical ignorance. Is the .dat format a TIFF variant? So, are these offset values indicating where the relevant bytes are located in the linear byte stream? We may need to meet and talk about this. It's different from the other formats I've been focusing on, which consist of chunked arrays + JSON metadata files. We should be documenting these TIFF-based formats, too, but I'm not sure of the best way to go about that. We can just describe them in manuals, but that feels very... manual.

[mkitti]

Is the .dat format a TIFF variant?

No. It's its own thing completely. Basically, the first kibibyte of the file stores a bunch of numbers according to the table I linked above. After that, the data follows.

Some example Python code to load it is here:
https://github.com/janelia-cellmap/fibsem-tools/blob/main/src/fibsem_tools/io/dat.py

[mkitti]

Here's another example from @clbarnes which uses that table above to parse the data:
https://github.com/clbarnes/jeiss-convert/blob/main/jeiss_convert/hdf5.py

[mkitti]

The dtype codes are described here:
https://numpy.org/doc/stable/reference/arrays.dtypes.html

[virginiascarlett]

Interesting. I guess how we want to document formats like this one depends on our overall strategy for supporting them. I can certainly start by just creating a folder that points to @clbarnes' documentation. Ideally, long-term, we should have some kind of data model that both microscopists and developers can understand so that they can make choices together based on a common language. I would be curious to know if @axtimwalde has thoughts about how we should be documenting/modeling formats like this one.

This issue is now on my radar. I will talk to @axtimwalde, @bogovicj, @StephanPreibisch, and others, and report back here.

[mkitti]

You could try git submodules or git subtrees:

1. https://git-scm.com/book/en/v2/Git-Tools-Submodules
2. https://www.atlassian.com/git/tutorials/git-subtree

They basically allow you to embed a git repository into another one.

[clbarnes]

Just for everyone's information, I will shortly be moving on from the field and so my level of investment in the vagaries of the Jeiss format will take a nosedive very soon. I'm happy to stay as nominal maintainer and merge PRs to the spec repo in the future (although won't be able to verify them myself), or to donate that repo to an appropriate person/ organisation.

[bogovicj]

@clbarnes this community was lucky to have you - thanks for everything you've done and wishing you the best!

[mkitti]

@clbarnes Thank you for letting us know. Should we create a Github organization around the various "Jeiss" packages so that we can share the burdens of maintainership?

[d-v-b]

the burdens of maintainership?

which are what, exactly? I feel like a github org is a bit overkill for this file format.

[mkitti]

@d-v-b If you want to read it another way, I would like to increase the bus factor for the canonical URL of the repository.

[d-v-b]

instead of creating a new org, why don't we just bring the relevant repos in to janeliascicomp?

[clbarnes]

I wouldn't call it canonical for as long as the people writing the format aren't contributing to it 😅 But yes, by all means adopt them! I can either donate them to the janeliascicomp, or you can fork them and I'll archive mine. I guess either is a bit tricky with the internals of jeiss-convert which uses jeiss-specs as a submodule.

[mkitti]

I will look into the details after lunch (in an hour). I've done repository transfers a few times in the past. I generally prefer transfers over forking when possible since Github will do automatic redirects and it is clear which repository is the primary one.

To initiate a transfer, I can think of two initial steps. One would be adding @clbarnes to a Github organization like JaneliaSciComp. The other would be adding someone in JaneliaSciComp as an owner of the repository.

https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#transferring-a-repository-owned-by-your-personal-account

More in an hour. Thanks for discussing.