BinaryCIF Auto Data Type Detection Pipeline

[hriday1136]

BinaryCIF Auto-Detection Encoding Pipeline

An optional dictionary-free column-type resolution layer integrated directly into py-mmcif's existing BinaryCIF (BCIF) writer. The writer can now use either the original DictionaryApi-driven type lookup or a single-pass data-type detector, removing the PDBx/mmCIF dictionary dependency when auto-detection is enabled while preserving the original dictionary-based behavior.

Documents every feature added on top of upstream rcsb/py-mmcif (baseline v1.1.1), how each works, and the implementation decisions behind them.

---

Overview

BCIF encoding requires every column to be typed integer, float, or string so the right encoder cascade is selected. Upstream py-mmcif gets those types from the loaded PDBx/mmCIF dictionary via DictionaryApi.getTypeCode(). That requires a multi-megabyte dictionary to be loaded, and it silently falls back to string for any attribute not in the dictionary — inflating file size for custom or extended CIF categories whose numeric columns get mis-encoded.

When enabled, this pipeline infers column types directly from the data in a single pass, with a small override table for the few attributes whose correct type can't be inferred safely from values alone. The same BinaryCifWriter class continues to support the original dictionary-based path when auto-detection is disabled.

Summary of changes

New module — bcif_type_detector.py

• ColumnProfile dataclass holding the full per-column classification result.
• classify_column() — single-pass, two-phase (type probe + statistics) classifier.
• Cheapest-first detection cascade: isinstance → char-level isdigit() → compiled regex.
• Consistent sentinel filtering (., ?, None) before type analysis.
• bool-before-int guard; leading-zero protection ("0004" → string).
• Integer-width narrowing from observed min/max; float precision (f32/f64).
• All-sentinel/empty column defaults to integer (total == 0 branch ordered first to avoid a crash).
• Encoding-hint statistics: is_sequential, has_long_runs, unique_ratio, etc.
• No dependency on the writer, dictionary, or any mmcif module — independently testable.

Modified writer — BinaryCifWriter.py

• Auto-detection is integrated into the existing BinaryCifWriter; the separate BinaryCifWriter_autoDetect.py class is no longer needed.
• dictionaryApi is optional when useAutoDetect=True and remains required for dictionary mode.
• Added useAutoDetect to select between auto-detection and the original dictionary-driven type lookup in the same writer.
• DataCategoryTyped pre-casting remains unchanged in dictionary mode and is skipped in auto-detection mode.
• Added sentinel-aware raw-string casting in __encodeColumnData() for auto-detected integer and float columns before struct.pack.
• Added forced-attribute override tables: _FORCE_STRING_ATTRS, _FORCE_INTEGER_ATTRS, _FORCE_FLOAT_ATTRS.
• __getAttributeType() now accepts colDataList and dispatches to either the original dictionary path or the auto-detection path.
• Dictionary fallback in auto-detection mode is scoped to all-sentinel columns when a dictionary is available.
• Upstream encoder cascade preserved unchanged.

Modified adapter — IoAdapterPy.py

• Imports only BinaryCifWriter; the separate auto-detect writer import is removed.
• Keeps useAutoDetect=True on writeFile() and passes the flag into the unified writer.
• useAutoDetect=True (default) instantiates BinaryCifWriter with dictionaryApi=None and enables auto-detection.
• useAutoDetect=False instantiates the same writer with the supplied dictionaryApi, preserving the legacy dictionary-driven path.
• All existing arguments and the readFile() path remain unchanged.

Architecture

                 IoAdapterPy.writeFile(..., useAutoDetect=...)
                                  │
                                  ▼
                         BinaryCifWriter
                                  │
                 useAutoDetect ───┼─── False ─► DictionaryApi type lookup
                                  │                    + DataCategoryTyped
                                  │
                                  └─── True ──► forced attribute override
                                                       │
                                                       ▼
                                              classify_column()
                                                       │
                                                       ▼
                                         optional dictionary fallback
                                           (all-sentinel columns only)

• bcif_type_detector.py — pure classification logic, no mmcif dependency.
• BinaryCifWriter.py — contains both the original dictionary path and the new auto-detection path, applies overrides, casts raw numeric strings in auto mode, and drives the existing encoder cascade.
• IoAdapterPy.py — constructs the unified writer and selects the mode through useAutoDetect.

---

Component deep-dive

1. bcif_type_detector.py — the classifier

What it does. For each column it decides integer / float / string — the choice upstream got from the dictionary — by inferring it from the data in one pass, while also gathering statistics the writer uses for tighter encoding. The result is a ColumnProfile: the primary col_type plus advisory fields (int_width, float_prec, col_min/col_max, max_decimals, is_sequential, has_long_runs, unique_ratio, value_count, sentinel_count). Only col_type is required.

How it does it. One loop, two concurrent phases:

• Sentinel gate first — None, ".", "?" are counted and skipped before any type logic, so missing markers never pollute the inferred type.
• Phase A — type probe — all_int/all_float flags start True and get knocked down. Checks are ordered cheapest-first: isinstance(v, bool) (before int, since bool subclasses int) → isinstance int/float → for strings, a char-level isdigit() fast path, then the compiled _FLOAT regex only if that fails. Once a non-numeric value appears, type_decided flips and Phase A stops working, but the loop does not break.
• Phase B — statistics (always runs) — feeds the unique set (unique_ratio), the adjacent run-length tracker (has_long_runs at run ≥ 4), and the sequential check for integers (is_sequential). Min/max and max decimals accumulate.

Finalization. total == 0 (all sentinels) → int, checked first because the numeric branches dereference col_min (still None). Else all_int → integer with narrowed width; all_float → float with f32/f64; otherwise → string.

The net effect is one O(n) pass that types correctly on custom/extended CIF where the dictionary would silently fall back to string.

2. BinaryCifWriter.py — the unified writer

Auto-detection is integrated into the existing writer rather than implemented in
a second writer class. Dictionary and auto-detection modes therefore share the
same serialization and encoder implementation.

What it does. Writes the .bcif file using either the original dictionary-based column typing or the new data-driven typing path. In dictionary mode, behavior stays aligned with upstream: values can be pre-cast through DataCategoryTyped, and types come from DictionaryApi. In auto-detection mode, the writer resolves types from the column values, casts raw strings to real int/float objects only at the encoder boundary, and leaves the encoder math untouched.

How it does it.

• Constructor — adds useAutoDetect. dictionaryApi may be None in auto mode, but dictionary mode still requires it.
• serialize() — applies DataCategoryTyped only when applyTypes=True and useAutoDetect=False. Auto-detected columns remain raw until encoding. The column data is passed to __getAttributeType() so it can classify values when needed.
• __getAttributeType() — first branches by mode. Dictionary mode retains the original DictionaryApi.getTypeCode() behavior. Auto mode resolves in a strict four-step order: forced override (O(1), skips the scan) → classify_column() → dictionary fallback only when value_count == 0 and a dictionary is available → MolStar integer hints.
• __encodeColumnData() — in auto mode, casts each value to int(v) / float(v) according to the resolved type. Sentinels (None, ., ?) are deliberately not cast, so getMask() continues to build the same valid incompleteness mask.

Everything below this — BinaryCifEncoders, TypedArray, and all encoders — remains unchanged, which keeps output spec-compliant and round-trip compatible.

3. IoAdapterPy.py — the adapter

What it does. A thin routing layer that constructs the unified BinaryCifWriter and selects dictionary or auto-detection mode for each BCIF write. The dictionary-free path remains the default.

How it does it. The separate BinaryCifWriter_autoDetect import and writer selection branch are removed. writeFile() keeps useAutoDetect=True, and the fmt == "bcif" branch now always creates BinaryCifWriter:

• useAutoDetect=True (default) passes useAutoDetect=True and dictionaryApi=None, preventing a stray dictionary argument from changing the
  auto-detection path.
• useAutoDetect=False passes useAutoDetect=False and the supplied dictionaryApi, reproducing the original dictionary-driven behavior through the same writer class.

All existing arguments keep their meaning; readFile() and the mmcif write path are untouched. The caller-facing switch remains the same, but the implementation no longer maintains two copies of the BinaryCIF writer.

---

Implementation decisions and pre-defined factors

Three-type vocabulary. The pipeline is constrained to integer/float/string because BCIF's masked-encoder layer has exactly three entry points (IntArrayMasked, FloatArrayMasked, StringArrayMasked). The detector emits "int"/"float"/"str" and the writer maps them explicitly, keeping the detector independent of the writer's encoder naming.

Integer-width boundaries are the exact representable ranges of BCIF's fixed-width types; the narrowest lossless type is chosen so IntegerPacking works from the smallest source. Unsigned-first at each width gives non-negative columns an extra bit before stepping up a size.

Condition	Width
min ≥ 0, max ≤ 255	uint8
min ≥ -128, max ≤ 127	int8
min ≥ 0, max ≤ 65535	uint16
min ≥ -32768, max ≤ 32767	int16
otherwise	int32

Float cutoff max_dec ≤ 6 → f32, else f64. float32 holds ~7 reliable decimal digits; capping at 6 places stays inside that envelope, so low-precision columns use the half-size float while higher-precision ones widen — conservative by default.

Run-length threshold ≥ 4. Four is where RunLength's (value, count) overhead starts paying off, so has_long_runs signals the writer to attempt it.

Sequential guard total > 1. A single value is trivially sequential, so the flag is suppressed below two values; floats reset the tracker since the consecutive-integer assumption doesn't apply.

Sentinel set {".", "?"} + None — CIF's two missing-value markers plus programmatic absence. Pinned identically in detector and writer; an earlier inconsistency across mask-building, classification, and casting was a real bug source.

Leading zeros force string ("0004", "00.5"): zero-padding is semantically meaningful in CIF IDs and casting would discard it, so correctness wins over the smaller numeric encoding.

bool before int: bool subclasses int, so an unguarded int check would swallow True/False as 1/0; the explicit check forces them to string.

All-sentinel → integer, checked first. A fully-masked integer column collapses under Delta+RunLength and decodes identically regardless of declared type, so it's the cheapest correct answer. The branch runs first because the numeric branches dereference col_min (None here).

Override tables (_FORCE_STRING_ATTRS, _FORCE_INTEGER_ATTRS, _FORCE_FLOAT_ATTRS) encode the minimum dictionary knowledge values can't express: char fields that look numeric (_audit_conform.dict_version = "5.281", atom_site.type_symbol, label_*_id), ID/sequence integers (atom_site.id, auth_seq_id), and coordinate floats (Cartn_x/y/z, occupancy, B_iso_or_equiv). Keys are stored lowercased for case-insensitive matching and looked up first, so known attributes return in O(1) and skip the scan.

Auto-detection resolution order (override → detect → dictionary-on-empty → MolStar) reflects priority: overrides are ground truth; detection is the general case; an optional dictionary is used only when data inference is impossible; MolStar hints preserve exact upstream behavior. When useAutoDetect=False, the writer uses the original dictionary lookup directly instead of this sequence.

Casting at the encoder boundary. In auto-detection mode, pre-casting is skipped, so values can arrive as strings while struct.pack needs real numerics. Casting therefore happens in __encodeColumnData — sentinels excluded so the mask stays valid. Dictionary mode retains the original DataCategoryTyped behavior. The encoder cascade is unchanged, which preserves spec-compliant, round-trip-compatible output.

Config flags. useFloat64=True globally overrides per-column float_prec (safety posture: no precision loss), making f32 unreachable. storeStringsAsBytes=False is the required RCSB default — deviating causes msgpack key mismatches.

---

Type-resolution order

When useAutoDetect=True:

1. Forced override — _FORCE_*_ATTRS. O(1), skips the column scan.
2. Auto-detection — classify_column(colDataList).
3. Dictionary fallback — only when value_count == 0 and a dictionary exists.
4. MolStar integer hints — last, unchanged from upstream.

When useAutoDetect=False, the writer follows the original dictionary-based type
lookup and DataCategoryTyped path.

---

Known behaviors and caveats

• useFloat64=True makes the f32 path unreachable (intentional safety posture).
• Round-trip validation uses normalized string comparison — byte-for-byte CIF text comparison isn't meaningful.
• Float columns are the main size-parity target — coordinate/B-factor columns the dictionary routes through FixedPoint+Delta+IntegerPacking are the largest source of size regression as raw IEEE-float ByteArray.

---

Statistical Significance for each feature

Not Forcing Attributes vs Forcing Attributes

File name	Runtime w/o Forcing types (ms)	Runtime w/t Forcing Types (ms)	Percentage Change
4HHB	199.7610	190.9438	-4.41%
11HB	413.1870	378.8590	-8.31%
3HQV	131.8750	123.5006	-6.35%
3IFX	101.5448	95.7890	-5.67%
3J3Q	56820.6240	51250.7458	-9.80%
9A25	16432.9492	16566.5262	0.81%
9AAO	1162.6970	1141.9092	-1.79%
9A8K	5951.0280	6142.9092	3.22%
AF_AFA0A017SEY2F1	132.2252	123.8124	-6.36%
MA_MABAKCEPC0001	269.0558	249.6076	-7.23%

• The Runtimes are an average of 5 consecutive tests in both scenarios.

Dictionary vs Auto DataType Detection System

File Name	Original BCIF Size (bytes)	Auto Detection BCIF Size (bytes)	Change in Size	Percentage Change	Dict Encoding Runtime (ms)	Auto Encoding Runtime (ms)	Change in Runtime	Percentage Change
4HHB	530,943	495,106	-35,837	-6.75%	245.507	248.467	2.960	1.21%
11HB	1,006,992	967,802	-39,190	-3.89%	330.127	378.052	47.925	14.52%
3HQV	272,430	253,892	-18,538	-6.80%	105.493	119.227	13.734	13.02%
3IFX	342,221	309,663	-32,558	-9.51%	92.050	96.160	4.110	4.46%
3J3Q	113,322,354	113,268,598	-53,756	-0.05%	46120.958	52497.397	6376.439	13.83%
9A25	9,965,128	8,994,539	-970,589	-9.74%	14283.559	15918.518	1634.959	11.45%
9AAO	2,382,897	1,837,472	-545,425	-22.89%	995.218	930.290	-64.928	-6.52%
9A8K	16,699,656	14,004,104	-2,695,552	-16.14%	4599.856	5794.691	1194.835	25.98%
AF_AFA0A017SEY2F1	196,016	186,204	-9,812	-5.01%	76.133	74.771	-1.362	-1.79%
MA_MABAKCEPC0001	585,919	561,323	-24,596	-4.20%	230.704	240.816	10.112	4.38%

• The Runtimes are an average of 6 consecutive test runs in each scenario.
• The encoding runtime refers to the time its takes to read the source mmCIF file, followed by writing out the output BCIF file.

---

Conclusive Summary

This work integrates an automatic column-type detection system directly into the existing BinaryCIF writer. The unified writer can determine each column's type from the actual data without loading the PDBx/mmCIF dictionary, while still retaining the original dictionary-based path. This avoids maintaining separate writer implementations and keeps the feature backward compatible.

The main features added are:

1. Automatic type detection — figures out each column's type (whole number, decimal, or text) by reading the actual data, instead of looking it up in a dictionary file.
2. One unified writer — automatic and dictionary-based typing now live in the same BinaryCifWriter, removing the duplicated auto-detect writer implementation.
3. No dictionary required in auto mode — the writer does not need the large dictionary file when useAutoDetect=True, which remains the default adapter behavior.
4. Old behavior still available — flipping one switch (useAutoDetect=False) uses the original dictionary-based method through the same writer class.
5. Smaller files — stores numbers using the smallest size that fits and the right decimal precision, without losing accuracy.
6. Smart handling of tricky cases — correctly deals with missing values and avoids common mistakes like stripping leading zeros from ID codes or turning version strings into decimals.

Together, these changes make the dictionary optional, improve type accuracy on files the dictionary does not fully cover, remove duplicate writer code, and keep output size competitive without disrupting existing workflows.

[yusufm99]

I left the comments with issues to be addressed

[hriday1136]

Line 91 and 92
Two new output paths added for auto detect: __testBcifAutoOutput and __testBcifAutoTranslated, kept separate from the dictionary-driven output paths.

[hriday1136]

useAutoDetect=False added to the BinaryCifWriter(...) call (bug fix — This accomodates for the new flag in BinaryCifWriter; without it, useAutoDetect defaults to True and the dictionary-driven type path is silently skipped)

[hriday1136]

Line 143-202:
New method __testSerializeAutoDetect to accomodate and test the new auto detection system.
Mirrors testSerialize()'s structure but feeds the writer raw containers (no DataCategoryTyped pre-cast) with useAutoDetect=True, dictionaryApi=None.

[hriday1136]

included the auto detect path. __testCifToBcif(...) call passes the new auto-detect path as a 4th argument.

[hriday1136]

Gained a 4th parameter: ofpAutoDetect

[piehld]

Rather than adding a new argument of ofpAutoDetect, I would suggest adding useAutoDetect=True and reuse the existing (3rd) argument for the output file path. That would let you re-use most of this function without duplicating code blocks. You would then just call it differently for "dictionary-based" vs "autodetect based tests, e.g.:

    def testBcifReaderWriterDictTypes(self):  # replaces existing testBcifReaderWriter()
        self.__testBcifToCif(self.__pathRcsbBcifGzip, self.__pathOutputRcsbBcifTranslated)
        self.__testCifToBcif(self.__pathPdbxDataFile, self.__pathOutputPdbxBcif, self.__pathOutputPdbxBcifTyped, useAutoDetect=False)

    def testBcifReaderWriterAutoDetect(self):  # new test for AutoDetect
        self.__testBcifToCif(self.__pathRcsbBcifGzip, self.__pathOutputRcsbBcifTranslated)
        self.__testCifToBcif(self.__pathPdbxDataFile, self.__pathOutputPdbxBcif, self.__pathOutputPdbxBcifAutoDetect, useAutoDetect=True)

[hriday1136]

Line 333-364
New read-back + comparison block to accommodate the auto detect system.
Reads ofpAutoDetect back, checks block counts, then runs the same type-coercing comparison loop used for the dictionary-typed scenario (duplicated inline rather than factored into a shared helper) against the auto-detected result. Error log message tagged "(bcif, auto-detect)" to distinguish it from a dictionary-path failure.

[Copilot]

Pull request overview

This PR integrates a dictionary-free, single-pass type detection pipeline into the BinaryCIF (BCIF) writer so BCIF column typing can be inferred directly from data (with forced attribute overrides), while retaining the legacy dictionary-driven mode when explicitly selected.

Changes:

• Added bcif_type_detector.py with ColumnProfile and classify_column() for column type inference.
• Unified dictionary-based and auto-detect type resolution inside BinaryCifWriter, and plumbed useAutoDetect through IoAdapterPy.writeFile().
• Expanded tests to exercise BCIF writing/round-tripping in auto-detect mode.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
mmcif/io/bcif_type_detector.py	New single-pass column classifier for int/float/string inference.
mmcif/io/BinaryCifWriter.py	Adds useAutoDetect, forced-type overrides, and auto-mode casting at encode boundary.
mmcif/io/IoAdapterPy.py	Adds useAutoDetect argument and routes BCIF writes through unified writer.
mmcif/tests/testIoAdapterPy.py	Adds an auto-detect BCIF write/read/compare path.
mmcif/tests/testBinaryCifWriter.py	Adds an auto-detect serialize test and adjusts writer instantiation for dictionary mode.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[piehld]

@hriday1136 I actually like the Co-pilot comment here. Can you fix the typos in your single-quoted comment, and convert them to just # prefaced comments? (Also, please delete the commented out/unused code).

[hriday1136]

Worked on all relevant comments. Ready to view.

[piehld]

Thank you @hriday1136! This is outstanding work!

I've left a handful of comments/suggestions below. Please have a look when you get a chance.

[piehld]

Suggested change

	used in BinaryCifWrite.py. The additional profile fields (int_width,
	used in BinaryCifWriter.py. The additional profile fields (int_width,

[piehld]

Suggested change

	Fields used directly by BinaryCifWrite.py
	Fields used directly by BinaryCifWriter.py

[piehld]

I'd suggest swapping the order of these, since by default the first will always return True while the second returns False, so we can escape this conditional a split millisecond faster if it hits the False first (i.e., it won't bother to check the second conditional).

Suggested change

	if self.__applyTypes and not self.__useAutoDetect:
	if not self.__useAutoDetect and self.__applyTypes:

[piehld]

Similar to my comment earlier, I'd suggest swapping the order of this if/else conditional—i.e., start with assumption that AutoDetect is True (if self.__useAutoDetect:)—since that will be the default.

[piehld]

I think you only need to pass the catName here, not the entire dObj:

Suggested change

	def __getAttributeType(self, dObj, atName, colDataList):
	def __getAttributeType(self, catName, atName, colDataList):

Then, replace occurrences of dObj.getName() with catName (both here and in downstream calls).

[piehld]

What is the purpose of this bit? Is it to use the dictionary API data type if all the values are sentinels? If so, I don't think we should bother with this, and instead just default to type "string" (which you're already doing). I can't imagine a situation where the type can't be determined by your script, but somehow is defined in the dictionary.

So basically, if useAudoDetect is True, then we would never use the dictionary even if provided.

[piehld]

Rather than adding a new argument of ofpAutoDetect, I would suggest adding useAutoDetect=True and reuse the existing (3rd) argument for the output file path. That would let you re-use most of this function without duplicating code blocks. You would then just call it differently for "dictionary-based" vs "autodetect based tests, e.g.:

    def testBcifReaderWriterDictTypes(self):  # replaces existing testBcifReaderWriter()
        self.__testBcifToCif(self.__pathRcsbBcifGzip, self.__pathOutputRcsbBcifTranslated)
        self.__testCifToBcif(self.__pathPdbxDataFile, self.__pathOutputPdbxBcif, self.__pathOutputPdbxBcifTyped, useAutoDetect=False)

    def testBcifReaderWriterAutoDetect(self):  # new test for AutoDetect
        self.__testBcifToCif(self.__pathRcsbBcifGzip, self.__pathOutputRcsbBcifTranslated)
        self.__testCifToBcif(self.__pathPdbxDataFile, self.__pathOutputPdbxBcif, self.__pathOutputPdbxBcifAutoDetect, useAutoDetect=True)

[piehld]

Can you reformat this to follow the same structure I proposed for Yusuf's PR (#52 (comment))? Just add a new line with your initials, then put these bullets below that.

[piehld]

Added a few more comments, and back-pedaled on a coupled old ones.

[piehld]

I would actually prefer you retain the casing in the _FORCE_...ATTRS lists below, as well as the leading underscore (e.g., _atom_site.group_PDB), since things can get confusing when we force .lower() them during comparison. Also, doing this will let us merge the predefined items in Yusuf's code with yours more easily.

Apologies that reverting them will require you to re-check each attribute in the dictionary (mmcif.wwpdb.org).

[piehld]

Please do not apply the .lower() transformation here, and include the leading underscore.

Suggested change

	atKey = "%s.%s" % (dObj.getName().lower(), atName.lower())
	atKey = "_%s.%s" % (catName, atName)

[piehld]

(If the answer is no, then we should be able to remove this part)