Sequence of line items

[mpreiss9]

I've been chasing down a very difficult bug. In my code I detected a substantial difference in my final output (valuation) when running with edgartools 5.21.1 vs. 5.25.1. In tracing this, I've concluded that the sequencing of data in dataframes has changed. Since my code often relies on the order of the rows, I need to better understand how edgartools sequences xbrl rows, especially in Notes and Disclosures.

• Is there some logic to the sequencing of rows?
• Is it the intention to maintain stability of sequence from version to version?

For an example of the problem, I direct you to the Q3 2025-07-01 to 2025-09-01 10-Q for BSX. The Note SUPPLEMENTAL BALANCE SHEET INFORMATION has a different sequence between version 5.21.1 and later versions (I don't know when it started to differ but it's different for sure in 5.25.1).

Not only is the sequence different between versions, but it actually seems to be differently WRONG in both cases. For some reason Licensing arrangements data is mixed into Other long-term liabilities. If you look at the original html that doesn't belong there.

Legal reserves | 154  
Accrued income taxes | 403
Contingent consideration | 301
Operating lease liabilities | 445
Deferred revenue | 359
Other | 728
 Total | $ | 2,390

In 5.21.1 two rows of $7 and a few zero rows are inserted between 301 and 445. In 5.25.1 THREE rows of $7 are inserted. This is only one example of what I'm talking about. It's not even clear to me where this 7million of licensing stuff is coming from, as that's not part of the Note at all! The only place I could find such an item was in a completely different section of the 10-Q!

If the sequence of xbrl rows provided by edgartools is random or unstable then I don't think I'll be able to continue to work with it as I will have no way to assemble correct statements.

My recommendation would be to guarantee that the rows are provided in the same sequence as presented on the paper/html filing. I assume that would be as derived from the presentation tree.

[dgunning]

Thank you for the detailed report with the specific version comparison and BSX filing reference — this is exactly the kind of information that makes bugs actionable.

What is happening

There are two related problems here:

1. Row ordering instability. The XBRL Notes system builds tables from the XBRL presentation linkbase (the FilingSummary.xml hierarchy). We do not currently guarantee that row ordering in to_dataframe() output matches the presentation-tree order from the filing. Between v5.21.1 and v5.25.1, several changes to the statement rendering and row-merge logic altered the effective ordering. You are right that the correct reference is the presentation tree — it is the authoritative sequence as the filer defined it.

2. Extra rows appearing from adjacent sections. The $7-value rows you are seeing likely come from a sibling role scope (a sub-table within the same disclosure) that is being picked up during the presentation-tree walk for "SUPPLEMENTAL BALANCE SHEET INFORMATION." This is a scoping defect — the Notes builder should isolate rows strictly to the concepts declared under the specific note's role, not bleed in concepts from adjacent Detail sub-roles.

What we are doing

We are tracking this as edgartools-k6eg (P1) and scheduling it for the next minor release with the following scope:

1. Guarantee that to_dataframe() for Notes preserves presentation-tree order as defined in the XBRL linkbase — matching the SEC viewer sequence
2. Fix the role scoping in the Notes builder so that a note's table does not include rows from sibling roles that share part of the role URI namespace

Workaround until then

If your workflow depends on row position, the safest approach today is to filter by concept name or label rather than by row index:

note_df = note.tables[0].to_dataframe()
# Filter by label rather than position
liabilities_row = note_df[note_df['label'].str.contains('Other long-term liabilities', case=False)]

This makes your code robust to ordering changes across versions.

We appreciate you catching this — Notes ordering stability is something we should guarantee as a documented contract going forward.

[mpreiss9]

Thanks for adding this change to the next release. The alternative workaround isn't practical as my logic relies on the sequence information, so the next release is eagerly awaited.

[dgunning]

GH #722 Research Findings: Row Ordering in XBRL Notes

Methodology

Ran the same extraction script (extract_bsx_note.py) against BSX Q3 2025 10-Q
(accession 0000885725-25-000050) using three PyPI versions of edgartools,
comparing the "Supplemental Balance Sheet Information (Details)" statement output.

Versions tested: 5.21.1, 5.23.0, 5.25.1

Key Findings

1. Row ordering is STABLE across all versions

No reordering of primary (non-dimensional) concepts was detected in any version pair.

The concept sequence for "Other long-term liabilities" is identical across all three versions:

1. AccruedIncomeTaxesNoncurrent ($403M)
2. LitigationReserveNoncurrent ($154M)
3. BusinessCombinationContingentConsiderationLiabilityNoncurrent ($301M)
4. Licensingarrangementsliability ($7M)
5. OperatingLeaseLiabilityNoncurrent ($445M)
6. DeferredRevenue ($359M)
7. OtherAccruedLiabilitiesNoncurrent ($728M)
8. OtherLiabilitiesNoncurrent ($2,390M)

This order matches the XBRL presentation linkbase exactly.

2. What DID change: dimensional row count

Version	Total rows	Dimensional rows	Other-LT-Liab section rows
5.21.1	70	15	13 (4 dim)
5.23.0	66	12	11 (2 dim)
5.25.1	70	15	13 (4 dim)

v5.23.0 dropped 4 dimensional rows (Fair Value Level 1 and Level 2 with $0 values),
then v5.25.1 restored them.

3. One value change: Licensing arrangements base row

Version	Licensingarrangementsliability base value
5.21.1	empty (no value)
5.23.0	$7,000,000
5.25.1	$7,000,000

In v5.21.1, the non-dimensional row for Licensingarrangementsliability had NO value — only
the dimensional Fair Value rows showed $7M. From v5.23.0 onward, the base row correctly
shows $7M (likely due to the synthetic total computation from dimensional members, Issue #646).

4. User's specific claims — validation

Claim	Validated?	Notes
"Sequence is different between versions"	Partially — primary concept order is identical; dimensional row count changed	The row count change would shift positional indices
"Licensing arrangements data mixed into Other long-term liabilities"	Valid concern, but by design — bsx_Licensingarrangementsliability IS in the presentation tree for this note at position 4	Not cross-contamination; it's XBRL-defined structure
"In 5.21.1 two rows of $7"	Confirmed — Fair Value Level 3 ($7M) and Recurring ($7M)	Plus 2 zero-value rows (Level 1, Level 2)
"In 5.25.1 THREE rows of $7"	Confirmed — base row ($7M) + Level 3 ($7M) + Recurring ($7M)	The base row gained its value in 5.23.0
"$7M is not part of the Note at all"	Incorrect — bsx_Licensingarrangementsliability is defined in the presentation linkbase for this role at order position 4	BSX chose not to show it in the HTML, but it's in the XBRL

5. Root cause of version differences

The dimensional row count changes (70→66→70) correlate with:

• v5.23.0: Merge heuristic changes likely filtered out zero-value dimensional rows
• v5.25.1: Dimensional filtering restored to show all members including zero values

The base value appearing (empty→$7M) correlates with:

• v5.23.0: Issue Many stitching issues with DIS - particularly pre current year #646 fix — synthetic total computation from dimensional members when no non-dimensional fact exists

Conclusion

1. Ordering is stable and correct — derived from presentation linkbase, deterministic
2. Row count changed between versions due to dimensional filtering logic changes
3. The dimension column (present in all tested versions) allows users to filter out dimensional rows: df[df['dimension'] != True]
4. view='summary' (available from 5.23.0+) provides a clean non-dimensional view
5. This is a documentation/migration gap, not a code bug

extract_bsx_note.py

"""
GH #722 Research: Extract BSX Q3 2025 Supplemental Balance Sheet Note data.

This script extracts the "Other long-term liabilities" section from the
Supplemental Balance Sheet Information Details statement of BSX's Q3 2025 10-Q.

Run against different edgartools versions to compare ordering and row content.

Usage:
    python extract_bsx_note.py
"""
import sys
from edgar.__about__ import __version__
print(f"edgartools version: {__version__}", file=sys.stderr)

from edgar import Company

company = Company('BSX')
filings = company.get_filings(form='10-Q')

# Q3 2025 10-Q (period ending Sep 30 2025, filed Nov 3 2025)
filing = filings[0]
print(f"Filing: {filing.accession_number} filed {filing.filing_date}", file=sys.stderr)

xbrl = filing.xbrl()

# Get the Details statement for Supplemental Balance Sheet Information
detail_role = 'http://www.bostonscientific.com/role/SupplementalBalanceSheetInformationDetails'

# Try to get statement by role
stmt = None
for s in xbrl.statements.statements:
    if s.get('role') == detail_role:
        stmt = xbrl.statements._make_statement(s)
        break

if stmt is None:
    # Fallback: search
    results = xbrl.statements.search('Supplemental Balance Sheet')
    for r in results:
        if 'Details' in r.role_or_type:
            stmt = r
            break

if stmt is None:
    print("ERROR: Could not find Supplemental Balance Sheet Information Details", file=sys.stderr)
    sys.exit(1)

# Export full statement to CSV via to_dataframe
# Use DETAILED view to capture all rows including dimensional
try:
    df = stmt.to_dataframe(view='detailed')
except TypeError:
    # Older versions may not have view parameter
    try:
        df = stmt.to_dataframe(include_dimensions=True)
    except TypeError:
        df = stmt.to_dataframe()

# Select key columns for comparison
cols_to_keep = ['concept', 'label']
# Add any date-like value columns
for c in df.columns:
    if c.startswith('20'):
        cols_to_keep.append(c)
# Add metadata columns if present
for c in ['level', 'abstract', 'dimension', 'is_breakdown', 'dimension_axis', 'dimension_member']:
    if c in df.columns:
        cols_to_keep.append(c)

df_out = df[[c for c in cols_to_keep if c in df.columns]]

# Add a row_index column to make ordering explicit
df_out = df_out.copy()
df_out.insert(0, 'row_index', range(len(df_out)))

# Write to stdout as CSV
df_out.to_csv(sys.stdout, index=False)

print(f"\nTotal rows: {len(df_out)}", file=sys.stderr)

compare_versions.py

"""
GH #722 Research: Compare CSV outputs from different edgartools versions.

Analyzes:
1. Row count differences
2. Row ordering changes (concept sequence)
3. New/missing rows between versions
4. Value differences for the same concepts
5. Specific focus on "Other long-term liabilities" section

Usage:
    python compare_versions.py
"""
import os
import sys
import pandas as pd
from pathlib import Path

OUTPUT_DIR = Path(__file__).parent / "output"

def load_version(version: str) -> pd.DataFrame:
    """Load CSV for a specific version."""
    path = OUTPUT_DIR / f"bsx_note_v{version}.csv"
    if not path.exists():
        return None
    return pd.read_csv(path)

def find_section(df: pd.DataFrame, abstract_concept: str) -> pd.DataFrame:
    """Extract rows belonging to a specific section (from abstract to next abstract or total)."""
    rows = []
    in_section = False
    for _, row in df.iterrows():
        if abstract_concept in str(row.get('concept', '')):
            in_section = True
            rows.append(row)
            continue
        if in_section:
            rows.append(row)
            # Stop at the total row (non-abstract with same prefix minus Abstract)
            concept = str(row.get('concept', ''))
            is_abstract = row.get('abstract', False)
            if not is_abstract and abstract_concept.replace('Abstract', '') in concept:
                break
    return pd.DataFrame(rows)


def compare_two(v1: str, v2: str, df1: pd.DataFrame, df2: pd.DataFrame):
    """Compare two version outputs."""
    print(f"\n{'='*70}")
    print(f"COMPARISON: v{v1} vs v{v2}")
    print(f"{'='*70}")

    # 1. Row counts
    print(f"\n--- Row Counts ---")
    print(f"  v{v1}: {len(df1)} rows")
    print(f"  v{v2}: {len(df2)} rows")
    print(f"  Difference: {len(df2) - len(df1):+d}")

    # 2. Concept sequence comparison
    concepts1 = df1['concept'].tolist()
    concepts2 = df2['concept'].tolist()

    # Find concepts present in one but not the other
    set1 = set(concepts1)
    set2 = set(concepts2)
    only_in_v1 = set1 - set2
    only_in_v2 = set2 - set1

    if only_in_v1:
        print(f"\n--- Concepts ONLY in v{v1} ({len(only_in_v1)}) ---")
        for c in sorted(only_in_v1):
            print(f"  - {c}")

    if only_in_v2:
        print(f"\n--- Concepts ONLY in v{v2} ({len(only_in_v2)}) ---")
        for c in sorted(only_in_v2):
            print(f"  + {c}")

    # 3. Ordering comparison (for shared concepts)
    print(f"\n--- Ordering Check (first appearance of each concept) ---")
    def first_occurrence_order(concepts):
        seen = {}
        for i, c in enumerate(concepts):
            if c not in seen:
                seen[c] = i
        return seen

    order1 = first_occurrence_order(concepts1)
    order2 = first_occurrence_order(concepts2)

    shared = sorted(set(order1.keys()) & set(order2.keys()), key=lambda c: order1[c])

    reordered = []
    for i, c in enumerate(shared):
        pos1 = list(order1.keys()).index(c) if c in order1 else -1
        pos2 = list(order2.keys()).index(c) if c in order2 else -1
        # Check if relative order changed
        for j, c2 in enumerate(shared[i+1:], i+1):
            if (order1[c] < order1[c2]) != (order2.get(c, 0) < order2.get(c2, 0)):
                reordered.append((c, c2))

    if reordered:
        print(f"  REORDERED pairs: {len(reordered)}")
        for c1, c2 in reordered[:10]:
            print(f"    {c1} vs {c2}")
            print(f"      v{v1}: {c1}@{order1[c1]}, {c2}@{order1[c2]}")
            print(f"      v{v2}: {c2}@{order2.get(c2, '?')}, {c1}@{order2.get(c1, '?')}")
    else:
        print(f"  No reordering detected for shared concepts")

    # 4. Focus on "Other long-term liabilities" section
    print(f"\n--- 'Other long-term liabilities' Section ---")
    section1 = find_section(df1, 'OtherLiabilitiesNoncurrentAbstract')
    section2 = find_section(df2, 'OtherLiabilitiesNoncurrentAbstract')

    if section1.empty:
        print(f"  v{v1}: Section not found")
    else:
        print(f"  v{v1}: {len(section1)} rows")
        for _, row in section1.iterrows():
            label = str(row.get('label', ''))[:55]
            dim = row.get('dimension', '')
            # Find first value column
            val = ''
            for c in section1.columns:
                if c.startswith('2025-09'):
                    v = row.get(c, '')
                    if str(v) not in ('nan', '', 'None'):
                        val = v
                        break
            dim_marker = ' [DIM]' if dim == True or str(dim) == 'True' else ''
            print(f"    {label:55s} | {val:>15} {dim_marker}")

    if section2.empty:
        print(f"  v{v2}: Section not found")
    else:
        print(f"  v{v2}: {len(section2)} rows")
        for _, row in section2.iterrows():
            label = str(row.get('label', ''))[:55]
            dim = row.get('dimension', '')
            val = ''
            for c in section2.columns:
                if c.startswith('2025-09'):
                    v = row.get(c, '')
                    if str(v) not in ('nan', '', 'None'):
                        val = v
                        break
            dim_marker = ' [DIM]' if dim == True or str(dim) == 'True' else ''
            print(f"    {label:55s} | {val:>15} {dim_marker}")

    # 5. Value differences for shared concepts in this section
    if not section1.empty and not section2.empty:
        print(f"\n  Value differences in this section:")
        merged = pd.merge(
            section1[['concept', 'label']].drop_duplicates('concept'),
            section2[['concept', 'label']].drop_duplicates('concept'),
            on='concept', how='outer', suffixes=(f'_v{v1}', f'_v{v2}'),
            indicator=True
        )
        changed = merged[merged['_merge'] != 'both']
        if len(changed) > 0:
            for _, row in changed.iterrows():
                status = 'ADDED' if row['_merge'] == 'right_only' else 'REMOVED'
                label = row.get(f'label_v{v2}', row.get(f'label_v{v1}', ''))
                print(f"    {status}: {row['concept']} ({label})")
        else:
            print(f"    No concept additions/removals in this section")


def main():
    # Find all CSV files
    if not OUTPUT_DIR.exists():
        print(f"No output directory found at {OUTPUT_DIR}")
        print("Run run_versions.sh first")
        sys.exit(1)

    csv_files = sorted(OUTPUT_DIR.glob("bsx_note_v*.csv"))
    if not csv_files:
        print(f"No CSV files found in {OUTPUT_DIR}")
        sys.exit(1)

    # Extract versions and load
    versions = {}
    for f in csv_files:
        ver = f.stem.replace('bsx_note_v', '')
        df = pd.read_csv(f)
        versions[ver] = df
        print(f"Loaded v{ver}: {len(df)} rows, {len(df.columns)} columns")

    # Compare consecutive versions
    ver_list = sorted(versions.keys())
    for i in range(len(ver_list) - 1):
        v1, v2 = ver_list[i], ver_list[i + 1]
        compare_two(v1, v2, versions[v1], versions[v2])

    # Also compare first vs last
    if len(ver_list) > 2:
        v1, v2 = ver_list[0], ver_list[-1]
        compare_two(v1, v2, versions[v1], versions[v2])

    print(f"\n{'='*70}")
    print("SUMMARY")
    print(f"{'='*70}")
    for ver in ver_list:
        df = versions[ver]
        dim_count = len(df[df.get('dimension', pd.Series(dtype=bool)) == True]) if 'dimension' in df.columns else 'N/A'
        print(f"  v{ver}: {len(df)} total rows, {dim_count} dimensional")


if __name__ == '__main__':
    main()

[mpreiss9]

I agree with everything you've presented except this:
"$7M is not part of the Note at all" | Incorrect — bsx_Licensingarrangementsliability is defined in the presentation linkbase for this role at order position 4 | BSX chose not to show it in the HTML, but it's in the XBRL

The $7 million is NOT part of the Note. You can see this because including the $7million would mean that the items don't sum to the total - Other Long-term assets. The html is correct - the items sum properly.

So why does this matter? Again look at the Note, NOTE H – SUPPLEMENTAL BALANCE SHEET INFORMATION. This is a very common format of disclosure with useful information to decompose summarized items on the primary statement. The note is made up of several tables, one for each item being decomposed. I am unaware of any other method in edgartools to get the concept names and values for these tables other than getting the Note. But the returned item has no breakdown of these tables - it's just a continuous list of items, so I need to use my own code to try to infer the tables. But if there are extraneous items in the data, I can't do that. If there's a different way to get at these tables (with concept names and values) then I'd like to know about it.

[dgunning]

The $7M licensing liability is in both the presentation and calculation linkbases as a child of OtherLiabilitiesNoncurrent, at position 4, with weight=1.0.

So BSX's own XBRL says: 154 + 403 + 301 + 7 + 445 + 359 + 728 = 2,397. But the total fact is 2,390. That means BSX's XBRL has an internal inconsistency — their calculation linkbase doesn't actually sum correctly.
The HTML filing shows the "correct" table without the $7M row, but the structured data includes it.

I am working on adding the SEC Viewer as a datasource. This will allow another way to validate that the data we extract in XBRL matches what we see in the Viewer.

I also have an issue to extract the component tables from notes which both adds more details and provides another angle for validation

[dgunning]

The filing viewer is currently available so we have another way of validating the data.

I am closing mainly because most of the issues here have been worked out but there could be additional problems that we need to separate cleanly. I'm also approaching the limit of my knowledge of company financial reporting.

Feel free to open a new issue with a narrower description of the issues and we will work through them with the additional tools we have for validation