feat(stachv3): Implemented a common methods to convert stachv3 json & arrow to table. by sainathsagar · Pull Request #54 · factset/stach-extensions

sainath (sainathsagar) · 2026-03-23T09:42:56Z

Description:
Implemented methods like ConvertJsonToTable, ConvertArrowFileToTable and ConvertArrowStreamToTable for the stachv3 responses in all the 3 languages.

…ding library

Rico (RicoFactset) · 2026-03-23T09:45:24Z

Checkmarx One – Scan Summary & Details – f42d17af-56d9-44aa-8aec-ab2a33eb37eb

New Issues (2)

Checkmarx found the following issues in this Pull Request

#	Severity	Issue	Source File / Package	Checkmarx Insight
1		CVE-2026-0994	Python-protobuf-6.32.0	details Recommended version: 6.33.5 Description: A Denial-of-Service (DoS) vulnerability exists in "google.protobuf.json_format.ParseDict()" in Python, where the "max_recursion_depth" limit can be... Attack Vector: NETWORK Attack Complexity: LOW `ID: 1TfYUfzFIewv35RdAcrzsK0t7yaO%2FphFoPQ%2FWpFdIxg%3D` Vulnerable Package
2		Cxfa47c4e4-5ef9	Maven-com.fasterxml.jackson.core:jackson-core-2.13.5	details Recommended version: 2.18.6 Description: The non-blocking (async) JSON parser in jackson-core bypasses the "maxNumberLength" constraint (default: 1000 characters) defined in "StreamReadCon... Attack Vector: NETWORK Attack Complexity: LOW `ID: 6nbnOeXz5elodBdu3K%2FdiAOoV4OUf%2BrT5fxrYTsHolk%3D` Vulnerable Package

Fixed Issues (3)

Great job! The following issues were fixed in this Pull Request

Severity	Issue	Source File / Package
	~~CVE-2019-0820~~	Nuget-System.Text.RegularExpressions-4.3.0
	~~CVE-2024-21907~~	Nuget-Newtonsoft.Json-12.0.2
	~~CVE-2021-22570~~	Nuget-Google.Protobuf-3.12.3

Communicate with Checkmarx by submitting a PR comment with Checkmarx (@Checkmarx) followed by one of the supported commands. Learn about the supported commands here.

Copilot

Pull request overview

Adds STACH v3 “convert to table” helpers across Python/Java/.NET so consumers can convert v3 JSON and Arrow IPC (file/stream) responses into tabular structures.

Changes:

Python: adds v3 JSON/Arrow helpers plus a StachV3ExtensionFactory and corresponding test resources.
Java: adds v3 JSON/Arrow helpers, a StachV3ExtensionsFactory, test suite/resources, and Arrow dependencies.
.NET: adds v3 JSON/Arrow helpers, a StachV3ExtensionFactory, tests/resources, and new package references (including Apache.Arrow).

Reviewed changes

Copilot reviewed 30 out of 37 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
python/tests/test_v3_extension_factory.py	Adds Python unit tests for v3 conversion entrypoints.
python/tests/resources/V3JsonStachResponse.json	Adds a STACH v3 JSON fixture for tests.
python/tests/resources/V3ArrowStreamStachResponse.arrow	Adds Arrow stream fixture for tests.
python/tests/resources/V3ArrowFileStachResponse.arrow	Adds Arrow file fixture for tests.
python/src/fds/protobuf/stach/extensions/v3/init.py	Introduces v3 Python package namespace.
python/src/fds/protobuf/stach/extensions/v3/TableSchema.py	Adds a small schema container used by v3 helpers.
python/src/fds/protobuf/stach/extensions/v3/StachV3SchemaProperties.py	Centralizes v3 JSON property names.
python/src/fds/protobuf/stach/extensions/v3/StachV3JsonHelper.py	Implements JSON-to-DataFrame conversion logic for v3.
python/src/fds/protobuf/stach/extensions/v3/StachV3Constants.py	Defines v3 conversion constants (ignored columns, separators, etc.).
python/src/fds/protobuf/stach/extensions/v3/StachV3ArrowHelper.py	Implements Arrow IPC (file/stream) to DataFrame conversion logic for v3.
python/src/fds/protobuf/stach/extensions/StachV3ExtensionFactory.py	Adds Python entrypoints: ConvertJson/ArrowFile/ArrowStream to table.
java/src/test/java/com/factset/protobuf/stach/extensions/tests/V3StachExtensionsTests.java	Adds Java tests for v3 conversion entrypoints.
java/src/test/java/Resources/V3JsonStachResponse.json	Adds Java JSON fixture for tests.
java/src/test/java/Resources/V3ArrowStreamStachResponse.arrow	Adds Java Arrow stream fixture for tests.
java/src/test/java/Resources/V3ArrowFileStachResponse.arrow	Adds Java Arrow file fixture for tests.
java/src/main/java/com/factset/protobuf/stach/extensions/v3/TableSchema.java	Adds Java schema container for v3 conversion helpers.
java/src/main/java/com/factset/protobuf/stach/extensions/v3/StachV3SchemaProperties.java	Centralizes v3 JSON property names (Java).
java/src/main/java/com/factset/protobuf/stach/extensions/v3/StachV3JsonHelper.java	Implements JSON-to-table conversion logic (Java).
java/src/main/java/com/factset/protobuf/stach/extensions/v3/StachV3Constants.java	Defines v3 conversion constants (Java).
java/src/main/java/com/factset/protobuf/stach/extensions/v3/StachV3ArrowHelper.java	Implements Arrow IPC parsing and conversion logic (Java).
java/src/main/java/com/factset/protobuf/stach/extensions/v3/ConversionContext.java	Adds a per-call state container for Arrow conversion (Java).
java/src/main/java/com/factset/protobuf/stach/extensions/v3/ArrowSchemaMetadataNames.java	Adds Arrow schema metadata keys (Java).
java/src/main/java/com/factset/protobuf/stach/extensions/StachV3ExtensionsFactory.java	Adds Java entrypoints: ConvertJson/ArrowFile/ArrowStream to tables.
java/pom.xml	Adds Arrow + STACH v3 deps and surefire argLine configuration.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/V3/TableSchema.cs	Adds .NET schema container for v3 conversion helpers.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/V3/StachV3SchemaProperties.cs	Centralizes v3 JSON property names (.NET).
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/V3/StachV3JsonHelper.cs	Implements JSON-to-DataTable conversion logic (.NET).
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/V3/StachV3Constants.cs	Defines v3 conversion constants (.NET).
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/V3/StachV3ArrowHelper.cs	Implements Arrow IPC parsing and conversion logic (.NET).
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/V3/ArrowSchemaMetadataNames.cs	Adds Arrow schema metadata keys (.NET).
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/StachV3ExtensionFactory.cs	Adds .NET entrypoints: ConvertJson/ArrowFile/ArrowStream to DataSet.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions/FactSet.Protobuf.Stach.Extensions.csproj	Updates target framework and adds v3/Arrow dependencies.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions.Tests/StachV3ExtensionFactoryTests.cs	Adds MSTest coverage for v3 conversion entrypoints.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions.Tests/Resources/V3JsonStachResponse.json	Adds .NET JSON fixture for tests.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions.Tests/Resources/V3ArrowStreamStachResponse.arrow	Adds .NET Arrow stream fixture for tests.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions.Tests/Resources/V3ArrowFileStachResponse.arrow	Adds .NET Arrow file fixture for tests.
dotnet/StachExtensions/FactSet.Protobuf.Stach.Extensions.Tests/FactSet.Protobuf.Stach.Extensions.Tests.csproj	Copies v3 fixtures to output and adds Newtonsoft.Json ref.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-23T09:50:37Z

+# Module-level configuration variables (set before calling conversion functions)
+column_headers_mapping = {}
+
+def ConvertArrowFileToTable(file_path):
+    global column_headers_mapping
+    column_headers_mapping = {}  # Reset for each conversion
+    data_set = {}


The global column_headers_mapping introduces shared mutable state; concurrent calls to Convert* can race and corrupt each other’s mappings. Use a local variable passed through helpers (or a context object) instead of a module-level global.

Copilot · 2026-03-23T09:50:38Z

+def get_levels(arrow_bytes: bytes):
+    levels = []
+    reader = get_arrow_stream(arrow_bytes)
+    if reader is None:
+        raise Exception("Failed to create ArrowStreamReader from arrow_bytes.")
+
+     # Read first record batch
+    try:
+       record_batch = reader.read_next_batch()
+    except AttributeError:


Indentation in get_levels is inconsistent (e.g., the try: block body is not aligned), which can lead to IndentationError/TabError depending on the actual whitespace in the file. Reformat this function to use consistent 4-space indentation throughout.

Copilot · 2026-03-23T09:50:38Z

+    data_table = pd.DataFrame(columns=columns)
+
+    if hasattr(reader, "num_record_batches"):
+        for idx in range(reader.num_record_batches):
+            batch = reader.get_batch(idx)
+            for row_index in range(batch.num_rows):
+                data_row = get_data_row(batch, row_index, data_table, column_headers_mapping)
+                data_table = pd.concat([data_table, pd.DataFrame([data_row])], ignore_index=True)
+    else:
+        for record_batch in reader:
+            for row_index in range(record_batch.num_rows):
+                data_row = get_data_row(record_batch, row_index, data_table, column_headers_mapping)
+                data_table = pd.concat([data_table, pd.DataFrame([data_row])], ignore_index=True)
+


pd.concat is used per row while building the DataFrame, which is O(n^2) and slow for non-trivial row counts. Consider collecting data_row dicts into a list and creating the DataFrame once (or using DataFrame.loc[len(df)] sparingly) to improve performance.

Suggested change

data_table = pd.DataFrame(columns=columns)

if hasattr(reader, "num_record_batches"):

for idx in range(reader.num_record_batches):

batch = reader.get_batch(idx)

for row_index in range(batch.num_rows):

data_row = get_data_row(batch, row_index, data_table, column_headers_mapping)

data_table = pd.concat([data_table, pd.DataFrame([data_row])], ignore_index=True)

else:

for record_batch in reader:

for row_index in range(record_batch.num_rows):

data_row = get_data_row(record_batch, row_index, data_table, column_headers_mapping)

data_table = pd.concat([data_table, pd.DataFrame([data_row])], ignore_index=True)

# Initialize an empty DataFrame structure to pass into get_data_row if needed.

data_table = pd.DataFrame(columns=columns)

rows = []

if hasattr(reader, "num_record_batches"):

for idx in range(reader.num_record_batches):

batch = reader.get_batch(idx)

for row_index in range(batch.num_rows):

data_row = get_data_row(batch, row_index, data_table, column_headers_mapping)

rows.append(data_row)

else:

for record_batch in reader:

for row_index in range(record_batch.num_rows):

data_row = get_data_row(record_batch, row_index, data_table, column_headers_mapping)

rows.append(data_row)

if rows:

data_table = pd.DataFrame(rows, columns=columns)

else:

data_table = pd.DataFrame(columns=columns)

Copilot · 2026-03-23T09:50:38Z

+        if os.path.exists(json_file):
+            try:
+                result = ConvertJsonToTable(json_file)
+                self.assertIsNotNone(result)
+                self.assertIn('table', result)
+
+            except Exception as e:
+                # Skip if implementation has unmet dependencies
+                self.skipTest(f"ConvertJsonToTable not fully implemented: {str(e)}")


These tests currently skip on any exception from the conversion functions (and also skip when resource files are missing). Skipping on broad exceptions can mask real regressions and makes CI pass even when conversions are broken. Prefer asserting expected output (and failing) for known-good resources, and only skip for truly optional dependencies by checking availability up-front (e.g., import checks for pyarrow).

Copilot · 2026-03-23T09:50:38Z

+                    if (levelIndex == groupIndex) {
+                        dataRow.put(columnName, securityNameColumnData != null ? getColumnValue(securityNameColumnData, rowIndex) : "");
+                    } else if (rowIndex > 0 && levelIndex > 0 && groupIndex < levelIndex) {
+                        dataRow.put(columnName, securityNameColumnData != null ? getColumnValue(securityNameColumnData, rowIndex - 1) : "");


For hierarchical/group columns, this copies the previous row’s security value (securityNameColumnData, rowIndex-1) rather than the previous row’s already-computed group column value. This will produce incorrect group values once you have deeper nesting or siblings. Use the previously built output row (e.g., arrowContext.dataTable.get(rowIndex - 1).get(columnName)) to propagate group column values, matching the JSON implementation.

Suggested change

dataRow.put(columnName, securityNameColumnData != null ? getColumnValue(securityNameColumnData, rowIndex - 1) : "");

Map<String, String> previousRow = arrowContext.dataTable.get(rowIndex - 1);

if (previousRow != null) {

String previousGroupValue = previousRow.get(columnName);

if (previousGroupValue != null) {

dataRow.put(columnName, previousGroupValue);

}

}

Copilot · 2026-03-23T09:50:40Z

+                String columnObject = columnObj.get(StachV3SchemaProperties.Name).toString();
+                if (columnObject == null) {
+                    continue;
+                }
+


columnObj.get(StachV3SchemaProperties.Name).toString() will throw a NullPointerException if the name field is missing or null. Check containsKey and null before calling toString() (and drop the redundant columnObject == null check after toString()).

Suggested change

String columnObject = columnObj.get(StachV3SchemaProperties.Name).toString();

if (columnObject == null) {

continue;

}

if (!columnObj.containsKey(StachV3SchemaProperties.Name)) {

continue;

}

Object nameValue = columnObj.get(StachV3SchemaProperties.Name);

if (nameValue == null) {

continue;

}

String columnObject = nameValue.toString();

Copilot · 2026-03-23T09:50:40Z

+      <None Update="Resources\StachV3ArrowFileResponse.arrow">
+        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
+      </None>
+      <None Update="Resources\StachV3ArrowStreamResponse.arrow">
+        <CopyToOutputDirectory>Always</CopyToOutputDirectory>
+      </None>


These None Update entries reference Resources\StachV3ArrowFileResponse.arrow and Resources\StachV3ArrowStreamResponse.arrow, but those files don’t exist in the repo. Since Update won’t add new items, these lines are ineffective and likely a copy/paste mistake—remove them or correct the filenames to the actual V3Arrow*.arrow resources.

Suggested change

<None Update="Resources\StachV3ArrowFileResponse.arrow">

<CopyToOutputDirectory>Always</CopyToOutputDirectory>

</None>

<None Update="Resources\StachV3ArrowStreamResponse.arrow">

<CopyToOutputDirectory>Always</CopyToOutputDirectory>

</None>

Copilot · 2026-03-23T09:50:41Z

+        }
+
+        [TestMethod]
+        public void ConvertArrowStremToTable_WithCompleteStachV3Structure_SuccessfullyConverts()


Typo in test name: ConvertArrowStremToTable... should be ConvertArrowStreamToTable... for consistency and easier discovery/search.

Suggested change

public void ConvertArrowStremToTable_WithCompleteStachV3Structure_SuccessfullyConverts()

public void ConvertArrowStreamToTable_WithCompleteStachV3Structure_SuccessfullyConverts()

Copilot · 2026-03-23T09:50:41Z

+import pyarrow as pa
+import pyarrow.ipc as ipc
+import base64
+import pandas as pd
+import json
+from google.protobuf import json_format
+from IPython.core.display import HTML
+import io


This module imports pyarrow and IPython at import time, but the Python package's install_requires currently doesn't include these dependencies. As a result, simply importing StachV3ExtensionFactory will raise ImportError in environments that only install the published requirements. Consider making these optional (move imports inside conversion functions with a clear error if unavailable) and/or add the required dependencies (pyarrow and the STACH v3 protobuf package) to the package requirements; also remove unused imports like HTML/json_format if not needed.

Suggested change

import pyarrow as pa

import pyarrow.ipc as ipc

import base64

import pandas as pd

import json

from google.protobuf import json_format

from IPython.core.display import HTML

import io

Copilot · 2026-03-23T09:50:41Z

+        } catch (Exception ex) {
+            System.out.println("Exception occurred: " + ex.getMessage());
+        }
+
+        List<List<Map<String, String>>> dataSet = new ArrayList<>();
+        dataSet.add(arrowHelper.arrowContext.dataTable);
+
+        return dataSet;
+    }


Exceptions are caught and only logged, but the method still returns a non-null dataset (potentially empty/partial), which can hide failures and make debugging difficult. Consider rethrowing (or returning null) and using a logger instead of System.out so callers/tests can reliably detect conversion errors.

sainath (sainathsagar) added 5 commits March 13, 2026 15:18

dotnet language changes

e342b42

SimplifiedRowOrganizedStachExtension class compile issues fixed by ad…

0d4f410

…ding library

java language changes

2dc5991

dotnet languages changes

95b968a

python language changes

53f59a4

sainath (sainathsagar) requested a review from Copilot March 23, 2026 09:43

Copilot started reviewing on behalf of sainath (sainathsagar) March 23, 2026 09:44 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

sainath (sainathsagar) closed this Mar 23, 2026

Navigation Menu

Uh oh!

feat(stachv3): Implemented a common methods to convert stachv3 json & arrow to table. - #54

feat(stachv3): Implemented a common methods to convert stachv3 json & arrow to table.#54
sainath (sainathsagar) wants to merge 5 commits into
mainfrom
feat/stachv3_changes

sainath (sainathsagar) commented Mar 23, 2026

Uh oh!

Rico (RicoFactset) commented Mar 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-                        dataRow.put(columnName, securityNameColumnData != null ? getColumnValue(securityNameColumnData, rowIndex - 1) : "");
+                        Map<String, String> previousRow = arrowContext.dataTable.get(rowIndex - 1);
+                        if (previousRow != null) {
+                            String previousGroupValue = previousRow.get(columnName);
+                            if (previousGroupValue != null) {
+                                dataRow.put(columnName, previousGroupValue);
+                            }
+                        }

	public void ConvertArrowStremToTable_WithCompleteStachV3Structure_SuccessfullyConverts()
	public void ConvertArrowStreamToTable_WithCompleteStachV3Structure_SuccessfullyConverts()

Uh oh!

Conversation

sainath (sainathsagar) commented Mar 23, 2026

Uh oh!

Rico (RicoFactset) commented Mar 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants