Skip to content

Fix data mapping regression and increase import timeout#34

Merged
sprucely merged 3 commits into
mainfrom
swe/SplitEfMappings
Jul 8, 2026
Merged

Fix data mapping regression and increase import timeout#34
sprucely merged 3 commits into
mainfrom
swe/SplitEfMappings

Conversation

@sprucely

@sprucely sprucely commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • New Features

    • Added support for a separate data database alongside the main application database.
    • Health checks now include the new data database connection.
  • Bug Fixes

    • Updated imports, seeding, and chart segment actions to use the correct database backend.
    • Improved database command handling during flat-file imports with more consistent timeouts.
  • Tests

    • Updated test setup to use the new data database context where needed.
    • Added in-memory test support for the data database.

@sprucely sprucely requested a review from rmartinsen-ucd July 7, 2026 21:01
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@sprucely, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 22 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7791a9f9-f5fd-4077-aae9-c0d7ddd2f3ee

📥 Commits

Reviewing files that changed from the base of the PR and between b9b0a69 and 07d2aff.

📒 Files selected for processing (4)
  • server.core/Data/DataDbConnection.cs
  • server.core/Import/PgmProjectsImportService.cs
  • server/Import/FlatFileImportService.cs
  • tests/server.tests/Import/FlatFileImportServiceTests.cs
📝 Walkthrough

Walkthrough

This PR splits the single AppDbContext into two contexts: AppDbContext (ImportLog, User) and a new DataDbContext (ChartStringSegment, hierarchy entities). Seeders, controllers, import services, DI registration, health checks, and tests are updated to use DataDbContext where applicable.

Changes

AppDbContext/DataDbContext split

Layer / File(s) Summary
Define DataDbContext and trim AppDbContext
server.core/Data/DataDbContext.cs, server.core/Data/AppDbContext.cs, server.core/createMigration.sh
New DataDbContext maps ChartStringSegment and hierarchy entities with schema/key/length configuration; AppDbContext retains only ImportLog and User; migration script targets AppDbContext explicitly.
Seeders and initializer wiring
server.core/Data/ChartStringSegmentSeed.cs, server.core/Data/HierarchySeed.cs, server.core/Data/DbInitializer.cs
Seed methods now accept DataDbContext; DbInitializer adds a _dataDb field and routes development seeding to it.
Controller and import services use DataDbContext
server/Controllers/ChartStringSegmentsController.cs, server.core/Import/PgmProjectsImportService.cs, server/Import/FlatFileImportService.cs
Controller and import services switch their DB dependency to DataDbContext; FlatFileImportService adds a DatabaseCommandTimeoutSeconds constant applied to SQL command and bulk-copy timeouts.
DI registration and health checks
server/Program.cs
Registers DataDbContext with a resolved data connection string and adds a "data_db" health check.
Test factory and test updates
tests/server.tests/TestDbContextFactory.cs, tests/server.tests/ChartStringSegments/*, tests/server.tests/Import/FlatFileImportServiceTests.cs
Adds CreateDataInMemory() factory method; updates chart/hierarchy and flat file import tests to use the new in-memory DataDbContext.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Program
  participant DataDbContext
  participant DbInitializer
  participant HierarchySeed
  participant ChartStringSegmentSeed

  Program->>DataDbContext: register with resolved data connection
  Program->>DbInitializer: construct with AppDbContext and DataDbContext
  DbInitializer->>HierarchySeed: EnsureSeededAsync(DataDbContext)
  DbInitializer->>ChartStringSegmentSeed: EnsureSeededAsync(DataDbContext)
  HierarchySeed-->>DataDbContext: save hierarchy rows
  ChartStringSegmentSeed-->>DataDbContext: save chart string segments
Loading

Possibly related PRs

  • ucdavis/AD419#10: Both PRs implement schema-ownership changes around AppDbContext startup behavior related to the split introduced here.
  • ucdavis/AD419#20: Relates to PgmProjectsImportService's use of DataDbContext for destination connection strings during import.
  • ucdavis/AD419#22: Relates to the FlatFileImportService refactor to use DataDbContext as part of the flat-file import flow.

Suggested reviewers: rmartinsen-ucd

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main changes: fixing the data mapping split and increasing import timeouts.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch swe/SplitEfMappings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (3)
server.core/Data/DataDbContext.cs (1)

39-61: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Naming-convention-based MaxLength sizing is fragile for future properties.

ConfigureHierarchy derives max length purely from property name suffix (Description → 1000, ends-with Name → 1000, else → 20). Any future string property on a hierarchy entity that isn't named Description/*Name (e.g. a ParentCode, ShortCode, or similar lookup key) silently falls into the 20-char default, which could truncate data or mismatch a related Code column (already explicitly sized at 50). Since this configuration is convention-driven rather than explicit per-property, it's easy for a new column to get an unintended narrow length without anyone noticing at compile time.

Consider making the length explicit per-property (or adding an allow-list/deny-list with a fail-fast default) rather than relying on naming conventions to select the width.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server.core/Data/DataDbContext.cs` around lines 39 - 61, The MaxLength setup
in ConfigureHierarchy is too dependent on property-name conventions and can
silently assign the wrong size to future string properties. Update the
string-property configuration in ConfigureHierarchy<T> to use explicit
per-property sizing (or a strict allow-list with a fail-fast default) instead of
the current switch on property.Name, and make sure known fields like
ISegmentHierarchy.Code, Description, and any other hierarchy lookup/code fields
are intentionally sized.
server.core/Import/PgmProjectsImportService.cs (1)

16-16: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicated 600s timeout constant across import services.

CommandTimeoutSeconds here and DatabaseCommandTimeoutSeconds in FlatFileImportService both hardcode 600s independently. Consider a shared constant (e.g., on DataDbConnection) if these are meant to stay in sync.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server.core/Import/PgmProjectsImportService.cs` at line 16, The 600-second
command timeout is duplicated across import services, so update
PgmProjectsImportService to stop hardcoding its own timeout and instead reuse a
shared timeout source used by FlatFileImportService. Move the timeout value to a
common symbol such as DataDbConnection (or another shared constant location) and
have both CommandTimeoutSeconds and DatabaseCommandTimeoutSeconds reference that
single definition so they stay in sync.
tests/server.tests/Import/FlatFileImportServiceTests.cs (1)

451-461: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

dataDb is never disposed in CreateService.

Every call creates a new InMemory DataDbContext that's never released, unlike the db context in each test method (await using var db = ...). With ~15+ call sites in this file, these accumulate for the life of the test process.

♻️ Proposed fix
-    private static FlatFileImportService CreateService(Server.Core.Data.AppDbContext db)
+    private static FlatFileImportService CreateService(Server.Core.Data.AppDbContext db, out DataDbContext dataDb)
     {
-        var dataDb = TestDbContextFactory.CreateDataInMemory();
+        dataDb = TestDbContextFactory.CreateDataInMemory();

         return new FlatFileImportService(
             db,
             dataDb,
             new FlatFileImportRegistry(),
             new ConfigurationBuilder().Build(),
             NullLogger<FlatFileImportService>.Instance);
     }

Then update each call site to capture and dispose dataDb (e.g. await using var dataDb; var service = CreateService(db, out dataDb);).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/server.tests/Import/FlatFileImportServiceTests.cs` around lines 451 -
461, CreateService currently hides a disposable DataDbContext by constructing
dataDb internally and never releasing it, so refactor FlatFileImportServiceTests
to make disposal explicit. Update CreateService to take an out/returned dataDb
(or otherwise expose the created context), and adjust each test call site that
uses CreateService so it captures dataDb and disposes it with await using
alongside db. Keep the changes centered around CreateService and the
FlatFileImportService test helpers so every in-memory context created in this
file is cleaned up.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@server.core/Data/DataDbContext.cs`:
- Around line 39-61: The MaxLength setup in ConfigureHierarchy is too dependent
on property-name conventions and can silently assign the wrong size to future
string properties. Update the string-property configuration in
ConfigureHierarchy<T> to use explicit per-property sizing (or a strict
allow-list with a fail-fast default) instead of the current switch on
property.Name, and make sure known fields like ISegmentHierarchy.Code,
Description, and any other hierarchy lookup/code fields are intentionally sized.

In `@server.core/Import/PgmProjectsImportService.cs`:
- Line 16: The 600-second command timeout is duplicated across import services,
so update PgmProjectsImportService to stop hardcoding its own timeout and
instead reuse a shared timeout source used by FlatFileImportService. Move the
timeout value to a common symbol such as DataDbConnection (or another shared
constant location) and have both CommandTimeoutSeconds and
DatabaseCommandTimeoutSeconds reference that single definition so they stay in
sync.

In `@tests/server.tests/Import/FlatFileImportServiceTests.cs`:
- Around line 451-461: CreateService currently hides a disposable DataDbContext
by constructing dataDb internally and never releasing it, so refactor
FlatFileImportServiceTests to make disposal explicit. Update CreateService to
take an out/returned dataDb (or otherwise expose the created context), and
adjust each test call site that uses CreateService so it captures dataDb and
disposes it with await using alongside db. Keep the changes centered around
CreateService and the FlatFileImportService test helpers so every in-memory
context created in this file is cleaned up.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e604f530-0c4f-44ad-848e-ac9089eed5d2

📥 Commits

Reviewing files that changed from the base of the PR and between 1ef6c6e and b9b0a69.

📒 Files selected for processing (17)
  • server.core/Data/AppDbContext.cs
  • server.core/Data/ChartStringSegmentSeed.cs
  • server.core/Data/DataDbContext.cs
  • server.core/Data/DbInitializer.cs
  • server.core/Data/HierarchySeed.cs
  • server.core/Import/PgmProjectsImportService.cs
  • server.core/createMigration.sh
  • server/Controllers/ChartStringSegmentsController.cs
  • server/Import/FlatFileImportService.cs
  • server/Program.cs
  • tests/server.tests/ChartStringSegments/ChartStringSegmentMappingTests.cs
  • tests/server.tests/ChartStringSegments/ChartStringSegmentSeedTests.cs
  • tests/server.tests/ChartStringSegments/ChartStringSegmentsControllerTests.cs
  • tests/server.tests/ChartStringSegments/HierarchyMappingTests.cs
  • tests/server.tests/ChartStringSegments/HierarchySeedTests.cs
  • tests/server.tests/Import/FlatFileImportServiceTests.cs
  • tests/server.tests/TestDbContextFactory.cs
💤 Files with no reviewable changes (1)
  • server.core/Data/AppDbContext.cs

@sprucely sprucely merged commit 0cf6416 into main Jul 8, 2026
5 checks passed
rmartinsen-ucd added a commit that referenced this pull request Jul 8, 2026
PR #34 split the data schema entities into DataDbContext, so the
PurposeHierarchy DbSet, mapping, seeds, and controller lookup follow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants