Add api request debug and improve table truncation

[sprucely]

Summary by CodeRabbit

• New Features

  • Added warning logs for failed API requests, including status, method, path, and response size.
  • Import processing now reports timing details for major stages, helping surface slower runs.

• Bug Fixes

  • Improved import handling for datasets without unique keys by using a direct replacement path.
  • Updated target-table replacement behavior to use a more efficient truncate-and-load approach.

[coderabbitai]

Warning

Review limit reached

@sprucely, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 44 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2e326383-2d3c-4c77-8fdc-6ffcd4e5ee35

📥 Commits

Reviewing files that changed from the base of the PR and between d54c522 and b8ac877.

📒 Files selected for processing (2)

• server/Helpers/LoggingMiddlewareHelper.cs
• server/Import/FlatFileImportService.cs

📝 Walkthrough

Walkthrough

Adds an API failure logging middleware (UseApiFailureLogging) registered in the pipeline to warn-log failed /api responses. Adds Stopwatch-based timing/logging to flat-file import, plus a new direct truncate-and-bulk-copy replacement path for target tables without unique keys, and updates target table SQL to TRUNCATE-based logic.

Changes

API Failure Logging

Layer / File(s)	Summary
Middleware and pipeline wiring server/Helpers/LoggingMiddlewareHelper.cs, server/Program.cs	New UseApiFailureLogging extension logs warnings for /api requests returning status >= 400, with method, path, and content length; registered in the pipeline before authentication/authorization.

Flat-File Import Performance and Direct Replacement

Layer / File(s)	Summary
Import timing/logging server/Import/FlatFileImportService.cs	Adds Stopwatch-based logging for import parsing, persistence, and total completion durations in ImportAsync.
Direct target table replacement server/Import/FlatFileImportService.cs	Adds a fast path in ReplaceTargetTableAsync for datasets without unique keys, calling new ReplaceTargetTableDirectlyAsync (truncate + bulk copy + commit) and TruncateTargetTableAsync helper; changes TargetTableSql to TRUNCATE + INSERT...SELECT instead of DELETE...INSERT.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ImportAsync
  participant ReplaceTargetTableAsync
  participant ReplaceTargetTableDirectlyAsync
  participant TruncateTargetTableAsync
  participant Database

  ImportAsync->>ReplaceTargetTableAsync: replace target table
  alt no unique keys
    ReplaceTargetTableAsync->>ReplaceTargetTableDirectlyAsync: call direct path
    ReplaceTargetTableDirectlyAsync->>TruncateTargetTableAsync: truncate target table
    TruncateTargetTableAsync->>Database: TRUNCATE TABLE
    ReplaceTargetTableDirectlyAsync->>Database: bulk copy rows
    ReplaceTargetTableDirectlyAsync->>Database: commit transaction
  else has unique keys
    ReplaceTargetTableAsync->>Database: staging validation + TRUNCATE/INSERT SELECT
  end
  ReplaceTargetTableAsync-->>ImportAsync: return

Loading

Possibly related PRs

• ucdavis/AD419#22: Introduces the flat-file import pipeline and backend import service that this PR's ReplaceTargetTableAsync/timing changes modify.

Suggested reviewers: rmartinsen-ucd

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main changes: API failure logging/debug output and improved table truncation behavior.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch swe/ImportDebug

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@server/Helpers/LoggingMiddlewareHelper.cs`:
- Around line 9-21: The logging middleware in LoggingMiddlewareHelper currently
only logs after await next(), so exceptions from downstream API handlers skip
logging entirely. Update the app.Use callback to wrap next() in a try/catch (or
equivalent filtered catch) and log unhandled exceptions for /api requests before
re-throwing, while keeping the existing status-code-based warning path for
non-exception failures.

In `@server/Import/FlatFileImportService.cs`:
- Around line 74-78: The import start log in Import/FlatFileImportService should
not write the raw IFormFile.FileName value. Update the logging in the import
শুরু path to sanitize the filename the same way the persistence flow does with
Path.GetFileName, or remove the filename from the log entirely, and keep the
change localized to the import-start logging call that uses
logger.LogInformation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7761a521-67ad-4f94-a292-ec7173ef6a98

📥 Commits

Reviewing files that changed from the base of the PR and between 0cf6416 and d54c522.

📒 Files selected for processing (3)

• server/Helpers/LoggingMiddlewareHelper.cs
• server/Import/FlatFileImportService.cs
• server/Program.cs