Skip to content

fix(create,update): standardize duplicate key error handling across endpoints#64

Open
Bodeayman wants to merge 1 commit into
istSOS:mainfrom
Bodeayman:fix/check-duplicates
Open

fix(create,update): standardize duplicate key error handling across endpoints#64
Bodeayman wants to merge 1 commit into
istSOS:mainfrom
Bodeayman:fix/check-duplicates

Conversation

@Bodeayman
Copy link
Copy Markdown
Contributor

Related Issue

Closes #63

Problem

When creating or updating an entity with a duplicate unique key (such as name),
the API bubbled up a raw UniqueViolationError directly to the client, resulting
in 400 Bad Request responses that leaked internal database details including
table names, constraint names, and row data from the DETAIL field.

Solution

Introduced a centralized error handling mechanism for uniqueness violations:

  • utils.handle_duplicate_error — a new utility function that catches
    UniqueViolationError exceptions from asyncpg
  • Detail Parsing — uses regex to safely extract the conflicting key name
    and value from the database error, returning a clean message:
    "An entity with [column] '[value]' already exists."
  • Standardized Status Codes — responses now correctly return 409 Conflict
    instead of generic 400 Bad Request
  • Full Coverage — applied across all primary entity creation and update routers

Changes Made

  • api/app/utils/utils.py — added handle_duplicate_error with regex parsing
  • api/app/v1/endpoints/create/ — Thing, Sensor, Location, ObservedProperty,
    FeaturesOfInterest, Network
  • api/app/v1/endpoints/update/ — Thing, Sensor, Location, ObservedProperty,
    FeaturesOfInterest, Network

Verification

Scenario Before After
POST duplicate name 400 + raw DB error 409 + clean message
PATCH rename to existing 400 + constraint details 409 + clean message
Raw constraint names visible Yes No

Before

{
  "code": 400,
  "type": "error",
  "message": "duplicate key value violates unique constraint \"unique_location_name\"\nDETAIL: Key (name)=(location name 1) already exists."
}

After

{
  "code": 409,
  "type": "error",
  "message": "An entity with name 'location name 1' already exists."
}

- Added [handle_duplicate_error](cci:1://file:///c:/Myfiles/OSGeo/istSOS4/api/app/utils/utils.py:274:0-294:5) utility to reliably catch `UniqueViolationError`.
- Implemented regex parsing of database error details to provide developer-friendly
  409 Conflict responses including specific conflicting columns and values.
- Integrated exception handling across all CREATE and UPDATE endpoints for:
  Thing, Sensor, Location, ObservedProperty, FeatureOfInterest, and Network.
- Prevented internal schema/constraint leakage by sanitizing database error messages.
Comment thread api/app/utils/utils.py
from urllib.parse import parse_qs, quote, urlencode, urlparse, urlunparse

from app import EPSG, HOSTNAME, TOP_VALUE
from asyncpg.exceptions import UniqueViolationError
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to import 'UniqueViolationError' if we are not using it

Comment thread api/app/utils/utils.py
if match:
column = match.group(1)
value = match.group(2)
message = f"An entity with {column} '{value}' already exists."
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im a bit unsure about parsing e.detail and returning the extracted column/value to the client. even if we no longer expose the constraint name, we are still deriving the response from internal database error details. would it be safer to return a fixed application-level message like with this name already exists. and keep the detailed exception only for logs?

from app import AUTHORIZATION, POSTGRES_PORT_WRITE, VERSIONING
from app.db.asyncpg_db import get_pool, get_pool_w
from app.utils.utils import validate_payload_keys
from app.utils.utils import handle_duplicate_error, validate_payload_keys
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you import from asyncpg.exceptions import InsufficientPrivilegeError, UniqueViolationError as it was used but never import.

"code": 400,
"type": "error",
"message": str(e),
"message": f"[{type(e).__name__}] {str(e)}"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think we should include type(e).name or raw str(e) in the client response here. That still exposes internal implementation details and can leak database/backend information. For this case, it would be safer to return a sanitized conflict message and keep the original exception only in server-side logs

@lightning-sagar
Copy link
Copy Markdown

if this PR only checks for duplicates before the DB write, i think it is still incomplete for concurrent requests. two requests can pass the pre check at the same time and one will still hit UniqueViolationError. we still need the DB level exception handling path mapped to a sanitized 409 conflict

@ClaudioPrimerano @danistrigaro @massimiliano-cannata, what do you guys think about the concurrent request problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database Exception Leakage on Duplicate Entities

2 participants