Skip to content

Any condition whose operator is not_exists is skipped when collecting "target names," #1821

Description

@pendingintent

Standard

Custom

Reference Rule ID(s)

TIG0489

Conformance Rule ID(s) (if published in CORE)

CORE-000183

JIRA Ticket

https://jira.cdisc.org/projects/CORERULES/issues/CORERULES-

CLI Command Used (if applicable)

python core.py -s SDTM -v 3.4 -d /path/to/datasets

Rule YAML

# Variable: GEN
# Condition: PP dataset present in study
# Rule: PC dataset present in study
Authorities:
  - Organization: CDISC
    Standards:
      - Name: SDTMIG
        References:
          - Citations:
              - Cited Guidance: Note, domain pairs have been modeled for microbiology data
                  (MB/MS domains) and PK data (PC/PP domains) to enable
                  dataset-level relationships to be described using RELREC.
                Document: IG v3.4
                Item: '1'
                Section: '2.6'
              - Cited Guidance: A findings domain that contains pharmacokinetic parameters
                  derived from pharmacokinetic concentration-time (PC) data.
                Document: IG v3.4
                Item: PP - Description/Overview
                Section: 6.3.5.6.2
            Origin: SDTM and SDTMIG Conformance Rules
            Rule Identifier:
              Id: CG0318
              Version: '1'
            Version: '2.0'
        Version: '3.4'
      - Name: SDTMIG
        References:
          - Citations:
              - Cited Guidance: IG v3.2[2.6][The domain pair uses DOMAIN as an Identifier to
                  group parent records (e.g., MB) from child records (e.g., MS)
                  and enables a dataset-level relationship to be described in
                  RELREC.]|IG v3.2[6.3][PP][Pharmacokinetic parameters derived
                  from pharmacokinetic concentration-time (PC) data.]
                Document: IG v3.2
                Item: IG v3.2[2.6]|IG v3.2[6.3][PP]
                Section: IG v3.2[2.6]|IG v3.2[6.3]
            Origin: SDTM and SDTMIG Conformance Rules
            Rule Identifier:
              Id: CG0318
              Version: '1'
            Version: '2.0'
        Version: '3.2'
      - Name: SDTMIG
        References:
          - Citations:
              - Cited Guidance: IG v3.3[2.6]|IG v3.3[6.3.11.2][The domain pair uses DOMAIN as an
                  Identifier to group parent records (e.g., MB) from child
                  records (e.g., MS) and enables a dataset-level relationship to
                  be described in RELREC.]|IG v3.3[6.3.11.2][A findings domain
                  that contains pharmacokinetic parameters derived from
                  pharmacokinetic concentration-time (PC) data.]
                Document: IG v3.3
                Item: IG v3.3[2.6]|IG v3.3[6.3.11.2]
                Section: IG v3.3[2.6]|IG v3.3[6.3.11.2]
            Origin: SDTM and SDTMIG Conformance Rules
            Rule Identifier:
              Id: CG0318
              Version: '1'
            Version: '2.0'
        Version: '3.3'
      - Name: TIG
        References:
          - Citations:
              - Cited Guidance: A findings domain that contains pharmacokinetic parameters
                  derived from pharmacokinetic concentration-time (PC) data.
                Document: TIG 1.0
                Item: Description
                Section: 2.8.10.17
            Origin: TIG Conformance Rules
            Rule Identifier:
              Id: TIG0489
              Version: '1'
            Version: '1.0'
        Substandard: SDTM
        Version: '1.0'
Check:
  all:
    - name: PP
      operator: exists
    - name: PC
      operator: not_exists
Core:
  Id: CORE-000183
  Status: Published
  Version: '1'
Description: Raise an error when a PP dataset is present in study, but a PC
  dataset is not present in study.
Executability: Fully Executable
Outcome:
  Message: PP dataset is present in  study, but PC dataset is not present in study.
Rule Type: Domain Presence Check
Scope:
  Classes:
    Include:
      - FINDINGS
  Domains:
    Include:
      - PC
      - PP
  Use Case: INDH
Sensitivity: Dataset

Attach any sample test data file(s)

_datasets.csv
_variables.csv
ae.csv
dm.csv
ft.csv
pp.csv

Attach any output report and/or log file(s)

results(1).csv

Expected output

results.csv

Any Additional Information

Regression in the vendored CORE engine

Check in rule.yml is PP exists AND PC not_exists. When the engine builds the error's reported values, it calls RuleProcessor._extract_targets_from_conditions in engine/cdisc_rules_engine/utilities/rule_processor.py:

for condition in conditions.values():
    if condition.get("operator") == "not_exists":
        continue          # <-- drops the PC condition entirely
    target = condition["value"].get("target")
    ...

Any condition whose operator is not_exists is skipped when collecting "target names," so for this rule the target list ends up as just ["PP"] — PC never makes it in. Downstream, generate_targeted_error_object in actions.py:150-168] computes targets_not_in_dataset from that target set, and since PC was never a target, it never gets added as "PC": "Not in dataset" — only get the PP value.

Running the engine directly against negative/01: it correctly detects the violation (PP dataset is present ... but PC dataset is not present), but the returned error value dict is only {'PP': 'pp.csv'}, matching the results(1).csv, instead of the expected results.csv which has both the PP and PC (Not in dataset) rows.

Root cause commit: ddf73a04 — "1516: Refactor target name handling to preserve order (#1570)" — introduced continue for not_exists. It was presumably intended to stop record-level rules from reporting a not-yet-existing variable as an output target, but it also silently breaks Domain Presence Check rules (Sensitivity: Dataset) that legitimately use not_exists as one half of the check — exactly this rule's pattern. CORE-000177/179/180 are the same rule type and likely hit the identical regression (worth spot-checking their results(1).csv too).

This isn't fixable from the rule YAML side — it needs a fix in the engine, e.g. not skipping not_exists conditions when the rule is a Domain Presence Check / has Dataset sensitivity, so PC still lands in targets_not_in_dataset.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

T-shirt sizing

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions