Schema backward-compatibility: first-class support for column deletion

[xiangfu0]

Feature request

Schema updates currently treat column deletion as backward-incompatible. Removing a column from a table schema is rejected unless the operator passes force=true, which bypasses all validation rather than handling the deletion safely.

Current behavior

• Schema.isBackwardCompatibleWith() (pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java) returns false if any column present in the old schema is missing from the new one.
• PinotHelixResourceManager.updateSchema() throws SchemaBackwardIncompatibleException → HTTP 400 "Only allow adding new columns", unless force=true.
• With force=true the check is skipped with only a warning log. The result is an inconsistent state:
  • Existing segments still physically contain the deleted column.
  • New segments don't have it.
  • Queries referencing the deleted column silently return null (PinotSegmentColumnReaderFactory) rather than erroring or being cleaned up.
• There is no allowDeletedColumn / managed-deletion config.

Proposed

Make column deletion a first-class, backward-compatible operation instead of an all-or-nothing force override:

1. Allow column removal through schema update without force, behind explicit intent (e.g. a dedicated flag or validation path that distinguishes "drop column" from accidental omission).
2. Define + document segment behavior for deleted columns — reclaim the column on segment reload, and decide query semantics (ignore vs. error on reference).
3. Preserve the existing guardrails (primary-key changes, type changes) — only column removal is relaxed.

Notes

• Backward-compatibility / rolling-upgrade sensitive: mixed-version brokers/servers must agree on how a queried-but-deleted column behaves.
• Related validation: FieldSpec.isBackwardCompatibleWith().