Skip to content

iceberg-catalog-glue pushes all historical schema fields to Glue, causing duplicate columns in Glue UI #2749

Description

@dahai1996

Apache Iceberg Rust version

None

Describe the bug

When using iceberg-catalog-glue, every call to update_table pushes ALL historical schema versions' fields to AWS Glue's column definition. This causes Glue Console to display duplicate columns — one entry per schema version for every field that existed in any past schema.

To Reproduce

  1. Create an Iceberg table via the Glue catalog
  2. Insert data with new fields, triggering schema evolution (AddSchema + SetCurrentSchema)
  3. Observe the table in AWS Glue Console → Table → Schema

Expected behavior

Only the current schema's fields should be pushed to Glue's column definition. Historical schema versions are already preserved in the Iceberg metadata file on S3.

https://github.com/apache/iceberg-rust/blob/main/crates/catalog/glue/src/schema.rs#L57-L63

for schema in metadata.schemas_iter() {
    if schema.schema_id() == current_schema.schema_id() {
        continue;
    }
    visit_schema(schema, &mut builder)?;
}

The from_iceberg function visits the current schema (marked as current=true), then iterates over all historical schemas and visits them too (marked as current=false). All of these are included in the Glue TableInput.storage_descriptor.columns. Glue's API does not filter by iceberg.field.current, so the UI displays all of them.

Willingness to contribute

None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions