Incremental jobs from Oracle to Iceberg do not pick up changes

[kimtore]

Environment

• Deploy method: Kubernetes with Helm chart 0.0.14
• Olake version: 0.5.0
• OS: N/A
• Cloud Provider: GCP
• Params: chart-provided NFS and Postgres, increased memory requests/limits

Description

We are trying to set up an Oracle to Iceberg job with incremental updates. We set the cursor to a datetime field that updates when a row is updated. However, OLake doesn't seem to copy the updated data, and no Iceberg snapshot is created. New rows do not seem to get picked up either. The cursor seems stuck on the initial highest value of the datetime field.

Steps to reproduce

• Set up a table in an Oracle database with ID (pk), datetime field, and two text fields
• Set up the Oracle to Iceberg job, cursor set to datetime field, and perform initial copy
• Try to modify some records, making sure that the datetime field is changed
• Run job again

Expected behavior

• A new Iceberg snapshot appears in catalog
• New parquet files are created on destination

Actual behavior

• Zero rows ingested, no new snapshot, no new parquet files

[nayanj98]

Hi @kimtore,

We tried to reproduce this issue by following the steps you mentioned. We tested with the following data types as the cursor field:

• timestamp
• timestamp with timezone
• timestamp with local timezone

Additionally, we attempted to replicate the issue by performing backward compatibility checks. We initially ran a few full refresh and incremental syncs on v0.4.2, and then upgraded to v0.5.0 and continued running incremental syncs. However, we were still unable to reproduce the issue.

It would be helpful if you could share the following details:

• The Oracle DB version you are using
• The table creation query for the table in question
• The timezone configuration of your database

This information should help us investigate the issue further.

[nayanj98]

Hi @kimtore,

Just following up on the issue. Are you still facing this issue, or is it working fine now on your end? If the issue is still occurring, it would be helpful if you could share the above details so that we can reproduce the same and try to resolve it.

[kimtore]

Hi @nayanj98

We are still experiencing the issue. It seems that TIMESTAMP will not work, but TIMESTAMP WITH TIME ZONE works as expected.

I won't know the rest of the info for a few days as I'm off work; my colleague will follow up with more details. But I suppose the timezone is set to either Europe/Oslo or possibly UTC.

[syasiukovich]

Thanks for looking into this @nayanj98

Below are the additional details you requested, along with the test cases to reproduce the behavior.

Oracle database version

Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0

Database timezone

• Database timezone: +02:00
• Session timezone: Europe/Oslo

Table creation query used for testing

Test Case 1 – TIMESTAMP

CREATE TABLE SBX_DATAPLATTFORM.ORACLE_CURSOR_TS (
    ORDER_ID      NUMBER GENERATED BY DEFAULT AS IDENTITY,
    CUSTOMER_NAME VARCHAR2(100) NOT NULL,
    PRODUCT_NAME  VARCHAR2(100) NOT NULL,
    AMOUNT        NUMBER(12,2) NOT NULL,
    STATUS        VARCHAR2(30) NOT NULL,
    UPDATED_AT    TIMESTAMP DEFAULT SYSTIMESTAMP NOT NULL,
    CONSTRAINT PK_ORACLE_CURSOR_TS PRIMARY KEY (ORDER_ID)
);

Insert initial rows:

INSERT INTO SBX_DATAPLATTFORM.ORACLE_CURSOR_TS (CUSTOMER_NAME, PRODUCT_NAME, AMOUNT, STATUS)
VALUES ('Anna', 'Laptop', 12000, 'NEW');

INSERT INTO SBX_DATAPLATTFORM.ORACLE_CURSOR_TS (CUSTOMER_NAME, PRODUCT_NAME, AMOUNT, STATUS)
VALUES ('Bob', 'Mouse', 250, 'NEW');

INSERT INTO SBX_DATAPLATTFORM.ORACLE_CURSOR_TS (CUSTOMER_NAME, PRODUCT_NAME, AMOUNT, STATUS)
VALUES ('Charlie', 'Keyboard', 800, 'NEW');

INSERT INTO SBX_DATAPLATTFORM.ORACLE_CURSOR_TS (CUSTOMER_NAME, PRODUCT_NAME, AMOUNT, STATUS)
VALUES ('Diana', 'Monitor', 3200, 'PROCESSING');

INSERT INTO SBX_DATAPLATTFORM.ORACLE_CURSOR_TS (CUSTOMER_NAME, PRODUCT_NAME, AMOUNT, STATUS)
VALUES ('Eva', 'Docking Station', 1800, 'SHIPPED');

COMMIT;
);

Incremental changes:

INSERT INTO SBX_DATAPLATTFORM.ORACLE_CURSOR_TS
(CUSTOMER_NAME, PRODUCT_NAME, AMOUNT, STATUS)
VALUES ('Julia','Phone',5500,'NEW');

COMMIT;

UPDATE SBX_DATAPLATTFORM.ORACLE_CURSOR_TS
SET STATUS='DELIVERED',
    AMOUNT=299,
    UPDATED_AT=SYSTIMESTAMP
WHERE ORDER_ID=2;

COMMIT;
);

Test Case 2 – TIMESTAMP WITH TIME ZONE

CREATE TABLE SBX_DATAPLATTFORM.ORACLE_CURSOR_TSTZ (
    ORDER_ID      NUMBER GENERATED BY DEFAULT AS IDENTITY,
    CUSTOMER_NAME VARCHAR2(100) NOT NULL,
    PRODUCT_NAME  VARCHAR2(100) NOT NULL,
    AMOUNT        NUMBER(12,2) NOT NULL,
    STATUS        VARCHAR2(30) NOT NULL,
    UPDATED_AT    TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
    CONSTRAINT PK_ORACLE_CURSOR_TSTZ PRIMARY KEY (ORDER_ID)
);

Test Case 3 – TIMESTAMP WITH LOCAL TIME ZONE

CREATE TABLE SBX_DATAPLATTFORM.ORACLE_CURSOR_TSLTZ (
    ORDER_ID      NUMBER GENERATED BY DEFAULT AS IDENTITY,
    CUSTOMER_NAME VARCHAR2(100) NOT NULL,
    PRODUCT_NAME  VARCHAR2(100) NOT NULL,
    AMOUNT        NUMBER(12,2) NOT NULL,
    STATUS        VARCHAR2(30) NOT NULL,
    UPDATED_AT    TIMESTAMP WITH LOCAL TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
    CONSTRAINT PK_ORACLE_CURSOR_TSLTZ PRIMARY KEY (ORDER_ID)
);

Observed behavior from our testing:

With TIMESTAMP, the incremental job reports: zero rows ingested, no new snapshot, no new parquet files and subsequent runs do not pick up newer rows even though later timestamps exist in the source table.

With TIMESTAMP WITH TIME ZONE and TIMESTAMP WITH LOCAL TIME ZONE, the incremental job correctly detects both inserts and updates and produces new iceberg snapshots.

Sync configuration

The OLake job was configured with the following settings:

Setting	Value
Sync mode	Full Refresh + Incremental
Cursor field	UPDATED_AT
Fallback cursor	ORDER_ID
Ingestion mode	Append
Normalization	Enabled
Data filter	Disabled

The same configuration was used for all three test tables:

• ORACLE_CURSOR_TS
• ORACLE_CURSOR_TSTZ
• ORACLE_CURSOR_TSLTZ

The only difference between the tests is the data type of the UPDATED_AT column.

[vikaxsh]

hey @syasiukovich , It would be helpful if you could share the output of the following query so we can further debug the issue.

SELECT
  DBTIMEZONE,
  SESSIONTIMEZONE,
  TO_CHAR(SYSTIMESTAMP, 'YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM TZR') AS systimestamp_tz,
  TO_CHAR(CURRENT_TIMESTAMP, 'YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM TZR') AS current_timestamp_tz
FROM dual;

It would also help if you could share the full load logs, the generated state.json, and the next incremental sync logs along with its state.json file (you may hide any sensitive information).

[syasiukovich]

@vikaxsh, please see below

Query Result

DBTIMEZONE	SESSIONTIMEZONE	SYSTIMESTAMP_TZ	CURRENT_TIMESTAMP_TZ
+02:00	Europe/Oslo	2026-03-18 11:34:41.533335 +01:00 +01:00	2026-03-18 11:34:41.533338 +01:00 EUROPE/OSLO

[vikaxsh]

@syasiukovich ,
While reproducing this with similar database settings, I was able to observe duplicate records, but not the zero-records behavior. Could you please confirm what you are seeing on your side: are new records not being picked up at all, or are previously synced records being ingested again as duplicates?

[syasiukovich]

@vikaxsh

Here are the requested state.json files from a fresh test run.

state.json after first run

Mar 18 11:35
/data/olake-jobs/2335035be7f01e31bc59ecde9f4acf7df77d0f5cc64032988bcf7dd606f13380/state.json

{
  "type": "STREAM",
  "version": 4,
  "streams": [
    {
      "stream": "ORACLE_CURSOR_TS",
      "namespace": "SBX_DATAPLATTFORM",
      "sync_mode": "",
      "state": {
        "ORDER_ID": 5,
        "UPDATED_AT": "2026-03-18T11:33:50.908315000Z",
        "chunks": []
      }
    }
  ]
}

state.json after second run (after inserts/updates in the source table)

Mar 18 11:43
/data/olake-jobs/a870ec47d1cccce1d1c2bbb77b483ccbd9f151a10a9968be7062487ff406dd91/state.json

{
  "type": "STREAM",
  "version": 4,
  "streams": [
    {
      "stream": "ORACLE_CURSOR_TS",
      "namespace": "SBX_DATAPLATTFORM",
      "sync_mode": "",
      "state": {
        "ORDER_ID": 5,
        "UPDATED_AT": "2026-03-18T11:33:50.908315000Z",
        "chunks": []
      }
    }
  ]
}

state.json after third run (after additional inserts/updates in the source table)

Mar 18 11:57
/data/olake-jobs/e939ac6475026ae9d7277e17c0f84fc310bcf604371a1ea82af42a2d583c2905/state.json

{
  "type": "STREAM",
  "version": 4,
  "streams": [
    {
      "stream": "ORACLE_CURSOR_TS",
      "namespace": "SBX_DATAPLATTFORM",
      "sync_mode": "",
      "state": {
        "ORDER_ID": 5,
        "UPDATED_AT": "2026-03-18T11:33:50.908315000Z",
        "chunks": []
      }
    }
  ]
}

[syasiukovich]

@syasiukovich , While reproducing this with similar database settings, I was able to observe duplicate records, but not the zero-records behavior. Could you please confirm what you are seeing on your side: are new records not being picked up at all, or are previously synced records being ingested again as duplicates?

@vikaxsh

In our case, we are not seeing duplicate records. Instead, we are observing that new records and updates are not being picked up at all. After the initial run, OLake ingests the data and sets the cursor correctly. However, on subsequent incremental runs:

• No new rows are ingested (after inserts/updates in the source table)
• No new Iceberg snapshots are created
• The state.json (as shared above) remains unchanged across runs

This issue only occurs when using TIMESTAMP as the cursor column.

When using:

• TIMESTAMP WITH TIME ZONE
• TIMESTAMP WITH LOCAL TIME ZONE

the incremental sync works as expected (both inserts and updates are detected)

[vikaxsh]

hey @syasiukovich ,Thanks for sharing all the information. I’ve raised a PR and will be releasing a fix for this issue soon.