31759 - update verify flow to use migration filter

[ketaki-deodhar]

Issue #: /bcgov/entity#31759

Description of changes:

• Earlier implementation remains as is (specify corp identifiers in where_clause
• Added logic to use USE_MIGRATION_FILTER, MIG_GROUP_IDS and MIG_BATCH_IDS config values

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the lear license (Apache 2.0).

[argush3]

can we keep using futures like we did in the previous code. that way we leverage prefect's parallelization functionality.

looking into this more...will have more for you in a bit

[argush3]

try something like this:

# Process batches concurrently (Prefect futures via .submit())
futures = []

# 1) Submit all batch tasks (non-blocking)
for cnt in range(batches):
    offset = cnt * batch_size
    print(f'🚀 Submitting batch {cnt + 1}/{batches}...')
    futures.append(
        verify.submit(
            colin_engine,
            lear_engine,
            mig_corp_candidates,
            batch_size,
            offset,
        )
    )

# 2) Resolve futures (blocking) and collect results
for f in futures:
    batch_results = f.result()
    results.extend(batch_results)

print(f'🌟 Verification complete. Processed {batches} batches.')