Migrate data from old veda-features-api stack

[ividito]

Description

We need to migrate the legacy features API (https://firenrt.delta-backend.com/) which is hosted in UAH to the production features API which is deployed to: https://openveda.cloud/api/features/

Periodic ingests in production will be happen using the SM2A automated vector DAG which will run on chron.

This stack is the one we are providing ongoing support to and will eventually deprecateveda-features-api.

Steps:

• tear down existing features API in MCP
• update production sm2a to veda-data-airflow release >= v0.13.6-dev.3
• create snapshot of veda-wfs3-west2-staging RDS in UAH
• encrypt and share snapshot to MCP
• deploy veda-features-api in MCP using snapshot id
• create and upload automation json in tf shared state bucket in MCP for each collection. See example from staging:

{
    "bucket": "veda-data-store-staging",
    "discovery": "s3",
    "collection": "eis-fedsoutput-lfarchive",
    "filename_regex": "^(.*)lf_(perimeter|fireline|newfirepix)_nrt_(conus|borealna|russiaeast).gpkg$",
    "prefix": "EIS/FEDSoutput/LFArchive/",
    "vector": true,
    "id_template": "{}",
    "schedule": "*/15 * * * *",
    "dag": "veda_ingest_vector"
  }

• update sm2a deploy secret TF_VAR_vector_secret_name to configure the new features service ingest and cycle change (re-deploy)

Tuning and acceptance testing moved to NASA-IMPACT/veda-architecture#600

• openveda.cloud/api/features <> firenrt.delta-backend.com parity testing and documentation
• enable cloudfront caching for api/features
• enable cloudfront invalidation in sm2a vector ingest by updating the SM2A secret and the scheduled configs
• update docs.openveda.cloud API links and notebook

[smohiudd]

@anayeaye do you have a list of the collections that have periodic ingests?

[anayeaye]

@anayeaye do you have a list of the collections that have periodic ingests?

I don't but I think with the additional logging we have added to the vector ingest DAG in SM2A we will be able to infer the list of collections from the logs.

[anayeaye]

Here are some cron schedules we are working around for the snapshot and statup window

https://github.com/Earth-Information-System/fireatlas/blob/conus-dps/.github/workflows/schedule-nrt-date-update-current-day.yaml

https://github.com/Earth-Information-System/fireatlas/blob/conus-dps/.github/workflows/schedule-conus-nrt-v3.yaml

[anayeaye]

prior art encrypting and sharing snapshots
NASA-IMPACT/veda-deploy#70 (comment)

[anayeaye]

7:07 PM MDT mwaa done with s3 event ingests
7:08 snapshot started
7:15 snapshot done
7:17 encryption started
7:32 encrypted & shared
7:33 local mcp copy started
7:52 copy available
8:30 features api + db deployed, sm2a update started
manual cloudfront edit for new apigw url
https://openveda.cloud/api/features/
scheduled ingests are live: validate tomorrow

[anayeaye]

I am moving this to the review stage instead of done because we need to determine if the ingest gap during the day March 26 is long enough to require backfilling data or starting from a fresh clone.

cc: @smohiudd