Database Error - MWAA 2.10.1

[sam-gen-cop]

HI,

@zhuowang10 First of all, great work! I'm trying to add this plugin to my MWAA environment. After adding the plugin as per the steps explained here, the plugin appears in the UI and when I try to create a backfill and click "Save" I get "Database Error". The MWAA Version I'm using is 2.10.1.

[zhuowang10]

I'm not familiar with AWS, but it seems the plugin has been installed. Can you share the webserver log? there should be exceptions shown in the log when clicking "save".

[sam-gen-cop]

@zhuowang10 Here is the webserver log,

Edit:
Updated the value from 50 to 200.
submitted_by = Column(VARCHAR(200))

[zhuowang10]

aha, unexpected long submit username! glad to see it got fixed.

[sam-gen-cop]

@zhuowang10 I updated the column size to 200 here but I still get the same error, do I have to update it anywhere else as well? Thanks.

[zhuowang10]

first, table schema change won't apply after table was created, so you need to delete the backfill table "custom_backfill" to make your changes take effect, be sure to backup your data if needed, or you can directly modify the table column in your db.
Second, what are the error details this time, the error may be the same, the complaining column may not be the same.
third, the submitted_by value looks like a long empty string, I suspect it's unsupported characters.

[sam-gen-cop]

@zhuowang10 The first one was the case, thanks. After I dropped the table and restarted the MWAA, the error disappeared. The backfill submissions were successful but backfill DAGs were not triggered. For example, I placed a backfill request for the test_bash_execution DAG but the backfill DAGs were never triggered. Any idea, why?

[zhuowang10]

The backfill system dags seem not there, please deploy them according to https://github.com/linkedin/Li-Airflow-Backfill-Plugin?tab=readme-ov-file#deploy-system-dags

if deployed, check backfill_creation dag's run logs for errors if fail.

The backfill will run in backfill dag like test_bash_execution_backfill_...