Just upgraded my environment to the current development state. I still have legacy event rules configured, so I expected some errors in the log, but then found a few more and spontaneously felt that they should be rather not flagged as error.
And my proposal now is to lower the log level of such and similar as they are not the origin of the error.
If the goal is to always include everything related in the same level (to not need wait for further feedback in case of a report), just close this as I don't think the messages can be improved, they are descriptive enough for anyone familiar with the schema and inner workings. 🤷♂️
Full log
2026-06-01T12:11:43.511Z ERROR runtime-updates Cannot validate entry, skipping element {"table": "rule", "id": 3, "name": "test", "source_id": 1, "error": "missing 'ast' field in object filter expression"}
2026-06-01T12:11:43.511Z ERROR runtime-updates Cannot validate entry, skipping element {"table": "rule", "id": 5, "name": "test2", "source_id": 1, "error": "missing 'ast' field in object filter expression"}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "channel", "took": "20.87µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 3}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contact", "took": "5.1µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contact_address", "took": "4.05µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contactgroup", "took": "2.23µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contactgroup_member", "took": "21.181µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "schedule", "took": "4.01µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2}
2026-06-01T12:11:43.513Z DEBUG channel Initializing channel plugin {"channel": {"id": 1, "name": "mail", "type": "email"}}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rotation", "took": "2.52µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1}
2026-06-01T12:11:43.513Z DEBUG channel Initializing channel plugin {"channel": {"id": 3, "name": "web2", "type": "webhook"}}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rotation_member", "took": "4.62µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2}
2026-06-01T12:11:43.513Z DEBUG channel Starting new channel plugin process {"channel": {"id": 3, "name": "web2", "type": "webhook"}, "path": "/usr/libexec/icinga-notifications/channels/webhook"}
2026-06-01T12:11:43.513Z DEBUG channel Initializing channel plugin {"channel": {"id": 2, "name": "web1", "type": "webhook"}}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "timeperiod", "took": "3.04µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1}
2026-06-01T12:11:43.513Z DEBUG channel Starting new channel plugin process {"channel": {"id": 2, "name": "web1", "type": "webhook"}, "path": "/usr/libexec/icinga-notifications/channels/webhook"}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "timeperiod_entry", "took": "7.98µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rule", "took": "2.77µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1}
2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation", "id": 3, "new": {"id": 3, "rule_id": 3, "name": "(no recipients)"}, "error": "creation callback error, rule escalation refers unknown rule 3"}
2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation", "id": 4, "new": {"id": 4, "rule_id": 3, "name": "(no recipients)", "condition": "incident_severity>warning"}, "error": "creation callback error, rule escalation refers unknown rule 3"}
2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation", "id": 9, "new": {"id": 9, "rule_id": 5, "name": "(no recipients)"}, "error": "creation callback error, rule escalation refers unknown rule 5"}
2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rule_escalation", "took": "24.52µs", "faulty_elements": 3, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1}
2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation_recipient", "id": 3, "new": {"id": 3, "rule_escalation_id": 3, "contact_id": 2}, "error": "creation callback error, rule escalation recipient refers to unknown escalation 3"}
2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation_recipient", "id": 4, "new": {"id": 4, "rule_escalation_id": 4, "contact_id": 3}, "error": "creation callback error, rule escalation recipient refers to unknown escalation 4"}
2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation_recipient", "id": 10, "new": {"id": 10, "rule_escalation_id": 9, "contact_id": 3}, "error": "creation callback error, rule escalation recipient refers to unknown escalation 9"}
Just upgraded my environment to the current development state. I still have legacy event rules configured, so I expected some errors in the log, but then found a few more and spontaneously felt that they should be rather not flagged as error.
The ones I got are:
rule escalation refers unknown rule <id>rule escalation recipient refers to unknown escalation <id>All six of them can be traced back to one of the two
missing 'ast' field in object filter expressionAnd my proposal now is to lower the log level of such and similar as they are not the origin of the error.
If the goal is to always include everything related in the same level (to not need wait for further feedback in case of a report), just close this as I don't think the messages can be improved, they are descriptive enough for anyone familiar with the schema and inner workings. 🤷♂️
Full log
2026-06-01T12:11:43.511Z ERROR runtime-updates Cannot validate entry, skipping element {"table": "rule", "id": 3, "name": "test", "source_id": 1, "error": "missing 'ast' field in object filter expression"} 2026-06-01T12:11:43.511Z ERROR runtime-updates Cannot validate entry, skipping element {"table": "rule", "id": 5, "name": "test2", "source_id": 1, "error": "missing 'ast' field in object filter expression"} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "channel", "took": "20.87µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 3} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contact", "took": "5.1µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contact_address", "took": "4.05µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contactgroup", "took": "2.23µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "contactgroup_member", "took": "21.181µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "schedule", "took": "4.01µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2} 2026-06-01T12:11:43.513Z DEBUG channel Initializing channel plugin {"channel": {"id": 1, "name": "mail", "type": "email"}} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rotation", "took": "2.52µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1} 2026-06-01T12:11:43.513Z DEBUG channel Initializing channel plugin {"channel": {"id": 3, "name": "web2", "type": "webhook"}} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rotation_member", "took": "4.62µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2} 2026-06-01T12:11:43.513Z DEBUG channel Starting new channel plugin process {"channel": {"id": 3, "name": "web2", "type": "webhook"}, "path": "/usr/libexec/icinga-notifications/channels/webhook"} 2026-06-01T12:11:43.513Z DEBUG channel Initializing channel plugin {"channel": {"id": 2, "name": "web1", "type": "webhook"}} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "timeperiod", "took": "3.04µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1} 2026-06-01T12:11:43.513Z DEBUG channel Starting new channel plugin process {"channel": {"id": 2, "name": "web1", "type": "webhook"}, "path": "/usr/libexec/icinga-notifications/channels/webhook"} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "timeperiod_entry", "took": "7.98µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 2} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rule", "took": "2.77µs", "faulty_elements": 0, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1} 2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation", "id": 3, "new": {"id": 3, "rule_id": 3, "name": "(no recipients)"}, "error": "creation callback error, rule escalation refers unknown rule 3"} 2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation", "id": 4, "new": {"id": 4, "rule_id": 3, "name": "(no recipients)", "condition": "incident_severity>warning"}, "error": "creation callback error, rule escalation refers unknown rule 3"} 2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation", "id": 9, "new": {"id": 9, "rule_id": 5, "name": "(no recipients)"}, "error": "creation callback error, rule escalation refers unknown rule 5"} 2026-06-01T12:11:43.513Z INFO runtime-updates Applied configuration updates {"table": "rule_escalation", "took": "24.52µs", "faulty_elements": 3, "deleted_unknown_elements": 0, "deleted_elements": 0, "updated_elements": 0, "new_elements": 1} 2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation_recipient", "id": 3, "new": {"id": 3, "rule_escalation_id": 3, "contact_id": 2}, "error": "creation callback error, rule escalation recipient refers to unknown escalation 3"} 2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation_recipient", "id": 4, "new": {"id": 4, "rule_escalation_id": 4, "contact_id": 3}, "error": "creation callback error, rule escalation recipient refers to unknown escalation 4"} 2026-06-01T12:11:43.513Z ERROR runtime-updates Creating configuration element failed {"table": "rule_escalation_recipient", "id": 10, "new": {"id": 10, "rule_escalation_id": 9, "contact_id": 3}, "error": "creation callback error, rule escalation recipient refers to unknown escalation 9"}