Haproxy fails to recover from validation errors in RequirerApplicationData #404
Description
Bug Description
I see on an HAProxy deployment a validation error for RequirerApplicationData in the haproxy_route interface. This is a validation error for old relation data. I see from inspecting the Juju unit that this relation is no longer present. E.g. Invalid requirer application data for remote-cdad94e595c9470488e45c9f844dcafc
If I juju show-unit haproxy-unit/X | grep remote-cdad94e595c9470488e45c9f844dcafc, There is no matches.
I would expect in this state that the charm would be in error state, which I could resolve by skipping the charm hook via juju resolved --no-retry haproxy-unit/X, however the units are in waiting status with message Failed validating the HAProxy config.. This seems to be an unescapable state.
I note that the haproxy service on the unit actually appears to be running without error, but is not receiving any new updates per the above issue.
Impact
Low (minor issue or cosmetic problem)
Impact Rationale
The fix for this environment includes teardown and re-creation of the environment. Identified as a part of ISREQ-488
Marked as low impact, as an accepted workaround is to redeploy the units.
To Reproduce
N/A unknown.
Environment
haproxy rev302 on PS7, deployed under JAAS with Juju 3.6.12. HA deployment (3 units)
Relevant log output
unit-ingress-ps7-landscape-canonical-com-0: 09:32:20 ERROR unit.ingress-ps7-landscape-canonical-com/0.juju-log 2 validation errors for RequirerApplicationData
service
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.12/v/missing
ports
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.12/v/missing
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ingress-ps7-landscape-canonical-com-0/charm/lib/charms/haproxy/v1/haproxy_route.py", line 253, in load
return cls.model_validate_json(json.dumps(data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-ingress-ps7-landscape-canonical-com-0/charm/venv/lib/python3.12/site-packages/pydantic/main.py", line 766, in model_validate_json
return cls.__pydantic_validator__.validate_json(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for RequirerApplicationData
service
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.12/v/missing
ports
Field required [type=missing, input_value={}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.12/v/missing
unit-ingress-ps7-landscape-canonical-com-0: 09:32:20 ERROR unit.ingress-ps7-landscape-canonical-com/0.juju-log Invalid requirer application data for remote-cdad94e595c9470488e45c9f844dcafc
unit-ingress-ps7-landscape-canonical-com-0: 09:32:20 WARNING unit.ingress-ps7-landscape-canonical-com/0.juju-log Requirers defined path(s) that map to multiple backends.This can cause unintended behaviors.
unit-ingress-ps7-landscape-canonical-com-0: 09:32:20 WARNING unit.ingress-ps7-landscape-canonical-com/0.juju-log Requirers defined hostname(s) that map to multiple backends.This can cause unintended behaviors.
unit-ingress-ps7-landscape-canonical-com-0: 09:32:21 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
alexdlukens@is-bastion-ps7:~$ juju show-unit ingress-ps7-landscape-canonical-com/0 | grep remote-cdad94e595c9470488e45c9f844dcafc
alexdlukens@is-bastion-ps7:~$
alexdlukens@is-bastion-ps7:~$ juju status ingress-ps7-landscape-canonical-com
Model Controller Cloud/Region Version SLA Timestamp
redacted jaas-ps7 prodstack7/prodstack7 3.6.12 unsupported 19:34:36Z
App Version Status Scale Charm Channel Rev Exposed Message
ingress-ps7-landscape-canonical-com waiting 3 haproxy 2.8/edge 302 yes Failed validating the HAProxy config.
keepalived-ingress-ps7-landscape-canonical-com active 3 keepalived 1.33/stable 64 no VIP ready
Unit Workload Agent Machine Public address Ports Message
ingress-ps7-landscape-canonical-com/0* waiting idle 47 redacted 80,443/tcp Failed validating the HAProxy config.
keepalived-ingress-ps7-landscape-canonical-com/1* active idle redacted VIP ready
ingress-ps7-landscape-canonical-com/1 waiting idle 48 redacted 80,443/tcp Failed validating the HAProxy config.
keepalived-ingress-ps7-landscape-canonical-com/0 active idle redacted VIP ready
ingress-ps7-landscape-canonical-com/2 waiting idle 49 redacted 80,443/tcp Failed validating the HAProxy config.
keepalived-ingress-ps7-landscape-canonical-com/2 active idle redacted VIP ready
Machine State Address Inst id Base AZ Message
47 started redacted manual:redacted ubuntu@24.04 Manually provisioned machine
48 started redacted manual:redacted ubuntu@24.04 Manually provisioned machine
49 started redacted manual:redacted ubuntu@24.04 Manually provisioned machine
# systemctl status from one of the impacted units:
ubuntu@redacted:~$ systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: enabled)
Active: active (running) since Tue 2026-03-17 15:04:20 UTC; 3 days ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Main PID: 869 (haproxy)
Status: "Ready."
Tasks: 3 (limit: 9484)
Memory: 18.4M (peak: 19.0M)
CPU: 25.903s
CGroup: /system.slice/haproxy.service
├─ 869 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
└─1058 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
Mar 17 15:04:19 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [NOTICE] (869) : haproxy version is 2.8.16-0ubuntu0.24.04.1
Mar 17 15:04:19 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [NOTICE] (869) : path to executable is /usr/sbin/haproxy
Mar 17 15:04:19 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [WARNING] (869) : config : parsing [/etc/haproxy/haproxy.cfg:68] : 'peer redacted' : ignoring i>
Mar 17 15:04:19 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [WARNING] (869) : config : parsing [/etc/haproxy/haproxy.cfg:69] : 'peer redacted' : ignoring i>
Mar 17 15:04:19 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [WARNING] (869) : config : parsing [/etc/haproxy/haproxy.cfg:70] : 'peer redacted' : ignoring i>
Mar 17 15:04:20 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [WARNING] (869) : config : log format ignored for frontend 'prometheus' since it has no log address.
Mar 17 15:04:20 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [WARNING] (869) : config : Removing incomplete section 'peers haproxy_peers' (no peer named 'ingres>
Mar 17 15:04:20 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [NOTICE] (869) : New worker (1058) forked
Mar 17 15:04:20 ingress-ps7-landscape-canonical-com-01 systemd[1]: Started haproxy.service - HAProxy Load Balancer.
Mar 17 15:04:20 ingress-ps7-landscape-canonical-com-01 haproxy[869]: [NOTICE] (869) : Loading success.Additional context
No response