[OpenResty Bouncer] 403 Forbidden on LAPI fallback/live query during metrics scraping (mTLS + Stream mode)

[amoureau]

What happened?

I have configured the crowdsec-openresty-bouncer in Stream mode with mTLS authentication. The stream synchronization works perfectly (the bouncer successfully fetches the stream via /v1/decisions/stream using the client certificates).

However, every 15 minutes, when our Prometheus server scrapes the Nginx metrics endpoint (port 9145, GET /metrics), the bouncer generates a 403 error in the Nginx error log.

The log shows that the bouncer is attempting to query the Live API (/v1/decisions?ip=...), and this specific Live LAPI request is rejected with an HTTP 403 error.

Error log:
[error] 229659#229659: *665493 [lua] crowdsec.lua:689: Allow(): [Crowdsec] bouncer error: Http error 403 while talking to LAPI (https://crowdsec-lapi.numberly.dev/v1/decisions?ip=10.6.31.28), client: 10.X.X.X, server: , request: "GET /metrics HTTP/1.1", host: "cdn1-node1.dv.par5.numberly.net:9145"

It seems that the bouncer is unexpectedly falling back to the Live API for this specific server block/port, and failing to pass the mTLS certificates for this specific call (or the LAPI rejects it).

What did you expect to happen?

If falling back to the Live API is the intended behavior in this case, I expect the bouncer to successfully use the configured mTLS certificates (TLS_CERT_PATH, TLS_KEY_PATH) just as it does for the stream sync, returning a 200 OK rather than a 403 Forbidden.

How can we reproduce it (as minimally and precisely as possible)?

Here is the steps we followed (but it might come from how we coded the whole thing) :

Set up a CrowdSec LAPI requiring mTLS authentication (no API key).
Install crowdsec-openresty-bouncer on an Nginx node.
Configure the bouncer with MODE=stream and provide valid TLS_CERT_PATH / TLS_KEY_PATH.
Ensure the Nginx worker (www-data) has the correct Linux permissions to read the private key.
Expose a Prometheus metrics endpoint on a different port (e.g., 9145) with CrowdSec access logic enabled.
Perform periodic GET /metrics requests to that port.

Anything else we need to know?

No response

Crowdsec version

Details

$ cscli version
version: v1.7.4-debian-pragmatic-amd64-469b374e
Codename: alphaga
BuildDate: 2025-12-04_15:31:39
GoVersion: 1.25.3
Platform: linux
libre2: C++
User-Agent: crowdsec/v1.7.4-debian-pragmatic-amd64-469b374e-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog, db_mysql, db_postgres, db_sqlite

OS version

Details

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

$ uname -a
Linux crowdsec1-node1.dv.par5.numberly.net 6.8.0-90-generic crowdsecurity/crowdsec#91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

Details

$ cscli hub list -o raw
# paste output here

Acquisition config

Details

# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
# paste output here
# On Windows:
C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
# paste output here

Config show

Details

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Notification Folder    : /etc/crowdsec/notifications
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 16
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : https://crowdsec-lapi.numberly.dev/
  - Login                   : 
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:443
  - Listen Socket           : 
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Cert File : /etc/ssl/certs/host-full_chain.cert
  - Key File  : /etc/ssl/private/host-certificate.key
  - CA Cert   : /etc/ssl/certs/host-ca_chain.cert
  - Allowed Agents OU       : IT servers
  - Allowed Agents OU       : team-infrastructure
  - Allowed Bouncers OU       : IT servers
  - Allowed Bouncers OU       : team-infrastructure

  - Trusted IPs:
      - X.X.X.X
  - Database:
      - Type                : postgres
      - Host                : rw-pgsql3-dv.numberly.in
      - Port                : 5433
      - User                : crowdsec_dv
      - DB Name             : crowdsec_dv
      - Max Open Conns      : 100
      - Flush age           : 168h0m0s
      - Flush size          : 5000

Prometheus metrics

Details

$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

Details

[github-actions]

@amoureau: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[blotus]

Hello,

The bouncer only supports a single way for querying LAPI so this does seem very strange: it looks like that for some reason this server block is using a different auth method + pull type (but not endpoint ??).

Can you provide the configuration used by the bouncer and also confirm you see requests in the LAPI logs for /v1/decisions/stream coming from the bouncer ?

What is serving the /metrics endpoint ? Is nginx just proxifying the request to an upstream or is it handled by a nginx module ?

You can also put the bouncer in debug mode by adding debug to your error_log directive, this could yield potentially useful logs (but be warned, this might be very verbose)

[amoureau]

Bouncer config :

cat /etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf
ENABLED=true
API_URL=https://crowdsec-lapi.numberly.dev
API_KEY=
USE_TLS_AUTH=true
TLS_CLIENT_CERT=/etc/crowdsec/bouncers/certs/host.cert
TLS_CLIENT_KEY=/etc/crowdsec/bouncers/certs/host.key
CACHE_EXPIRATION=10
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# By default internal requests are ignored, such as any path affected by rewrite rule.
# set ENABLE_INTERNAL=true to allow checking on these internal requests.
ENABLE_INTERNAL=false
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=
# Captcha Secret Key
SECRET_KEY=
# Captcha Site key
SITE_KEY=
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/templates/captcha.html
CAPTCHA_EXPIRATION=3600

APPSEC_URL=
APPSEC_FAILURE_ACTION=passthrough
APPSEC_CONNECT_TIMEOUT=
APPSEC_SEND_TIMEOUT=
APPSEC_PROCESS_TIMEOUT=
ALWAYS_SEND_TO_APPSEC=false
SSL_VERIFY=true

I checked my crowdsec.log, 200 OK stream requests from bouncers don't seem to be logged. However, the stream synchronization seems to work with my tests: the bouncer successfully blocks bad IPs on regular traffic.

The /metrics endpoint is not proxified to an upstream. It is handled directly by Nginx using the knyar/nginx-lua-prometheus module.
In my nginx.conf, both CrowdSec and Prometheus are initialized in the same init_worker_by_lua_block:

prometheus = require("prometheus").init("prometheus_metrics")
-- [...]
cs = require "crowdsec"
local mode = cs.get_mode()
if string.lower(mode) == "stream" then
    cs.SetupStream()
end

Then, the /metrics endpoint is exposed via a content_by_lua_block (to print the Prometheus metrics).