Feature Proposal: Enterprise-grade API enhancements (Rate Limiting, Webhooks, Audit Logging)

[novastate]

Feature Proposal: Enterprise-grade API enhancements (Rate Limiting, Webhooks, Audit Logging)

Hi @jaredhendrickson13,

I've been using this excellent API package in production and found myself needing several enterprise features. Rather than maintaining a private fork, I'd love to contribute these back to the project if you're interested.

What I've Implemented

I've developed and tested the following features on pfSense 2.8.1:

1. Rate Limiting

Configurable request throttling to protect against abuse:

• Per-minute and per-hour limits (default: 60/min, 1000/hour)
• Burst allowance for traffic spikes
• Per-IP and per-API-key tracking
• Whitelist support for trusted IPs
• Standard headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset

2. Webhook Events

Real-time notifications when resources change:

• Event patterns: firewall.*, *.created, *.updated, *.deleted
• HMAC-SHA256 signatures for verification
• Automatic retries (10s, 60s, 300s)
• Async delivery (non-blocking)

3. Audit Logging

Complete request/response logging for compliance:

• Full request details (method, endpoint, user, IP, duration)
• Automatic PII masking (passwords, secrets, tokens)
• Log rotation (10MB files, 10 files max)
• Syslog integration for SIEM systems
• Configurable retention (7-365 days)

4. Response Caching

Performance optimization with intelligent caching:

• APCu integration (with file fallback)
• ETag support with If-None-Match
• 304 Not Modified responses
• Configurable TTL (30-3600 seconds)

5. Cursor Pagination

Stable pagination for large datasets:

• Cursor-based navigation (stable during data changes)
• Forward/backward navigation
• HATEOAS prev/next links
• Backward compatible with limit/offset

6. New Endpoints

• GET /api/v2/system/health - System health checks (disk, memory, CPU, services, gateways)
• GET/POST /api/v2/system/backup - Backup management with optional encryption
• POST /api/v2/system/backup/restore - Restore with partial restore support
• GET/DELETE /api/v2/system/cache - Cache statistics and management
• GET /api/v2/audit/logs - Query audit logs
• POST /api/v2/diagnostics/ping - Run ping from pfSense
• POST /api/v2/diagnostics/traceroute - Run traceroute from pfSense
• POST /api/v2/diagnostics/dns_lookup - DNS lookups from pfSense
• Full webhook CRUD endpoints

7. Bug Fixes

• Fixed FirewallRule.gateway ForeignModelField including RoutingGatewayStatus (non-unique name field)
• Fixed PortForward.associated_rule_id using ForeignModelField for non-unique field
• Fixed memory exhaustion when reading large log files
• Fixed unsafe $_SERVER access (16 instances)
• Fixed unsafe array access after explode() (8 instances)
• Fixed CURLOPT_SSL_VERIFYHOST value (was 1, should be 2)
• Fixed PHP 8.1+ DateTime::getLastErrors() compatibility

Technical Details

• All features disabled by default - Zero impact on existing installations
• Backward compatible - No breaking changes
• New settings in RESTAPISettings model for configuration
• 43 new/modified files across Core, Models, Endpoints, and Responses

Questions

1. Are you interested in PRs for any/all of these features?
2. Would you prefer one large PR or separate PRs per feature?
3. Any architectural concerns or coding standards I should follow?

I'm happy to split this into smaller, focused PRs if that's easier to review. I can also add tests and documentation as needed.

Thanks for building such a solid foundation!

[IAmParadox27]

+1 just for logging, particularly useful to try and diagnose why a request is getting lost in consumers that don't log

[jaredhendrickson13]

Wow this seems like a lot of hard work! Smaller, specific PRs are definitely preferred. In general, the design philosophy for this project is to simply leverage what pfSense already provides, not reinvent it; so some of these may also go against that principle. Some of these changes would conflict with changes in the next_minor branch (will be in v2.7.0) as well. Here's what I think for each:

1. Rate Limiting

I'd be interested in this if it does not add too much complexity and can be implemented in a way that rate limits are imposed using the REST API's Access List feature. That way specific IPs, subnets and/or users can all have different rate limits.

Ultimately, the value of rate limiting is relatively small for this API since pfSense should not be exposed to the public internet, pfSense is not intended to be multi-tenant, and login protection mitigates brute force risks already.

2. Webhook Events

I don't think there would be enough demand to consider adding and supporting this at the moment.

3. Audit Logging

I'm definitely open to this. However, it will likely be better implemented after v2.7.0. pfSense already has builtin logging facilities for packages which will be leveraged in v2.7.0. pfSense's built-in log retention, rotation and remote syslog settings will automatically work with the package logs.

4. Response Caching

TL;DR: This is already being implemented in v2.7.0 using a different approach.

v2.7.0's primary focus is this exact thing, but it is a tricky thing to handle on pfSense. There are several issues with implementing something like this:

1. The APCu extension is not installed on pfSense by default and the REST API is purposefully designed to not introduce new system level dependencies for security and compatibility reasons.
2. Persistent caching like this is not ideal because it allows for inconsistencies between the REST API and the webConfigurator. The cache files would not be aware of changes made in the webConfigurator. Such inconsistencies can result in corrupted configs or unexpected results when users rely on both the REST API and the webConfigurator.

The middle ground (and what is already implemented in the next_minor branch) is that each session will cache and index objects that have already been loaded in memory as static properties. Cached objects will automatically be cleared after any change as well. This will still greatly improve performance for users with larger configurations as it ensures the API does not constantly reload objects that have already been loaded. The drawback here is that the cache will only persist for that request.

In the event that Netgate does integrate something like APCu or Redis in pfSense in the future, this approach could easily be expanded to a more persistent caching experience.

5. Cursor Pagination

Definitely interested in this if it does not add too much complexity. PR is welcome.

6. New Endpoints

I'd be interested in these endpoints particularly:

• GET/POST /api/v2/system/backup (but rename to /api/v2/diagnostics/backup to better match webConfigurator)
• POST /api/v2/system/backup/restore (but rename to /api/v2/diagnostics/restore to better match webConfigurator)
• POST /api/v2/diagnostics/ping
• POST /api/v2/diagnostics/traceroute
• POST /api/v2/diagnostics/dns_lookup

PRs are welcome for each.

7. Bug Fixes

Some of these are already addressed in v2.7.0, but PRs for bug fixes are always welcome. Just make sure they are separate PRs for each bug fix.

---

For coding standards and project structure, you can refer to advanced topics documentation.

Thanks!

[novastate]

Thanks for the detailed feedback! Good to know about the v2.7.0 changes - I'll check the next_minor branch before doing anything.
I'll start with the diagnostic endpoints (ping/traceroute/dns_lookup) since those seem straightforward. Will rename to /diagnostics/* as you suggested.

For cursor pagination - any preference on the cursor format?

Will submit separate PRs. Thanks!