Correct way to throttle notifications?

[kenh-sq]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate.
• Described the feature in detail and justified the reason for the request.
• Provided specific use cases and examples.

Feature description

A callback or method to know when a specific ACL packet (or packets) has left the controller.

Or, some method to know that the BLE controller will be able to handle an upcoming connection event for a given connection with sufficient time to enqueue notifications to be sent during that event.

Use cases

In our device, the BLE controller is quite busy handling many things all at once:

• Being a GATT Peripheral for a 1M PHY device
• Passively scanning for CODED PHY devices
• Updating and transmitting a CODED PHY extended advert
• Being a GATT Central to a CODED PHY device or listening to a CODED PHY periodic sync

I'd like to be able to send timestamped notifications to the 1M PHY Central connected to this device at the known connection interval with an attempt at latency no worse than the connection interval. With everything the controller is doing, I know (I've seen) that it won't be able to send a notification on every connection event and in some cases the delivery of the notification is quite delayed (ex: 180ms late when trying to use a 30ms interval).

BLE_GAP_EVENT_NOTIFY_TX events occur after the notification has been enqueued in either the controller or the host; not when its been sent. It is possible to fill up both queues and not have those enqueued notifications sent in a timely fashion. The Central ends up getting very stale data and new data gets dropped due to the queues being full.

Alternatives

Our current method is to use the value of ble_hs_hci_avail_pkts as a water line:

1. Mark the value of ble_hs_hci_avail_pkts before any timestamped notifications have been enqueued as the water line (grabbed within a ble_hs_lock()/ble_hs_unlock() block).
2. Enqueue timestamped notification(s) (ble_gatts_notify_custom()).
3. Don't enqueue new timestamped data until ble_hs_hci_avail_pkts is at or above the water line (value grabbed within a lock/unlock).

This might be flawed in some manner, but we've had success with this approach for a while.

Additional context

Given the HCI API, I know my alternate feature request (callback before when the controller might be able to handle a connection event for a given connection) is unlikely.

I decided to post this request after trying to build against ESP-IDF 5.5.3. The new BLE_STATIC_TO_DYNAMIC flag being on by default made ble_hs_hci_avail_pkts disappear. I know peaking at internal variables can be dangerous, but I otherwise have no clue how we are supposed to throttle notifications.

[rahult-github]

Hi @kenh-sq ,

Thanks for reporting this issue. Please try attached patch that should help to fix your case. Please let me know in case of any issues. Have addressed the typo of incorrect variable name and also introduced a new public API.

Your existing waterline approach will continue to work as-is:

ble_hs_lock();
uint16_t avail = ble_hs_hci_avail_pkts; /* works as before */
ble_hs_unlock();

Going forward, we recommend using the public API instead:

ble_hs_lock();
uint16_t avail = ble_hs_hci_get_avail_pkts();
ble_hs_unlock();

The patch should be applied in $IDF_PATH/components/bt/host/nimble/nimble path.

fix_hci_avail_pkt.txt

[kenh-sq]

Hi @rahult-github,

As you are going to expose this as a public API, maybe the lock/unlock should be within ble_hs_hci_get_avail_pkts()? ble_hs_lock() and ble_hs_unlock() really weren't meant to be exposed to the application, either.

I know this patch would allow me to continue to use my current throttling approach, but did you have any ideas on how else throttling might be done?

[rahult-github]

Hi @kenh-sq

maybe the lock/unlock should be within ble_hs_hci_get_avail_pkts()?

Thank you for review. Will also have it reviewed internally and tested before pushing to github. The patch was primarily given to first ensure your use case is satisfied.

did you have any ideas on how else throttling might be done?

Currently i have thought of below. Will try to come up with a best possible solution.

1. Completion callback / event: Introduce a new GAP event (e.g., BLE_GAP_EVENT_TX_COMPLETE) that fires when the controller's Number-of-Completed-Packets HCI event is received. This would tell the application "N buffers just freed up for connection handle X" — giving a direct signal to enqueue the next batch of notifications rather than polling avail_pkts. This , currently i believe, is the most aligned with what HCI already provides.

2. Rate-limited notify API: A variant of ble_gatts_notify_custom() that internally waits/blocks until a controller buffer is available before enqueuing rather than failing immediately when buffers are full. This simplifies the application logic but trades off control.

3. Per-connection flow control: Expose a per-connection-handle available buffer count (derived from the Number-of-Completed-Packets HCI event) so applications can make smarter decisions per link rather than using the global count.

[kenh-sq]

@rahult-github

Thanks for having this discussion with me.

An issue I know I have with my implementation is that it really only works okay when there is one connection. If I have an additional, much slower, connection that I am also sending notifications, and that is enqueued after a packet for a faster connection, I won't be able to send data to the faster connection until after the packet for the slower connection is also sent. So yes, a connection-relative event for changes to packet counts would be helpful.

Your third suggestion makes the most sense to me; an event which would be triggered whenever there is a change to a connection's bhc_outstanding_pkts. Knowing the outstanding packet count, as well as the global remaining allowed ACL packet count in the controller, would allow the application to do its own flow control per connection level while also allowing it to be mindful of the controller's overall congestion in its decisions.

A complication to adding application-based flow control is the host-side connection's bhc_tx_q. If/when mbufs are added to that queue, there is currently no indication to the application that its been done. I suppose the new event could be triggered with a BLE_HS_EAGAIN status (or something else?) when the number of ACL fragments required for a given L2CAP mbuf is more than the controller can handle at this time. The application could be allowed to decide to let the mbuf be added to the queue or not: the GAP event information could contain an om field that the application is allowed to free & NULL to imply the mbuf will not be added to the queue.

• The number of ACL packets that a given L2CAP mbuf will require should either be given as part of the GAP event information or inherent.
• This event would occur for all ACL transmissions which means there would be the need for the application or host task that initiated the transmission to receive failure status (BLE_HS_EBUSY ?) when the application has opposed the use of the queue so that the transmission might be tried again later. I have not looked at any of the host-task-initiated transmissions to see how they might be affected.
• It might be important to know just how many ACL fragments are held in the host-side queue and when something leaves that queue. I can imagine someone enqueuing multiple packets all at once for one connection and wanting to know when all of them have completed (an almost-no-flow-control bulk transfer?).

[rahult-github]

Hi @kenh-sq ,

Thanks for the detailed follow-up. You are correct that the current public visibility is limited for applications trying to implement tighter transmit pacing, especially when multiple connections share the same controller ACL buffer budget.

That said, I would like to clarify one point: the current implementation already keeps separate per-connection state such as
bhc_outstanding_pkts and per-connection host queues. So the limitation is not purely that one slower connection globally blocks all faster ones in the host scheduler; rather, the application does not currently have a supported way to observe enough of that state to make well-informed pacing decisions.

Your broader point still stands: a global available-packets count alone is not sufficient for robust multi-connection flow control. Better visibility into per-connection transmit progress could be useful. We agree this is worth discussing further.

For the rest of the proposal, especially application control over whether an mbuf is queued, status-code semantics, and whether such notifications should apply to all ACL traffic, we would need to evaluate the design carefully. Those aspects affect internal host behavior beyond GATT notifications and are not something we can commit to in their current form.

For now, the public ble_hs_hci_get_avail_pkts() API is intended as a supported replacement for directly accessing the internal global credit counter. It improves the current situation, but it is not meant to fully solve per-connection flow-control policy for every multi-connection use case.

[kenh-sq]

@rahult-github

One further clarification: I was not trying to say that the host implementation wasn't handling per-connection queues and that's why a transfer to a slow device would block other transfers. Rather, my current approach in using the global ACL buffer water line causes the slow device blocking. Example:

1. ACL buffer water line is 19 and is noted by "fast" transfer task.
2. Multi-buffer transfer to "fast" device delivered to controller.
3. ACL buffer water line is now 17 and is noted by "slow" transfer task.
4. Single buffer transfer to "slow" device delivered to controller.
5. ACL buffer water line is now 16.
6. "Fast" transfer completes.
7. ACL buffer water line is now 18.
8. Because the water line did not return to the previously noted 19, another transfer to the "fast" device won't be tried until the "slow" transfer completes.

I had a similar event occur earlier today; it wasn't a slow device, but a GATT Client moving out of range from a Server such that connection events weren't being seen. The particular data being transferred wasn't using my throttling approach, but the connections that were using the throttling became blocked as the missing-client connection stole all the mbufs until the connection timed out.

I find myself more in need of per-connection flow control/throttling and may take a stab at it. I agree it will be more finicky given it will affect all ACL transmissions, not just GATT.