High CPU Saturation at 2Gbps/700 Viewers

[Kavi-dc]

Hi Team,

We are experiencing abnormal CPU saturation during peak hours. Despite having hardware that exceeds MistServer’s recommended specifications, the system CPU becomes saturated at approximately 700 (HLS/HTTPS) concurrent viewers (total throughput ~2 Gbps).

Given the EPYC 4244P's high single-thread performance and AVX-512 support, we expected significantly higher density. We are seeking guidance on whether this is a configuration bottleneck (SSL/Socket) or a known limitation in the Mistserver.

Comparison Data

On the exact same hardware and SSL certificate:
Nimble Streamer: Handles 4 Gbps at only 30% CPU load.
MistServer (v3.10): Reaches 100% CPU load at 2 Gbps.

Server spec

CPU: AMD EPYC 4244P 6-Core Processor, 12 cores (AES,AVX/AV512) supported
RAM: 2x 32GB DDR5 5600 Mhz
Root Drive: 500GB NVME
NIC: 10Gig card
ISP: 3Gbps uplink
Passmark score:
Single thread: 3761
Multithread: 27034
OS: Ubuntu 22.04
Optimizations: BBR, high somaxconn, adjusted txqueuelen.

Server CPU load Graph

Server Outgoing traffic Graph

Viewer count

[Thulinma]

Hey there!

TS-based HLS is by far the least-efficient output MistServer has - mostly due to the nature of how the TS segments need to be generated.
Switching to MP4-based HLS (or pretty much any other protocol, for that matter) will improve performance significantly. That said, we're also working on a new caching mechanism that should help speed up HLS-based outputs by storing the generated segments in cache (this is not quite ready for production use yet, though).

Another performance bottleneck you may be encountering is MistServer's HTTPS process: our standard binaries are generic builds that have no CPU-specific optimizations, which means that especially the TLS implementation in there tends to be on the slow side. Compiling your own binaries (thus allowing CPU-specific optimizations to be enabled) or using more dedicated TLS proxy software (like nginx, haproxy, etc) can in some cases result in almost a 2X speed-up compared to the built-in TLS implementation.

Hope that helps!

[Kavi-dc]

Hi Thulinma,

Thanks for the information. I’ll try the recommended workarounds. Thanks again for your help!