im-server

A robust, single-node Instant Messaging backend written in modern C++17.

---

Core Architecture

• Reactor: Non-blocking I/O (Epoll) with thread pool
• Protocol: Protobuf + length-prefixed framing over TCP
• Storage: MySQL (durable), Redis (online state), Kafka (write-ahead log)
• Design: Stateless except SessionManager; soft-fail on backend outages

---

Architecture at a glance

Data flow: MySQL (SOR) ← Kafka ← live delivery (Redis). Online users in Redis; offline messages queued; Kafka consumed asynchronously by im-consumer for durability.

---

Module map

Dir	What's in it
src/common/	Logger + Config (JSON, nlohmann). Singletons, no deps.
src/net/	Reactor core: EventLoop, Channel, Epoller, Buffer, Socket, Acceptor, TcpConnection, TcpServer, EventLoopThreadPool, TimerQueue.
src/session/	Session (per-user auth state + idle timestamp) and SessionManager (process-wide uid/fd map).
src/codec/	Wire framing (Codec) and MessageDispatcher (msgType → handler).
src/proto/	Generated im_message.pb.{cc,h} + source .proto.
src/auth/	LoginReq / LogoutReq / Heartbeat handlers.
src/chat/	SingleChat / GroupChat / MsgAck / MsgRead handlers + Kafka publish.
src/offline/	PullOfflineReq handler + auto-flush on login.
src/db/	MysqlPool (libmysqlclient, bounded connection pool).
src/cache/	RedisClient (hiredis, thread-safe command interface).
src/mq/	KafkaProducer / KafkaConsumer (librdkafka C++ binding).
src/metrics/	Atomic counters + HttpMetricsServer for Prometheus scrapes.
tests/	GoogleTest suite (fetched via CMake FetchContent at configure time).
scripts/	smoke_chat.py end-to-end client, bench_chat.py load tester, measure_rss.sh RSS monitor + generated im_message_pb2.py.
sql/	schema.sql with users, groups, messages, reads; seed_bench.sql for 5 000 bench users.
docs/	Architecture guide, benchmarks, Valgrind + Perf reports.

---

Wire protocol

Frame format: [totalLen:4][msgType:2][seqId:4][protobuf payload]

• Max payload: 16 MiB; oversized frames close connection
• msgType → im::MsgType enum; seqId for response correlation

msgType	Direction	Payload
1 LOGIN_REQ	C→S	LoginReq
2 LOGIN_RSP	S→C	LoginRsp
3 LOGOUT_REQ	C→S	LogoutReq
4 HEARTBEAT	C→S	Heartbeat
10 SINGLE_CHAT	C⇄S	SingleChatMsg
11 GROUP_CHAT	C⇄S	GroupChatMsg
20 MSG_ACK	S→C	MsgAck
21 MSG_READ	C→S	MsgRead
30 PULL_OFFLINE_REQ	C→S	PullOfflineReq
31 PULL_OFFLINE_RSP	S→C	PullOfflineRsp

---

Prerequisites

Ubuntu 22.04+: g++ >= 11, cmake >= 3.20, libprotobuf-dev, libmysqlclient-dev, libhiredis-dev, librdkafka-dev (optional)

---

Build from source

cd IM
cmake -S . -B build -G Ninja -DBUILD_TESTING=ON
cmake --build build -j$(nproc)

Outputs:

• build/im-server — main chat reactor
• build/im-consumer — Kafka → MySQL persistence worker
• build/test_* — unit-test binaries (only with -DBUILD_TESTING=ON)

Run tests:

ctest --test-dir build --output-on-failure

---

Quick start (Docker)

docker compose up --build
cd scripts && python3 smoke_chat.py 127.0.0.1 18080  # test protocol
curl http://localhost:19090/metrics                  # Prometheus
docker compose down -v                                # cleanup

Quick start (native)

1. Start MySQL 3306, Redis 6379, Kafka 9092 locally
2. mysql -h 127.0.0.1 < sql/schema.sql
3. Edit config.json with local endpoints
4. ./build/im-server config.json & ./build/im-consumer config.json
5. python3 scripts/smoke_chat.py 127.0.0.1 18080

---

Configuration reference

Key	Type	Default	Meaning
server.port	uint16	required	Listen port for chat connections.
server.thread_num	int	required	Number of sub-reactor threads.
server.metrics_port	uint16	0 (off)	Prometheus scrape port on 127.0.0.1. Set to enable.
mysql.host	string	required	MySQL host.
mysql.port	uint16	required	MySQL port.
mysql.user / mysql.password	string	required	Credentials.
mysql.database	string	required	Database name (matches sql/schema.sql).
mysql.pool_size	int	required	Bounded pool size (blocks when exhausted).
redis.host / redis.port	string / int	required	Redis endpoint.
kafka.brokers	string	required	Kafka bootstrap servers (comma-separated).
kafka.topic	string	required	Topic for ChatEvent records.
log.file	string	required	Log file path (appended).
log.level	string	DEBUG	One of DEBUG / INFO / WARN / ERROR.

---

Metrics

GET http://127.0.0.1:<metrics_port>/metrics returns Prometheus format.

Counters: im_connections_total, im_messages_sent_*_total, im_read_receipts_total, im_offline_queued_total, im_kafka_published_total, im_backpressure_disconnects_total, im_retry_*_total

Gauges: im_connections_current, im_sessions_current

---

Testing

• Unit tests: test_buffer, test_codec, test_dispatcher, test_session_manager (GoogleTest)
• Integration: smoke_chat.py end-to-end protocol validation
• CI: GitHub Actions, runs unit suite on every push

---

Design notes

• Singleton Handlers: Hot-pluggable but not multi-instanceable per-process
• shared_ptr lifecycle: TcpConnection::enable_shared_from_this extends lifetime across async work
• Backpressure: Hard 4 MiB limit; exceeding it force-closes connection
• Kafka fire-and-forget: Chat RTT independent of Kafka; async durability via im-consumer
• In-process sessions: SessionManager requires externalization (Redis/registry) for scaling

---

Performance

Measured (i5-13600KF, Ubuntu 24.04, Redis only):

• Connections: 4 094 (WSL2 cap; fd-limit ~10k)
• Send→ack latency: 1.33 ms avg, 5.10 ms p99
• Memory: 42 MB peak
• Reliability: 0 decode errors, 100% ack-match

Optimizations: Buffer reuse · readv() scatter I/O · Redis mutex serialization · Fire-and-forget Kafka

---

Roadmap

Done: Reactor core · Protobuf · Auth/chat/offline/reads · MySQL/Redis/Kafka · Prometheus metrics · Unit tests · Docker Compose · Retry/backpressure · Load harness · Perf profiling

Todo: TLS · Horizontal scaling · JSON logs/tracing · Message edit/pin · Typing/delivered indicators · Admin APIs · Friend/blocklist · Kafka DLQ · Clients (CLI/mobile/web)

---

License

MIT