[RFC] Deep-dive EuroSys 2026 LLM / Multimodal Serving Papers

[hsliuustc0106]

Summary

Request for interns to deep-dive the LLM / multimodal serving and inference papers from EuroSys 2026. Each deep dive should produce: a paper summary, key insights relevant to InferMatrix, and a 5-minute lightning-talk slide deck.

This is the EuroSys 2026 companion to #5 (OSDI 2026). Scope here is narrowed to serving & inference — training systems are out of scope for this round.

Context

EuroSys 2026 covers systems-level work across the stack. From the accepted list, ~22 papers are directly relevant to our inference serving work — LLM serving, scheduling, KV cache, MoE inference, multimodal serving, and inference on heterogeneous/edge hardware. Several come from industry groups (Alibaba, ByteDencent, Huawei, Tsinghua, SJTU) describing production-scale systems.

Papers index: https://2026.eurosys.org/papers.html

Papers to Deep Dive

LLM Serving & Scheduling

• AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
• FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters
• TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
• MFS: An Efficient Model Family Serving System for LLMs
• AdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving
• PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping
• KUNSERVE: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving
• PiLLM: Resource-Efficient LLM Inference Using Workload Prediction

KV Cache & Attention

• High Throughput and Low Latency LLM Serving via Adaptive KV Caching
• SAS: Sparse Attention Synthesizer for Efficient Language Model Inference
• LLMFolder: Revisiting Constant Folding in Large Language Models

MoE & Heterogeneous Inference

• Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
• SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference
• TailorLLM: Collaborative End-Cloud Inference of Large and Small Language Models Based on Low-Rank Adaptation
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones

Multimodal / Omni-Modal Serving

• Efficient Multimodal Serving via Module Multiplexing
• MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production (training system, included for the multimodal-at-scale infra story)

On-Device, Edge & Agent Serving

• TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone
• AIMS: Cost-Efficient LLM-Based Agent Deployment in Hybrid Cloud-Edge Environments

Supporting Inference Systems (broader stack)

• FlexiQ: Adaptive Mixed-Precision Quantization for Latency/Accuracy Trade-Offs in Deep Neural Networks
• FlashPS: Efficient Generative Image Editing with Mask-aware Caching and Scheduling
• Automated End-to-End Model Serving with Cooperative Compilation and Scheduling

Deliverables per Paper

1. One-page summary (markdown) covering:

   • Problem statement: what gap does this paper fill?
   • Key insight/technique: the one idea that makes it work
   • System design: architecture diagram or data flow
   • Evaluation highlights: workload, scale, key numbers
   • Relevance to InferMatrix: which part of our stack could benefit?

2. 5-minute lightning talk (slides) for team sync

3. Action items (optional): concrete experiments or integrations worth trying

Process

• Claim a paper by commenting on this issue (first-come, check the checkboxes)
• Target: 2 papers per intern per week
• Post deep dives as comments on this issue, then we will promote the best ones to docs/deep-dives/
• Cross-reference with [RFC] Deep-dive OSDI 2026 Cloud LLM Inference Papers #5: if a EuroSys paper overlaps thematically with an OSDI paper someone already summarized, note the comparison

Priority Order (suggested)

1. Efficient Multimodal Serving via Module Multiplexing — directly tackles multimodal/omni-model serving; most aligned with our omni work
2. AdaGen — workload-adaptive cluster scheduler for latency-optimal inference; scheduling is our core
3. Taming Latency-Memory Trade-Off in MoE-Based LLM Serving — MoE expert offloading; our MoE inference story
4. MFS — model family serving; relevant to multi-model routing/placement
5. TokenFlow — preemptive scheduling under request burst; our burst handling
6. Everything else: pick what interests you

References

• Papers list: https://2026.eurosys.org/papers.html
• Companion RFC: [RFC] Deep-dive OSDI 2026 Cloud LLM Inference Papers #5 (OSDI 2026 Cloud LLM Inference Papers)

[LJX-xixi]

AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

[potatoZhx]

KUNSERVE: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving

[wtz2333]

FlashPS: Efficient Generative Image Editing with Mask-aware Caching and Scheduling

[specture724]

AdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving

[ShwStone]

LLMFolder: Revisiting Constant Folding in Large Language Models

[jiaran-king]

FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters

[zwhzzz0821]

High Throughput and Low Latency LLM Serving via Adaptive KV Caching