Skip to content

Add serving-llms-on-epyc walkthrough - #83

Open
amd-lalithnc wants to merge 1 commit into
amd:mainfrom
amd-lalithnc:add-serving-llms-on-epyc-walkthrough
Open

Add serving-llms-on-epyc walkthrough#83
amd-lalithnc wants to merge 1 commit into
amd:mainfrom
amd-lalithnc:add-serving-llms-on-epyc-walkthrough

Conversation

@amd-lalithnc

Copy link
Copy Markdown
Contributor

Adds a walkthrough for serving-llms-on-epyc at walkthroughs/serving-llms-on-epyc.md, with a link in walkthroughs/README.md. Resolves #82

It follows the same sections as the existing walkthroughs: prerequisites, checking which skills are visible, installing the skill, running it, calling the endpoint, and a comparison of doing the task without the skill.

It also re-registers the skill in the marketplace (the entry was removed in #76) and folds in a few fixes found while testing the walkthrough on real EPYC hosts:

  • Require AVX-512 (Zen4+) — the skill now stops on pre-Zen4 EPYC.
  • Drop --shm-size from the container launch — it conflicts with --ipc=host on podman.
  • Recognize lettered EPYC SKUs in detect.py (e.g. 9B45 → Turin).

Tested end-to-end on two EPYC hosts: the endpoint comes up and answers.

Change-Id: Ia2bf2b8f40c2c709f8ad3b3d394a7946d4949b26

- Add walkthroughs/serving-llms-on-epyc.md (+ README link) for issue amd#82.
- Re-register the skill in the marketplace (needs a walkthrough to be listed).
- Require AVX-512 (Zen4+): hard gate in Step 1; scope in the description.
- Fix launch: drop --shm-size (conflicts with --ipc=host on podman).
- detect.py: recognize lettered EPYC SKUs (e.g. 9B45 -> Turin/Zen5).
- Note re-run name collision, rootless-podman cpuset, and HF_HOME cache mount.

Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com>
Change-Id: Ia2bf2b8f40c2c709f8ad3b3d394a7946d4949b26
@danielholanda

Copy link
Copy Markdown
Collaborator

@shailensobhee Can you please help review this here as well?

@danielholanda

Copy link
Copy Markdown
Collaborator

@shailensobhee Any updates here?

@shailensobhee

shailensobhee commented Jul 28, 2026

Copy link
Copy Markdown
Member

@amd-lalithnc Preparing to approve this. Some tasks still needs to be done:

  1. The code drifted a bit and needs to be adapted to resolve conflicts. Can you please check?
  2. I see that other skills have an eval. Can we have one too? See this: https://github.com/amd/skills/tree/main/skills/local-ai-use/evals
  3. Can yo ucomment if all "EPYC" works? Example, EPYC 4000 series have AVX-512, so technically would work, but we know that this is not ofcicially listed in the ZenDNN release notes. The skill must align accordingly.
  4. Also, please manually read the SKILL.md file. Some inconsistencies (which AI may overlook):
    or launch an LLM on AMD EPYC, Zen CPU - what exactly is a Zen CPU here (can lead to ambiguity). All our x86 CPUs have Zen architecture, and because of that part, the AI may think the skill would work OOB on Ryzen without issues.
    Also, that skills do not use on Instinct systems. I may argue this here; with semantic routers becoming popular, we may start seeing the channeling of small queries to small models, freeing a beefy model served on the GPU from handling simple tasks. On the same system, you could have multiple serving engines, on CPU and GPU. With this skill right now, you may completely strike out this scenario.
  5. Venice went public at Advancing AI Day. Have the skill cover this as well (even if you have "or newer"). Tops at Turin currently.
  6. in the walkthrough, first occurrence of AMD EPYC CPU, you may need the (TM) [legal]

## Step 1 - Understanding which skills are available

* Run `claude "Which skills can you see?" --model sonnet`. You should see a list of skills that does **not** include anything about serving LLMs on EPYC / CPU.
* Make sure there is no `AGENTS.md` file in your local folder.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you comment on why this line is important? What happens if you have an AGENTS.md file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add walkthrough for serving-llms-on-epyc

3 participants