[WIP] Initial DeepSeek reference implementation by denys-fridman · Pull Request #861 · mlcommons/training

denys-fridman · 2026-01-14T11:21:16Z

No description provided.

github-actions · 2026-01-14T11:21:27Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

suexu1025 · 2026-02-06T00:36:17Z

+export GBS=1024
+# Dataloader: Micro batch size
+export MBS=1
+export MAX_LR="2e-4"


update the final tuned HPs?

suexu1025 · 2026-02-06T00:40:08Z

+    grad_accumulation_steps = mini_batch_size // args.mbs
+
+    logging_configs = {
+        mllogger.constants.SEED: args.seed,


we need more mllog event like
[ self.mllogger.event(
key=constants.SUBMISSION_BENCHMARK,
value=self.submission_info["submission_benchmark"],
)], detailed list:(https://github.com/mlcommons/training/blob/master/llama2_70b_lora/scripts/mlperf_logging_utils.py)

Added missing (I think all). SUBMISSION_BENCHMARK is logged above with mllogger.mlperf_submission_log(bmark)

pbaumstarck · 2026-02-20T18:25:43Z

@@ -0,0 +1,16 @@
+git+https://github.com/denys-fridman/logging.git@dfridman/deepseek-v3  # TODO(dfridman): revert to main repo once merged


I think we need this reverted before merging. There is another TODO in the PR.

Right. I'll update after mlcommons/logging#445 is merged

pbaumstarck · 2026-02-20T18:26:33Z

+    pip install -e .
+
+## 2. Megatron-bridge and megatron-core
+ARG MBRIDGE_REVISION=main


Can we pin this like NEMORUN_REVISION?

…onfig learning rates

ShriyaRishab · 2026-02-26T16:35:48Z

+
+#### Run model conversion
+
+Assuming that we have downloaded the HuggingFace checkpoint to a `<SRC_PATH>` directory, the checkpoint must be converted to Megatron-Bridge format before training. After conversion is done, set `MODEL_CKPT=<DST_PATH>` when launching the job.


Add a section on typical runtime expected and share reference hardware used along with # of nodes

…m-up procedure

…0-iteration warm-up

…aph args

…tion

…t download

…version (TODO: script)

…alized

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ual override

ShriyaRishab · 2026-02-27T15:40:23Z

@@ -0,0 +1,606 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION. All rights reserved.


Move everything to llm_moe_pretraining folder

Updated instructions for using the repository and downloading checkpoints.

Updated README to clarify GBS requirements and evaluation process.

ShriyaRishab

LGTM

denys-fridman requested a review from a team as a code owner January 14, 2026 11:21

[WIP] Initial DeepSeek reference implementation

18410e3

denys-fridman force-pushed the dfridman/deepseek-reference-implementation branch from 7e03fe1 to 18410e3 Compare January 14, 2026 14:16

denys-fridman added 5 commits January 20, 2026 11:14

Add NemoRun launcher for DeepSeek V3 pretraining

412ae53

Update run_deepseek_v3_671b.sh to use NemoRun launcher

3ac907c

cd to current dir

b7bb8cd

detach = True by default

ff7e160

Fix env vars

38e318c

denys-fridman force-pushed the dfridman/deepseek-reference-implementation branch from 44a6723 to 38e318c Compare January 21, 2026 09:50

denys-fridman added 10 commits January 21, 2026 11:22

Remove unused arguments and fix eval config bug

200f496

add missing pretrain args

b0423ff

fix pipeline layout + tokenizer config

c59fd89

fix import

48fd489

remove force_success_status var

55ac23c

fix args

0932bad

fix args

99df101

use 8b data paths

6a45841

use TMP_NPY_INDEX

f166413

update config

ea2b48f

suexu1025 reviewed Feb 6, 2026

View reviewed changes

pbaumstarck reviewed Feb 20, 2026

View reviewed changes

denys-fridman added 8 commits February 25, 2026 13:47

set adam epsilon to 1e-8

544f076

set MBRIDGE_REVISION to 38858ef0ae3b835af236ec1cf8bf1feca4d400fa

18c50a8

update TODO comment to reference logging PR mlcommons#445

6580086

add missing mllogger submission logs (CACHE_CLEAR, POC_NAME, POC_EMAIL)

163128e

add config files for GBS 256/288/320

1caf442

set max_steps=12000, warmup_steps=4, eval_check_interval=1, and per-c…

f116f4a

…onfig learning rates

remove quotes around MAX_LR

ae8ff28

cherry-pick eval batch size knob from dfridman/Megatron-Bridge

829f618

denys-fridman added 2 commits February 26, 2026 17:34

Update README: use Llama 3.1 8B tokenizer and dataset download links

9f2edb6

Update README: set quality target validation loss to 3.60

f0ca37d

ShriyaRishab reviewed Feb 26, 2026

View reviewed changes

denys-fridman added 14 commits February 26, 2026 17:37

Set target validation loss to 3.60

4ea28dc

Update README: mention Mixture of Experts in problem statement

952b243

Update README: describe checkpoint origin and 50-iteration router war…

b54be3c

…m-up procedure

Update README: clarify checkpoint is distributed in HF format after 5…

85349c6

…0-iteration warm-up

Hardcode moe_aux_loss_coeff to 0.01

bd71e68

Rename 320 config to 240, update GBS to 15360 and LR to 0.000023238

cff96ec

Disable CUDA graph: hardcode implementation to "none", remove cuda_gr…

a68c6c9

…aph args

Update README: specify 1024 sequences (1024 * 4096 tokens) for valida…

b94ede7

…tion

Update README: remove confidentiality agreement mention for checkpoin…

e0cc30b

…t download

Update README: note checkpoint is HF format requiring Megatron-LM con…

c7f5514

…version (TODO: script)

Add HF-to-Megatron-LM checkpoint conversion scripts

a51fade

Add reference hardware and expected runtime section to README

8eac4de

Fix get_rank() to use RANK/SLURM_PROCID env vars before dist is initi…

4c3eeb9

…alized

Use recompute_modules mlp,moe_act in all configs

9596eba

ShriyaRishab reviewed Feb 26, 2026

View reviewed changes

Comment thread moe_pretraining/nemo/README.md Outdated

Fix LR: use MAX_LR to match config variable name

011b9f0

ShriyaRishab reviewed Feb 26, 2026

View reviewed changes

Comment thread moe_pretraining/nemo/README.md Outdated

denys-fridman and others added 4 commits February 26, 2026 22:17

Add MIN_LR=1e-8 to all configs

8764c08

[DSv3] Add SEGMENT export to configs and --segment arg to launcher

79f635a

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

[DSv3] Remove SEGMENT export from configs, keep --segment arg for man…

0963bac

…ual override

[DSv3] Disable checkpoint saving during MLPerf training

bb02912

ShriyaRishab reviewed Feb 27, 2026

View reviewed changes

ShriyaRishab added 2 commits February 27, 2026 11:04

Address comments

49621da

Updated instructions for using the repository and downloading checkpoints.

Clarify GBS requirements and evaluation details in README

3dcdf20

Updated README to clarify GBS requirements and evaluation process.

ShriyaRishab approved these changes Feb 27, 2026

View reviewed changes

ShriyaRishab merged commit f0e0607 into mlcommons:master Feb 27, 2026
1 check passed

github-actions Bot locked and limited conversation to collaborators Feb 27, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Initial DeepSeek reference implementation#861

[WIP] Initial DeepSeek reference implementation#861
ShriyaRishab merged 53 commits intomlcommons:masterfrom
denys-fridman:dfridman/deepseek-reference-implementation

denys-fridman commented Jan 14, 2026

Uh oh!

github-actions Bot commented Jan 14, 2026 •

edited

Loading

Uh oh!

suexu1025 Feb 6, 2026

Uh oh!

suexu1025 Feb 6, 2026

Uh oh!

denys-fridman Feb 25, 2026

Uh oh!

pbaumstarck Feb 20, 2026

Uh oh!

denys-fridman Feb 25, 2026

Uh oh!

pbaumstarck Feb 20, 2026

Uh oh!

denys-fridman Feb 25, 2026

Uh oh!

ShriyaRishab Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

ShriyaRishab Feb 27, 2026

Uh oh!

ShriyaRishab left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1,16 @@
		git+https://github.com/denys-fridman/logging.git@dfridman/deepseek-v3 # TODO(dfridman): revert to main repo once merged


		#### Run model conversion

		Assuming that we have downloaded the HuggingFace checkpoint to a `<SRC_PATH>` directory, the checkpoint must be converted to Megatron-Bridge format before training. After conversion is done, set `MODEL_CKPT=<DST_PATH>` when launching the job.

		@@ -0,0 +1,606 @@
		# Copyright (c) 2024-2025, NVIDIA CORPORATION. All rights reserved.

Conversation

denys-fridman commented Jan 14, 2026

Uh oh!

github-actions Bot commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShriyaRishab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Jan 14, 2026 •

edited

Loading