Skip to content

Add new pretraining configurations and update paths for Qwen3 model#25

Open
T4ras123 wants to merge 1 commit into
mainfrom
config_updates-main-pr
Open

Add new pretraining configurations and update paths for Qwen3 model#25
T4ras123 wants to merge 1 commit into
mainfrom
config_updates-main-pr

Conversation

@T4ras123

Copy link
Copy Markdown
Contributor

This pull request introduces significant updates to the data configuration and pretraining setup for Qwen3-0.6B models in the molgen3D project. The changes expand support for new "revisited" datasets, add new model checkpoints and paths (including Weka storage), and introduce three new pretraining configuration files for different revisited dataset variants. The main pretraining config is also updated to use new datasets and tokenizers, and training hyperparameters are adjusted.

Key changes:

1. Data and Path Configuration Enhancements

  • Added new base paths for Weka storage and additional local directories to base_paths for improved dataset and checkpoint access. [1] [2]
  • Expanded the data section to include multiple new "revisited" datasets (cartesian, quantile binned, uniform binned, and their isomeric/grouped variants) for train, valid, and test splits. [1] [2]
  • Registered new tokenizer (qwen3_0.6b_binned_258) and associated path.
  • Added new model entries for pretraining on the revisited datasets, specifying checkpoint steps and root paths.

2. Pretraining Configuration Updates

  • Updated the main Qwen3-0.6B pretraining config (qwen3_06b.toml) to use the new revisited uniform binned dataset and the new tokenizer (qwen3_0.6b_binned_258). Adjusted training steps, checkpoint intervals, and scheduler milestones to match the new dataset. [1] [2] [3] [4] [5]

3. Addition of New Pretraining Job Configurations

  • Introduced three new TOML config files for pretraining on:
    • revisited_cartesian_isomeric
    • revisited_quantile_binned_isomeric
    • revisited_uniform_binned_isomeric
  • Each config specifies dataset, tokenizer, model, optimizer, scheduler, and checkpointing tailored for its respective dataset variant.

These changes collectively enable training and evaluation on a broader set of revisited molecular datasets, improve data accessibility, and provide ready-to-use configurations for different data preprocessing strategies.
[Copilot is generating a summary...]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant