From 2db869e12ee82c8360de9bb2c958999f680938f5 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Mon, 9 Feb 2026 21:20:12 -0800 Subject: [PATCH 01/30] Rename and restructure Supported Models page - Rename supported-models.qmd to supported-model-frameworks.qmd - Add alias for old URL to preserve links - Restructure content into clearer sections: - Framework support with wrapper classes and install commands - Test input requirements (predict vs predict_proba) - Custom model wrappers (FunctionModel, PipelineModel) - GenAI and LLM support - RAG evaluation with RAGAS - Python and dependency compatibility - Add Mermaid diagrams for class hierarchy and test input flow - Update references in faq-integrations.qmd and overview-model-documentation.qmd --- site/about/overview-model-documentation.qmd | 2 +- site/developer/supported-model-frameworks.qmd | 269 ++++++++++++++++++ site/developer/supported-models.qmd | 158 ---------- site/faq/faq-integrations.qmd | 4 +- 4 files changed, 272 insertions(+), 161 deletions(-) create mode 100644 site/developer/supported-model-frameworks.qmd delete mode 100644 site/developer/supported-models.qmd diff --git a/site/about/overview-model-documentation.qmd b/site/about/overview-model-documentation.qmd index 17c56cb014..443492984f 100644 --- a/site/about/overview-model-documentation.qmd +++ b/site/about/overview-model-documentation.qmd @@ -146,7 +146,7 @@ How the {{< var validmind.developer >}} works: [^1]: [Model risk management](overview-model-risk-management.qmd) -[^2]: [Supported models](/developer/supported-models.qmd) +[^2]: [Supported model frameworks](/developer/supported-model-frameworks.qmd) [^3]: [Customize document templates](/guide/templates/customize-document-templates.qmd) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd new file mode 100644 index 0000000000..801c6df8f8 --- /dev/null +++ b/site/developer/supported-model-frameworks.qmd @@ -0,0 +1,269 @@ +--- +# Copyright © 2023-2026 ValidMind Inc. All rights reserved. +# Refer to the LICENSE file in the root of this repository for details. +# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial +title: "Supported model frameworks" +date: last-modified +aliases: + - /guide/supported-models.html + - /developer/model-documentation/supported-models.html + - /developer/supported-models.html +listing: + - id: next-models + type: grid + max-description-length: 250 + sort: false + fields: [title, description] + contents: + - ../model-testing/testing-overview.qmd + - ../model-testing/test-descriptions.qmd + - ../samples-jupyter-notebooks.qmd +--- + +The {{< var validmind.developer >}} supports a wide range of model frameworks for testing and documentation. This page explains which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. + +::: {.callout} +## The library is framework-agnostic + +Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. +::: + +## Framework support + +The {{< var validmind.developer >}} provides wrapper classes for common ML frameworks. Choose the wrapper that matches your model's framework: + +| Framework | Wrapper Class | Installation | +|-----------|---------------|--------------| +| scikit-learn, XGBoost, CatBoost, LightGBM, StatsModels | `SKlearnModel` | `pip install validmind` | +| PyTorch | `PyTorchModel` | `pip install validmind[pytorch]` | +| Hugging Face Transformers | `HFModel` | `pip install validmind` | +| LLM endpoints (OpenAI, Azure, etc.) | `FoundationModel` | `pip install validmind[llm]` | +| Any Python callable | `FunctionModel` | `pip install validmind` | +| R models (via rpy2) | `RModel` | `pip install validmind rpy2` | + +To install all optional dependencies: + +```bash +pip install validmind[all] +``` + +### Class hierarchy + +The following diagram shows how model wrapper classes relate to each other: + +```{mermaid} +classDiagram + VMModel <|-- SKlearnModel + VMModel <|-- PyTorchModel + VMModel <|-- HFModel + VMModel <|-- FunctionModel + VMModel <|-- RModel + SKlearnModel <|-- XGBoostModel + SKlearnModel <|-- CatBoostModel + SKlearnModel <|-- StatsModelsModel + FunctionModel <|-- FoundationModel + FunctionModel <|-- PipelineModel +``` + +## Test input requirements + +Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case. + +### When `predict()` is required + +Most model tests call your model's `predict()` method to generate predictions. This includes: + +- Performance metrics (accuracy, precision, recall, F1) +- Error analysis tests +- Robustness tests + +### When `predict_proba()` is needed + +Classification metrics that evaluate probability outputs require `predict_proba()`: + +- ROC-AUC score +- Precision-recall curves +- Calibration tests +- Probability distribution analysis + +If your model doesn't have `predict_proba()`, these tests will be skipped or return an error. + +### Test input flow + +```{mermaid} +flowchart LR + subgraph Inputs + Model[Model Object] + Dataset[Dataset] + Precomputed[Precomputed Predictions] + end + + Model --> predict["predict()"] + Model --> predict_proba["predict_proba()"] + Precomputed --> Dataset + + subgraph TestTypes[Test Types] + ModelTests[Model Tests] + DatasetTests[Dataset Tests] + ClassificationTests[Classification Metrics] + end + + predict --> ModelTests + predict --> DatasetTests + predict_proba --> ClassificationTests + Dataset --> DatasetTests +``` + +### Using precomputed predictions + +If you can't provide a model object (for example, if your model runs in a separate environment), you can pass precomputed predictions directly to the dataset: + +```python +vm_dataset = vm.init_dataset( + dataset=df, + target_column="target", + prediction_values=predictions, # numpy array of predictions + probability_values=probabilities # optional: for classification +) +``` + +Alternatively, if predictions are already in your dataframe: + +```python +vm_dataset = vm.init_dataset( + dataset=df, + target_column="target", + prediction_column="predicted", # column name in df + probability_column="probability" # optional: for classification +) +``` + +### Dataset-only tests + +Some tests analyze data quality and don't require a model at all: + +- Missing value analysis +- Class imbalance detection +- Feature correlation +- Outlier detection +- Data drift tests + +## Custom model wrappers + +For models that don't fit standard framework patterns, use these flexible wrappers. + +### FunctionModel + +Wrap any Python callable as a model: + +```python +from validmind.models import FunctionModel + +def my_predict(X): + # Your prediction logic here + return predictions + +vm_model = vm.init_model( + model=FunctionModel(predict_fn=my_predict), + input_id="my_model" +) +``` + +### PipelineModel + +Chain multiple models or processing steps: + +```python +from validmind.models import PipelineModel + +vm_model = vm.init_model( + model=PipelineModel(models=[preprocessor, model]), + input_id="my_pipeline" +) +``` + +## GenAI and LLM support + +The {{< var validmind.developer >}} provides specialized support for large language models and generative AI. + +### FoundationModel + +Use `FoundationModel` for LLM endpoints: + +```python +from validmind.models import FoundationModel +from validmind.prompt import Prompt + +prompt = Prompt( + template="Classify the sentiment: {text}", + variables=["text"] +) + +vm_model = vm.init_model( + model=FoundationModel( + prompt=prompt, + model="gpt-4", # or your model endpoint + ), + input_id="sentiment_classifier" +) +``` + +### LLM test suites + +Install LLM dependencies: + +```bash +pip install validmind[llm] +``` + +Available test suites for LLMs include: + +- Prompt injection detection +- Output consistency +- Hallucination detection +- Toxicity analysis +- Bias evaluation + +## RAG evaluation + +For retrieval-augmented generation (RAG) systems, the {{< var validmind.developer >}} integrates with [RAGAS](https://docs.ragas.io/) for comprehensive evaluation. + +### Dataset requirements + +RAG evaluation requires specific fields in your dataset: + +| Field | Type | Description | +|-------|------|-------------| +| `user_input` | str | The user's query | +| `response` | str | Model output | +| `retrieved_contexts` | List[str] | Retrieved context chunks | +| `reference` | str | Ground truth (required for some tests) | + +### Available RAG tests + +- **Faithfulness** — Measures how well the response is grounded in retrieved contexts +- **Context Recall** — Evaluates if relevant information was retrieved +- **Context Precision** — Measures relevance of retrieved contexts +- **Answer Relevancy** — Assesses if the response addresses the query + +## Python and dependency compatibility + +The {{< var validmind.developer >}} requires: + +- **Python:** >=3.9, <3.13 +- **Core dependencies:** pandas, numpy, scikit-learn + +Optional dependencies for specific frameworks: + +| Extra | Frameworks | +|-------|------------| +| `pytorch` | PyTorch, torchvision | +| `llm` | OpenAI, langchain, ragas | +| `xgboost` | XGBoost | +| `catboost` | CatBoost | +| `all` | All optional dependencies | + +## What's next + +:::{#next-models} +::: diff --git a/site/developer/supported-models.qmd b/site/developer/supported-models.qmd deleted file mode 100644 index 76e961af43..0000000000 --- a/site/developer/supported-models.qmd +++ /dev/null @@ -1,158 +0,0 @@ ---- -# Copyright © 2023-2026 ValidMind Inc. All rights reserved. -# Refer to the LICENSE file in the root of this repository for details. -# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial -title: "Supported models" -date: last-modified -aliases: - - /guide/supported-models.html - - /developer/model-documentation/supported-models.html -listing: - - id: next-models - type: grid - max-description-length: 250 - sort: false - fields: [title, description] - contents: - - ../model-testing/testing-overview.qmd - - ../model-testing/test-descriptions.qmd - - ../samples-jupyter-notebooks.qmd ---- - -The {{< var validmind.developer >}} provides out-of-the-box support for testing and documentation for an array of model types and modeling packages. - -## What is a supported model? - -A _supported model_ refers to a model for which predefined testing or documentation functions exist in the {{< var validmind.developer >}}, provided that the model you are developing is documented using a supported version of our {{< var vm.developer >}}. These model types cover a very large portion of the models used in commercial and retail banking. - -::: {.callout} -## {{< var vm.product >}} does not limit our users to specific model types. - -- The {{< var validmind.developer >}} is extensible to support future model types or modeling packages to accomodate rapid developments in the AI space, including the advent of large language models (LLMs). -- You always have the flexibility to implement custom tests and integrate external test providers.[^1] -::: - - - -## Supported model types - -::: {.column-margin} -::: {.feature} -Vendor models -: {{< var vm.product >}} offers support for both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). - -::: - -::: - -### Traditional statistical models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-30-ns} - -#### Linear regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} - -#### Logistic regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} -#### Time series -Analyzes data points collected or sequenced over time. - -::: - -:::: - -### Machine learning models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns .pr2} -#### Hugging Face-compatible models -- Natural language processing (NLP) text classification — Categorizes text into predefined classes. -- Tabular classification — Assigns categories to tabular dataset entries. -- Tabular regression — Predicts continuous outcomes from tabular data. - -#### Neural networks -- Long short-term memory (LSTM) — Processes sequences of data, remembering inputs over long periods. -- Recurrent neural network (RNN) — Processes sequences by maintaining a state that reflects the history of processed elements. -- Convolutional neural network (CNN) — Primarily used for processing grid-like data such as images. - - -::: - -::: {.w-50-ns .pl2 .pr2} - -#### Tree-based models
(XGBoost / CatBoost / random forest) -- Classification — Predicts categorical outcomes using decision trees. -- Regression — Predicts continuous outcomes using decision trees. - -#### K-nearest neighbors (KNN) -- Classification — Assigns class by majority vote of the k-nearest neighbors. -- Regression — Predicts value based on the average of the k-nearest neighbors. - -#### Clustering -- K-means — Partitions _n_ observations into _k_ clusters in which each observation belongs to the cluster with the nearest mean. - - -::: - -:::: - -### Generative AI models - -#### Large language models (LLMs) -- Classification — Categorizes input into predefined classes. -- Text summarization — Generates concise summaries from longer texts. - -## Supported modeling libraries and other tools - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns} - -- **[scikit-learn](https://scikit-learn.org/stable/)** — A Python library for machine learning, offering a range of supervised and unsupervised learning algorithms. - -- **[statsmodels](https://www.statsmodels.org/stable/index.html)** — A Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests and exploring data. - -- **[PyTorch](https://pytorch.org/)** — An open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing. - -- **[Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index)** — Provides thousands of pre-trained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, and text generation. - -- **[XGBoost](https://xgboost.readthedocs.io/en/stable/)** — An optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable, implementing machine learning algorithms under the Gradient Boosting framework. - -::: - -::: {.w-50-ns} - -- **[CatBoost](https://catboost.ai/)** — An open-source gradient boosting on decision trees library with categorical feature support out of the box, for ranking, classification, regression, and other ML tasks. - -- **[LightGBM](https://lightgbm.readthedocs.io/en/stable/)** — A fast, distributed, high-performance gradient boosting (GBDT, GBRT, GBM, or MART) framework based on decision tree algorithms, used for ranking, classification, and many other machine learning tasks. - -- **R models, via [rpy2 - R in Python](https://rpy2.github.io/)** — Facilitates the integration of R's statistical computing and graphics capabilities with Python, allowing for R models to be called from Python. - -- **Large language models (LLMs), via [OpenAI-compatible APIs](https://platform.openai.com/docs/introduction)** — Access to advanced AI models trained by OpenAI for a variety of natural language tasks, including text generation, translation, and analysis, through a compatible API interface. This support includes both the OpenAI API and the Azure OpenAI Service via API. -::: - -:::: - -## What's next - -:::{#next-models} -::: - - - - -[^1]: - - - [Implement custom tests](/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb) - - [Integrate external test providers](/notebooks/code_samples/custom_tests/integrate_external_test_providers.ipynb) \ No newline at end of file diff --git a/site/faq/faq-integrations.qmd b/site/faq/faq-integrations.qmd index b7c3b0c3bf..9f0b6b8168 100644 --- a/site/faq/faq-integrations.qmd +++ b/site/faq/faq-integrations.qmd @@ -14,7 +14,7 @@ listing: sort: false fields: [title, description] contents: - - ../developer/supported-models.qmd + - ../developer/supported-model-frameworks.qmd - ../about/overview-llm-features.qmd - ../about/deployment/deployment-options.qmd categories: ["supported libraries", "supported languages", "integrations", "images", "large language models", "explainability", "deployment options", "validmind library"] @@ -82,7 +82,7 @@ We will be implementing connector interfaces allowing extraction of relationship [^1]: [{{< var validmind.developer >}}](/developer/validmind-library.qmd) -[^2]: [Supported modeling libraries and other tools](/developer/supported-models.qmd#supported-modeling-libraries-and-other-tools) +[^2]: [Supported model frameworks](/developer/supported-model-frameworks.qmd) [^3]: [Matplotlib](https://matplotlib.org/) From 83e00838083ed7f7ddad98850165fcbd29567907 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Tue, 10 Feb 2026 12:38:28 -0800 Subject: [PATCH 02/30] Fix missed sidebar update --- site/developer/_sidebar.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/developer/_sidebar.yaml b/site/developer/_sidebar.yaml index 3b1680ea69..0569551bfe 100644 --- a/site/developer/_sidebar.yaml +++ b/site/developer/_sidebar.yaml @@ -10,7 +10,7 @@ website: # USING THE VARIABLE IN THE LINK TEXT MESSES UP THE MOBILE VIEW - text: "ValidMind Library" file: developer/validmind-library.qmd - - developer/supported-models.qmd + - developer/supported-model-frameworks.qmd - text: "---" - text: "Quickstart" - notebooks/quickstart/quickstart_model_documentation.ipynb From bba9d36cc46c33c0a88885eed54cf81440024bea Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Wed, 4 Mar 2026 17:11:28 -0800 Subject: [PATCH 03/30] Fix broken link --- site/releases/2024/2024-mar-27/highlights.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/releases/2024/2024-mar-27/highlights.qmd b/site/releases/2024/2024-mar-27/highlights.qmd index 5d499af572..4c4d929f7c 100644 --- a/site/releases/2024/2024-mar-27/highlights.qmd +++ b/site/releases/2024/2024-mar-27/highlights.qmd @@ -458,7 +458,7 @@ We improved our supported models documentation with additional information about ::: ::: {.w-30-ns .tc} -[Supported models](/developer/supported-models.qmd){.button} +[Supported model frameworks](/developer/supported-model-frameworks.qmd){.button} ::: From 11a49cb37c6d2bc22341395a03ac843fb82d5da9 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Wed, 4 Mar 2026 17:53:30 -0800 Subject: [PATCH 04/30] Add bridging sentence after callout --- site/developer/supported-model-frameworks.qmd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 63ac30c411..182ce124c7 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -30,6 +30,8 @@ The {{< var validmind.developer >}} supports a wide range of model frameworks fo Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. ::: +"Supported" here means the library provides dedicated wrappers, install options, and guidance for these frameworks; you can still use other code, such as any callable or precomputed predictions, without them. + ## Framework support The {{< var validmind.developer >}} provides wrapper classes for common ML frameworks. Choose the wrapper that matches your model's framework: From 2e343dfcdf707754774d79783be8cb14c571e1be Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 09:20:24 -0800 Subject: [PATCH 05/30] Re-add a what does supported mean heading --- site/developer/supported-model-frameworks.qmd | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 182ce124c7..2b070627c5 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -24,17 +24,19 @@ listing: The {{< var validmind.developer >}} supports a wide range of model frameworks for testing and documentation. This page explains which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. +## What does "supported" mean? + +"Supported" here means the library provides dedicated wrappers, install options, and guidance for these frameworks. You can still use other code without them. For example, you can wrap any Python callable with [FunctionModel](#functionmodel) or pass [precomputed predictions](#using-precomputed-predictions) when you don't have a model object. + ::: {.callout} ## The library is framework-agnostic Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. ::: -"Supported" here means the library provides dedicated wrappers, install options, and guidance for these frameworks; you can still use other code, such as any callable or precomputed predictions, without them. - ## Framework support -The {{< var validmind.developer >}} provides wrapper classes for common ML frameworks. Choose the wrapper that matches your model's framework: +The {{< var validmind.developer >}} provides wrapper classes for common frameworks. Choose the wrapper that matches your model's framework: | Framework | Wrapper Class | Installation | |-----------|---------------|--------------| From 02970026252948910b4fdf735f5421f4c828199c Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 15:33:45 -0800 Subject: [PATCH 06/30] Minor edit --- site/developer/supported-model-frameworks.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 2b070627c5..c46e543a77 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -122,7 +122,7 @@ flowchart LR ### Using precomputed predictions -If you can't provide a model object (for example, if your model runs in a separate environment), you can pass precomputed predictions directly to the dataset: +If you can't provide a model object because your model runs in a separate environment, you can pass precomputed predictions directly to the dataset: ```python vm_dataset = vm.init_dataset( From 33466afd08e3486a600c4f7663fbe8d53b512e01 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 16:52:23 -0800 Subject: [PATCH 07/30] Re-add some old goodness --- site/developer/supported-model-frameworks.qmd | 119 +++++++++++++++++- 1 file changed, 117 insertions(+), 2 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index c46e543a77..139c08b93f 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -26,7 +26,9 @@ The {{< var validmind.developer >}} supports a wide range of model frameworks fo ## What does "supported" mean? -"Supported" here means the library provides dedicated wrappers, install options, and guidance for these frameworks. You can still use other code without them. For example, you can wrap any Python callable with [FunctionModel](#functionmodel) or pass [precomputed predictions](#using-precomputed-predictions) when you don't have a model object. +"Supported" here means the library provides dedicated wrappers, install options, and guidance for these frameworks. + +You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns: for example, wrap any Python callable with [FunctionModel](#functionmodel) or pass [precomputed predictions](#using-precomputed-predictions) when you don't have a model object. ::: {.callout} ## The library is framework-agnostic @@ -34,10 +36,119 @@ The {{< var validmind.developer >}} supports a wide range of model frameworks fo Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. ::: +## Supported model types + +::: {.column-margin} +::: {.feature} +Vendor models +: {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). + +::: + +::: + +### Traditional statistical models + +:::: {.flex .flex-wrap .justify-around} + +::: {.w-30-ns} + +#### Linear regression +Models the relationship between a scalar response and one or more explanatory variables. + +::: + +::: {.w-30-ns} + +#### Logistic regression +Models the probability of a binary outcome based on one or more predictor variables. + +::: + +::: {.w-30-ns} +#### Time series +Analyzes data points collected or sequenced over time. + +::: + +:::: + +### Machine learning models + +:::: {.flex .flex-wrap .justify-around} + +::: {.w-50-ns .pr2} +#### Hugging Face-compatible models +- Natural language processing (NLP) text classification — Categorizes text into predefined classes. +- Tabular classification — Assigns categories to tabular dataset entries. +- Tabular regression — Predicts continuous outcomes from tabular data. + +#### Neural networks +- Long short-term memory (LSTM) — Processes sequences of data, remembering inputs over long periods. +- Recurrent neural network (RNN) — Processes sequences by maintaining a state that reflects the history of processed elements. +- Convolutional neural network (CNN) — Primarily used for processing grid-like data such as images. + + +::: + +::: {.w-50-ns .pl2 .pr2} + +#### Tree-based models
(XGBoost / CatBoost / random forest) +- Classification — Predicts categorical outcomes using decision trees. +- Regression — Predicts continuous outcomes using decision trees. + +#### K-nearest neighbors (KNN) +- Classification — Assigns class by majority vote of the k-nearest neighbors. +- Regression — Predicts value based on the average of the k-nearest neighbors. + +#### Clustering +- K-means — Partitions _n_ observations into _k_ clusters in which each observation belongs to the cluster with the nearest mean. + + +::: + +:::: + +### Generative AI models + +#### Large language models (LLMs) +- Classification — Categorizes input into predefined classes. +- Text summarization — Generates concise summaries from longer texts. + ## Framework support The {{< var validmind.developer >}} provides wrapper classes for common frameworks. Choose the wrapper that matches your model's framework: +:::: {.flex .flex-wrap .justify-around} + +::: {.w-50-ns} + +- **[scikit-learn](https://scikit-learn.org/stable/)** — A Python library for machine learning, offering supervised and unsupervised learning algorithms. Use `SKlearnModel`. + +- **[statsmodels](https://www.statsmodels.org/stable/index.html)** — A Python module for statistical models, tests, and data exploration. Use `SKlearnModel`. + +- **[PyTorch](https://pytorch.org/)** — An open-source machine learning library for computer vision and natural language processing. Use `PyTorchModel`. + +- **[Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index)** — Pre-trained models for text classification, summarization, translation, and generation. Use `HFModel`. + +::: + +::: {.w-50-ns} + +- **[XGBoost](https://xgboost.readthedocs.io/en/stable/)** — An optimized gradient boosting library for classification and regression. Use `SKlearnModel`. + +- **[CatBoost](https://catboost.ai/)** — Gradient boosting on decision trees with categorical feature support. Use `SKlearnModel`. + +- **[LightGBM](https://lightgbm.readthedocs.io/en/stable/)** — A fast gradient boosting framework for ranking, classification, and regression. Use `SKlearnModel`. + +- **R models, via [rpy2](https://rpy2.github.io/)** — Integrates R's statistical computing with Python. Use `RModel`. + +- **LLMs, via [OpenAI-compatible APIs](https://platform.openai.com/docs/introduction)** — Access to OpenAI and Azure OpenAI models for text generation and analysis. Use `FoundationModel`. + +::: + +:::: + | Framework | Wrapper Class | Installation | |-----------|---------------|--------------| | scikit-learn, XGBoost, CatBoost, LightGBM, StatsModels | `SKlearnModel` | `pip install validmind` | @@ -73,7 +184,7 @@ classDiagram ## Test input requirements -Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case. +Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case.[^1] ### When `predict()` is required @@ -273,3 +384,7 @@ Optional dependencies for specific frameworks: :::{#next-models} ::: + + + +[^1]: [How to run tests and test suites](/developer/how-to/testing-overview.qmd). \ No newline at end of file From 04c88863f1f10c7a03e15aa23cac80fad152c853 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 18:07:38 -0800 Subject: [PATCH 08/30] Much better detail --- site/developer/supported-model-frameworks.qmd | 188 ++++++++++++------ 1 file changed, 125 insertions(+), 63 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 139c08b93f..6fdb68a69c 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -42,121 +42,183 @@ Any Python object with a `predict()` method works with {{< var vm.product >}}. F ::: {.feature} Vendor models : {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). - ::: - ::: -### Traditional statistical models +### AI systems -:::: {.flex .flex-wrap .justify-around} +#### LLM classifiers +LLM-based classification models via OpenAI-compatible APIs. -::: {.w-30-ns} +- Test suite: `LLMClassifierFullSuite` +- Includes prompt validation, text data quality, classifier metrics -#### Linear regression -Models the relationship between a scalar response and one or more explanatory variables. +#### NLP classifiers +Hugging Face and other NLP classification models. + +- Test suite: `NLPClassifierFullSuite` +- Text classification, sentiment analysis, named entity recognition + +#### Summarization models +Models that generate concise summaries from longer texts. + +- Test suite: `SummarizationMetrics` +- BERT, BLEU, METEOR, ROUGE score evaluation + +#### Embeddings models +Text and vector embedding models. + +- Test suite: `EmbeddingsFullSuite` +- Similarity metrics, stability analysis, visualization + +#### RAG systems +Retrieval-augmented generation pipelines. + +- Tests: RAGAS integration +- Faithfulness, context recall/precision, answer relevancy +::: {.callout-note} +## Prompt validation tests + +Tests such as bias, clarity, robustness, and toxicity can be applied to any text-generating model, regardless of type. ::: -::: {.w-30-ns} +### Traditional statistical models + +#### Linear regression +Models the relationship between a scalar response and one or more explanatory variables. + +- Test suite: `RegressionFullSuite` +- R² score, regression errors, feature importance #### Logistic regression Models the probability of a binary outcome based on one or more predictor variables. -::: +- Test suite: `ClassifierFullSuite` +- Confusion matrix, ROC/AUC, precision/recall, feature importance -::: {.w-30-ns} -#### Time series -Analyzes data points collected or sequenced over time. +#### Time series and forecasting +ARIMA, VAR, and other statsmodels time series models. -::: +- Test suite: `TimeSeriesModelValidation` +- Stationarity tests, autocorrelation analysis, forecast evaluation -:::: ### Machine learning models -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns .pr2} #### Hugging Face-compatible models -- Natural language processing (NLP) text classification — Categorizes text into predefined classes. -- Tabular classification — Assigns categories to tabular dataset entries. -- Tabular regression — Predicts continuous outcomes from tabular data. +NLP and tabular models using Hugging Face Transformers. -#### Neural networks -- Long short-term memory (LSTM) — Processes sequences of data, remembering inputs over long periods. -- Recurrent neural network (RNN) — Processes sequences by maintaining a state that reflects the history of processed elements. -- Convolutional neural network (CNN) — Primarily used for processing grid-like data such as images. +- Test suite: `NLPClassifierFullSuite` (for NLP), `ClassifierFullSuite` or `RegressionFullSuite` (for tabular) +- Text classification, tabular classification/regression +#### PyTorch models +Any PyTorch neural network architecture, including LSTM, RNN, CNN, transformers, and custom architectures. -::: +- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` (depending on task) +- Model performance, feature importance, diagnosis -::: {.w-50-ns .pl2 .pr2} +#### Tree-based models (XGBoost / CatBoost / random forest) +Gradient boosting and ensemble tree methods. -#### Tree-based models
(XGBoost / CatBoost / random forest) -- Classification — Predicts categorical outcomes using decision trees. -- Regression — Predicts continuous outcomes using decision trees. +- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` +- Confusion matrix, ROC/AUC, feature importance, SHAP values #### K-nearest neighbors (KNN) -- Classification — Assigns class by majority vote of the k-nearest neighbors. -- Regression — Predicts value based on the average of the k-nearest neighbors. +Distance-based classification and regression. -#### Clustering -- K-means — Partitions _n_ observations into _k_ clusters in which each observation belongs to the cluster with the nearest mean. +- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` +- Performance metrics, model comparison +#### Clustering +Any sklearn-compatible clustering algorithm (K-means, DBSCAN, hierarchical, etc.). -::: - -:::: +- Test suite: `ClusterFullSuite` +- Silhouette score, homogeneity, completeness, cluster distribution -### Generative AI models +#### Embeddings models +Text and vector embedding models. -#### Large language models (LLMs) -- Classification — Categorizes input into predefined classes. -- Text summarization — Generates concise summaries from longer texts. +- Test suite: `EmbeddingsFullSuite` +- Similarity metrics, stability analysis, visualization ## Framework support The {{< var validmind.developer >}} provides wrapper classes for common frameworks. Choose the wrapper that matches your model's framework: -:::: {.flex .flex-wrap .justify-around} +::: {.panel-tabset} -::: {.w-50-ns} +### CatBoost -- **[scikit-learn](https://scikit-learn.org/stable/)** — A Python library for machine learning, offering supervised and unsupervised learning algorithms. Use `SKlearnModel`. +Gradient boosting on decision trees with categorical feature support. -- **[statsmodels](https://www.statsmodels.org/stable/index.html)** — A Python module for statistical models, tests, and data exploration. Use `SKlearnModel`. +- **Wrapper:** `SKlearnModel` +- **Install:** `pip install validmind` -- **[PyTorch](https://pytorch.org/)** — An open-source machine learning library for computer vision and natural language processing. Use `PyTorchModel`. +### HF Transformers -- **[Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index)** — Pre-trained models for text classification, summarization, translation, and generation. Use `HFModel`. +Pre-trained models for text classification, summarization, translation, and generation from the Hugging Face Transformers library. -::: +- **Wrapper:** `HFModel` +- **Install:** `pip install validmind` -::: {.w-50-ns} +### LightGBM -- **[XGBoost](https://xgboost.readthedocs.io/en/stable/)** — An optimized gradient boosting library for classification and regression. Use `SKlearnModel`. +A fast gradient boosting framework for ranking, classification, and regression. -- **[CatBoost](https://catboost.ai/)** — Gradient boosting on decision trees with categorical feature support. Use `SKlearnModel`. +- **Wrapper:** `SKlearnModel` +- **Install:** `pip install validmind` -- **[LightGBM](https://lightgbm.readthedocs.io/en/stable/)** — A fast gradient boosting framework for ranking, classification, and regression. Use `SKlearnModel`. +### LLMs via API -- **R models, via [rpy2](https://rpy2.github.io/)** — Integrates R's statistical computing with Python. Use `RModel`. +Access to OpenAI and Azure OpenAI models for text generation and analysis via OpenAI-compatible APIs. -- **LLMs, via [OpenAI-compatible APIs](https://platform.openai.com/docs/introduction)** — Access to OpenAI and Azure OpenAI models for text generation and analysis. Use `FoundationModel`. +- **Wrapper:** `FoundationModel` +- **Install:** `pip install validmind[llm]` -::: +### PyTorch + +An open-source machine learning library for computer vision and natural language processing. + +- **Wrapper:** `PyTorchModel` +- **Install:** `pip install validmind[pytorch]` + +### R models via rpy2 + +Integrates R's statistical computing with Python. + +- **Wrapper:** `RModel` +- **Install:** `pip install validmind rpy2` + +### scikit-learn -:::: +A Python library for machine learning, offering supervised and unsupervised learning algorithms. -| Framework | Wrapper Class | Installation | -|-----------|---------------|--------------| -| scikit-learn, XGBoost, CatBoost, LightGBM, StatsModels | `SKlearnModel` | `pip install validmind` | -| PyTorch | `PyTorchModel` | `pip install validmind[pytorch]` | -| Hugging Face Transformers | `HFModel` | `pip install validmind` | -| LLM endpoints (OpenAI, Azure, etc.) | `FoundationModel` | `pip install validmind[llm]` | -| Any Python callable | `FunctionModel` | `pip install validmind` | -| R models (via rpy2) | `RModel` | `pip install validmind rpy2` | +- **Wrapper:** `SKlearnModel` +- **Install:** `pip install validmind` + +### statsmodels + +A Python module for statistical models, tests, and data exploration. + +- **Wrapper:** `SKlearnModel` +- **Install:** `pip install validmind` + +### XGBoost + +An optimized gradient boosting library for classification and regression. + +- **Wrapper:** `SKlearnModel` +- **Install:** `pip install validmind` + +### Any Python callable + +Wrap custom prediction functions or models from unsupported frameworks. + +- **Wrapper:** `FunctionModel` +- **Install:** `pip install validmind` + +::: To install all optional dependencies: From 0f0e65a4d7dc5b1c4272be8c241aa86dc5fec8be Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 18:34:14 -0800 Subject: [PATCH 09/30] Edits --- site/developer/supported-model-frameworks.qmd | 42 ++++++++++++------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 6fdb68a69c..744c7bbcb5 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -26,6 +26,13 @@ The {{< var validmind.developer >}} supports a wide range of model frameworks fo ## What does "supported" mean? +::: {.column-margin} +::: {.feature} +Vendor models +: {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). +::: +::: + "Supported" here means the library provides dedicated wrappers, install options, and guidance for these frameworks. You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns: for example, wrap any Python callable with [FunctionModel](#functionmodel) or pass [precomputed predictions](#using-precomputed-predictions) when you don't have a model object. @@ -38,15 +45,10 @@ Any Python object with a `predict()` method works with {{< var vm.product >}}. F ## Supported model types -::: {.column-margin} -::: {.feature} -Vendor models -: {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). -::: -::: - ### AI systems +::: {.panel-tabset} + #### LLM classifiers LLM-based classification models via OpenAI-compatible APIs. @@ -77,14 +79,18 @@ Retrieval-augmented generation pipelines. - Tests: RAGAS integration - Faithfulness, context recall/precision, answer relevancy -::: {.callout-note} -## Prompt validation tests +::: + +::: {.column-margin} +#### Prompt validation tests Tests such as bias, clarity, robustness, and toxicity can be applied to any text-generating model, regardless of type. ::: ### Traditional statistical models +::: {.panel-tabset} + #### Linear regression Models the relationship between a scalar response and one or more explanatory variables. @@ -103,23 +109,26 @@ ARIMA, VAR, and other statsmodels time series models. - Test suite: `TimeSeriesModelValidation` - Stationarity tests, autocorrelation analysis, forecast evaluation +::: ### Machine learning models -#### Hugging Face-compatible models +::: {.panel-tabset} + +#### Hugging Face-compatible NLP and tabular models using Hugging Face Transformers. - Test suite: `NLPClassifierFullSuite` (for NLP), `ClassifierFullSuite` or `RegressionFullSuite` (for tabular) - Text classification, tabular classification/regression -#### PyTorch models +#### PyTorch Any PyTorch neural network architecture, including LSTM, RNN, CNN, transformers, and custom architectures. - Test suite: `ClassifierFullSuite` or `RegressionFullSuite` (depending on task) - Model performance, feature importance, diagnosis -#### Tree-based models (XGBoost / CatBoost / random forest) -Gradient boosting and ensemble tree methods. +#### Tree-based +Gradient boosting and ensemble tree methods (XGBoost, CatBoost, random forest). - Test suite: `ClassifierFullSuite` or `RegressionFullSuite` - Confusion matrix, ROC/AUC, feature importance, SHAP values @@ -131,17 +140,18 @@ Distance-based classification and regression. - Performance metrics, model comparison #### Clustering -Any sklearn-compatible clustering algorithm (K-means, DBSCAN, hierarchical, etc.). +Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hierarchical clustering. - Test suite: `ClusterFullSuite` - Silhouette score, homogeneity, completeness, cluster distribution -#### Embeddings models +#### Embeddings Text and vector embedding models. - Test suite: `EmbeddingsFullSuite` - Similarity metrics, stability analysis, visualization +::: ## Framework support The {{< var validmind.developer >}} provides wrapper classes for common frameworks. Choose the wrapper that matches your model's framework: @@ -155,7 +165,7 @@ Gradient boosting on decision trees with categorical feature support. - **Wrapper:** `SKlearnModel` - **Install:** `pip install validmind` -### HF Transformers +### Hugging Face Transformers Pre-trained models for text classification, summarization, translation, and generation from the Hugging Face Transformers library. From c695ba7b6dc004e969097eea6f1707d86d401809 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 18:49:14 -0800 Subject: [PATCH 10/30] More edits --- site/developer/supported-model-frameworks.qmd | 28 +++---------------- 1 file changed, 4 insertions(+), 24 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 744c7bbcb5..6af8a48a53 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -24,7 +24,7 @@ listing: The {{< var validmind.developer >}} supports a wide range of model frameworks for testing and documentation. This page explains which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. -## What does "supported" mean? +## What is supported? ::: {.column-margin} ::: {.feature} @@ -43,8 +43,6 @@ You can also use other code with the {{< var vm.developer >}} for models that do Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. ::: -## Supported model types - ### AI systems ::: {.panel-tabset} @@ -77,7 +75,7 @@ Text and vector embedding models. Retrieval-augmented generation pipelines. - Tests: RAGAS integration -- Faithfulness, context recall/precision, answer relevancy +- See [RAG evaluation](#rag-evaluation) for dataset requirements and available tests ::: @@ -236,24 +234,6 @@ To install all optional dependencies: pip install validmind[all] ``` -### Class hierarchy - -The following diagram shows how model wrapper classes relate to each other: - -```{mermaid} -classDiagram - VMModel <|-- SKlearnModel - VMModel <|-- PyTorchModel - VMModel <|-- HFModel - VMModel <|-- FunctionModel - VMModel <|-- RModel - SKlearnModel <|-- XGBoostModel - SKlearnModel <|-- CatBoostModel - SKlearnModel <|-- StatsModelsModel - FunctionModel <|-- FoundationModel - FunctionModel <|-- PipelineModel -``` - ## Test input requirements Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case.[^1] @@ -371,9 +351,9 @@ vm_model = vm.init_model( ) ``` -## GenAI and LLM support +## Setting up for LLMs and GenAI -The {{< var validmind.developer >}} provides specialized support for large language models and generative AI. +This section covers how to configure and test large language models and generative AI systems. ### FoundationModel From 16f3f6fc7b8cbb6c25367dbc58744879afd6e49a Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 19:37:11 -0800 Subject: [PATCH 11/30] Edits --- site/developer/supported-model-frameworks.qmd | 126 +++++++++++------- 1 file changed, 77 insertions(+), 49 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 6af8a48a53..6d0af50a91 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -46,7 +46,6 @@ Any Python object with a `predict()` method works with {{< var vm.product >}}. F ### AI systems ::: {.panel-tabset} - #### LLM classifiers LLM-based classification models via OpenAI-compatible APIs. @@ -76,7 +75,6 @@ Retrieval-augmented generation pipelines. - Tests: RAGAS integration - See [RAG evaluation](#rag-evaluation) for dataset requirements and available tests - ::: ::: {.column-margin} @@ -85,30 +83,6 @@ Retrieval-augmented generation pipelines. Tests such as bias, clarity, robustness, and toxicity can be applied to any text-generating model, regardless of type. ::: -### Traditional statistical models - -::: {.panel-tabset} - -#### Linear regression -Models the relationship between a scalar response and one or more explanatory variables. - -- Test suite: `RegressionFullSuite` -- R² score, regression errors, feature importance - -#### Logistic regression -Models the probability of a binary outcome based on one or more predictor variables. - -- Test suite: `ClassifierFullSuite` -- Confusion matrix, ROC/AUC, precision/recall, feature importance - -#### Time series and forecasting -ARIMA, VAR, and other statsmodels time series models. - -- Test suite: `TimeSeriesModelValidation` -- Stationarity tests, autocorrelation analysis, forecast evaluation - -::: - ### Machine learning models ::: {.panel-tabset} @@ -148,19 +122,42 @@ Text and vector embedding models. - Test suite: `EmbeddingsFullSuite` - Similarity metrics, stability analysis, visualization +::: + + +### Traditional statistical models + +::: {.panel-tabset} +#### Linear regression +Models the relationship between a scalar response and one or more explanatory variables. + +- Test suite: `RegressionFullSuite` +- R² score, regression errors, feature importance + +#### Logistic regression +Models the probability of a binary outcome based on one or more predictor variables. +- Test suite: `ClassifierFullSuite` +- Confusion matrix, ROC/AUC, precision/recall, feature importance + +#### Time series and forecasting +ARIMA, VAR, and other statsmodels time series models. + +- Test suite: `TimeSeriesModelValidation` +- Stationarity tests, autocorrelation analysis, forecast evaluation ::: + + ## Framework support -The {{< var validmind.developer >}} provides wrapper classes for common frameworks. Choose the wrapper that matches your model's framework: +The {{< var validmind.developer >}} provides wrapper classes for common frameworks. When you call `vm.init_model()`, the library automatically detects your model's framework from its module, such as `sklearn`, `torch`, or `transformers`, and selects the appropriate wrapper. You don't need to specify the wrapper class manually. ::: {.panel-tabset} - ### CatBoost Gradient boosting on decision trees with categorical feature support. -- **Wrapper:** `SKlearnModel` +- **Wrapper:** `CatBoostModel` - **Install:** `pip install validmind` ### Hugging Face Transformers @@ -168,14 +165,8 @@ Gradient boosting on decision trees with categorical feature support. Pre-trained models for text classification, summarization, translation, and generation from the Hugging Face Transformers library. - **Wrapper:** `HFModel` -- **Install:** `pip install validmind` - -### LightGBM - -A fast gradient boosting framework for ranking, classification, and regression. - -- **Wrapper:** `SKlearnModel` -- **Install:** `pip install validmind` +- **Install:** `pip install validmind[huggingface]` +- **Supported tasks:** `text_classification`, `text2text_generation` (summarization), `feature_extraction` (embeddings) ### LLMs via API @@ -193,10 +184,12 @@ An open-source machine learning library for computer vision and natural language ### R models via rpy2 -Integrates R's statistical computing with Python. +Integrates R's statistical computing with Python via rpy2. - **Wrapper:** `RModel` - **Install:** `pip install validmind rpy2` +- **Supported model types:** `glm` (including logistic regression), `lm`, `xgb.Booster` +- **Initialization:** Use `vm.init_r_model(model_path, input_id)` to load R model objects ### scikit-learn @@ -209,15 +202,15 @@ A Python library for machine learning, offering supervised and unsupervised lear A Python module for statistical models, tests, and data exploration. -- **Wrapper:** `SKlearnModel` -- **Install:** `pip install validmind` +- **Wrapper:** `StatsModelsModel` +- **Install:** `pip install validmind[stats]` ### XGBoost An optimized gradient boosting library for classification and regression. -- **Wrapper:** `SKlearnModel` -- **Install:** `pip install validmind` +- **Wrapper:** `XGBoostModel` +- **Install:** `pip install validmind[xgboost]` ### Any Python callable @@ -225,7 +218,6 @@ Wrap custom prediction functions or models from unsupported frameworks. - **Wrapper:** `FunctionModel` - **Install:** `pip install validmind` - ::: To install all optional dependencies: @@ -238,6 +230,8 @@ pip install validmind[all] Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case.[^1] +::: {.panel-tabset} + ### When `predict()` is required Most model tests call your model's `predict()` method to generate predictions. This includes: @@ -257,7 +251,9 @@ Classification metrics that evaluate probability outputs require `predict_proba( If your model doesn't have `predict_proba()`, these tests will be skipped or return an error. -### Test input flow +::: + +### Test input flow ```{mermaid} flowchart LR @@ -317,6 +313,17 @@ Some tests analyze data quality and don't require a model at all: - Outlier detection - Data drift tests +### Supported dataset formats + +The library accepts multiple data formats when initializing datasets with `vm.init_dataset()`: + +| Format | Notes | +|--------|-------| +| Pandas DataFrame | Primary format, used internally | +| Polars DataFrame | Converted to pandas internally | +| NumPy ndarray | Requires column names to be specified | +| PyTorch TensorDataset | Requires `pip install validmind[pytorch]` | + ## Custom model wrappers For models that don't fit standard framework patterns, use these flexible wrappers. @@ -351,6 +358,21 @@ vm_model = vm.init_model( ) ``` +### MetadataModel + +Use when you have model metadata but no inference capability — for example, when documenting a model deployed in an external system: + +```python +from validmind.models import MetadataModel + +vm_model = vm.init_model( + model=MetadataModel(), + input_id="external_model" +) +``` + +With `MetadataModel`, you can still run dataset-only tests and use precomputed predictions. + ## Setting up for LLMs and GenAI This section covers how to configure and test large language models and generative AI systems. @@ -424,12 +446,18 @@ The {{< var validmind.developer >}} requires: Optional dependencies for specific frameworks: -| Extra | Frameworks | -|-------|------------| +| Extra | Includes | +|-------|----------| | `pytorch` | PyTorch, torchvision | -| `llm` | OpenAI, langchain, ragas | +| `llm` | OpenAI, langchain, ragas, deepeval | | `xgboost` | XGBoost | -| `catboost` | CatBoost | +| `huggingface` | Transformers, sentencepiece | +| `nlp` | langdetect, nltk, textblob, evaluate, rouge, bert-score | +| `stats` | scipy, statsmodels, arch | +| `explainability` | SHAP | +| `credit_risk` | scorecardpy | +| `datasets` | Hugging Face datasets | +| `pii-detection` | presidio-analyzer, presidio-structured | | `all` | All optional dependencies | ## What's next @@ -439,4 +467,4 @@ Optional dependencies for specific frameworks: -[^1]: [How to run tests and test suites](/developer/how-to/testing-overview.qmd). \ No newline at end of file +[^1]: [How to run tests and test suites](/developer/how-to/testing-overview.qmd). From eaef82b1a23036a36d4cfa7acb390e12e5a506d7 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 19:41:09 -0800 Subject: [PATCH 12/30] Remove old supported models page --- site/developer/supported-models.qmd | 160 ---------------------------- 1 file changed, 160 deletions(-) delete mode 100644 site/developer/supported-models.qmd diff --git a/site/developer/supported-models.qmd b/site/developer/supported-models.qmd deleted file mode 100644 index 930db0533f..0000000000 --- a/site/developer/supported-models.qmd +++ /dev/null @@ -1,160 +0,0 @@ ---- -# Copyright © 2023-2026 ValidMind Inc. All rights reserved. -# Refer to the LICENSE file in the root of this repository for details. -# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial -title: "Supported models" -date: last-modified -aliases: - - /guide/supported-models.html - - /developer/model-documentation/supported-models.html -listing: - - id: next-models - type: grid - grid-columns: 2 - max-description-length: 250 - sort: false - fields: [title, description] - contents: - - /how-to/testing-overview.qmd - - test-descriptions.qmd - - /how-to/feature-overview.qmd - - samples-jupyter-notebooks.qmd ---- - -The {{< var validmind.developer >}} provides out-of-the-box support for testing and documentation for an array of model types and modeling packages. - -## What is a supported model? - -A _supported model_ refers to a model for which predefined testing or documentation functions exist in the {{< var validmind.developer >}}, provided that the model you are developing is documented using a supported version of our {{< var vm.developer >}}. These model types cover a very large portion of the models used in commercial and retail banking. - -::: {.callout} -## {{< var vm.product >}} does not limit our users to specific model types. - -- The {{< var validmind.developer >}} is extensible to support future model types or modeling packages to accomodate rapid developments in the AI space, including the advent of large language models (LLMs). -- You always have the flexibility to implement custom tests and integrate external test providers.[^1] -::: - - - -## Supported model types - -::: {.column-margin} -::: {.feature} -Vendor models -: {{< var vm.product >}} offers support for both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). - -::: - -::: - -### Traditional statistical models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-30-ns} - -#### Linear regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} - -#### Logistic regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} -#### Time series -Analyzes data points collected or sequenced over time. - -::: - -:::: - -### Machine learning models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns .pr2} -#### Hugging Face-compatible models -- Natural language processing (NLP) text classification — Categorizes text into predefined classes. -- Tabular classification — Assigns categories to tabular dataset entries. -- Tabular regression — Predicts continuous outcomes from tabular data. - -#### Neural networks -- Long short-term memory (LSTM) — Processes sequences of data, remembering inputs over long periods. -- Recurrent neural network (RNN) — Processes sequences by maintaining a state that reflects the history of processed elements. -- Convolutional neural network (CNN) — Primarily used for processing grid-like data such as images. - - -::: - -::: {.w-50-ns .pl2 .pr2} - -#### Tree-based models
(XGBoost / CatBoost / random forest) -- Classification — Predicts categorical outcomes using decision trees. -- Regression — Predicts continuous outcomes using decision trees. - -#### K-nearest neighbors (KNN) -- Classification — Assigns class by majority vote of the k-nearest neighbors. -- Regression — Predicts value based on the average of the k-nearest neighbors. - -#### Clustering -- K-means — Partitions _n_ observations into _k_ clusters in which each observation belongs to the cluster with the nearest mean. - - -::: - -:::: - -### Generative AI models - -#### Large language models (LLMs) -- Classification — Categorizes input into predefined classes. -- Text summarization — Generates concise summaries from longer texts. - -## Supported modeling libraries and other tools - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns} - -- **[scikit-learn](https://scikit-learn.org/stable/)** — A Python library for machine learning, offering a range of supervised and unsupervised learning algorithms. - -- **[statsmodels](https://www.statsmodels.org/stable/index.html)** — A Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests and exploring data. - -- **[PyTorch](https://pytorch.org/)** — An open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing. - -- **[Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index)** — Provides thousands of pre-trained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, and text generation. - -- **[XGBoost](https://xgboost.readthedocs.io/en/stable/)** — An optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable, implementing machine learning algorithms under the Gradient Boosting framework. - -::: - -::: {.w-50-ns} - -- **[CatBoost](https://catboost.ai/)** — An open-source gradient boosting on decision trees library with categorical feature support out of the box, for ranking, classification, regression, and other ML tasks. - -- **[LightGBM](https://lightgbm.readthedocs.io/en/stable/)** — A fast, distributed, high-performance gradient boosting (GBDT, GBRT, GBM, or MART) framework based on decision tree algorithms, used for ranking, classification, and many other machine learning tasks. - -- **R models, via [rpy2 - R in Python](https://rpy2.github.io/)** — Facilitates the integration of R's statistical computing and graphics capabilities with Python, allowing for R models to be called from Python. - -- **Large language models (LLMs), via [OpenAI-compatible APIs](https://platform.openai.com/docs/introduction)** — Access to advanced AI models trained by OpenAI for a variety of natural language tasks, including text generation, translation, and analysis, through a compatible API interface. This support includes both the OpenAI API and the Azure OpenAI Service via API. -::: - -:::: - -## What's next - -:::{#next-models} -::: - - - - -[^1]: - - - [Implement custom tests](/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb) - - [Integrate external test providers](/notebooks/how_to/tests/custom_tests/integrate_external_test_providers.ipynb) \ No newline at end of file From e7ac6ebbdd7ad3c48cb6ae7a9cb77a550503a7ea Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 5 Mar 2026 19:48:22 -0800 Subject: [PATCH 13/30] Minor edit --- site/developer/supported-model-frameworks.qmd | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 6d0af50a91..7e829e50ac 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -212,9 +212,9 @@ An optimized gradient boosting library for classification and regression. - **Wrapper:** `XGBoostModel` - **Install:** `pip install validmind[xgboost]` -### Any Python callable +### Python callable -Wrap custom prediction functions or models from unsupported frameworks. +Wrap custom prediction functions or models from otherwise unsupported frameworks. - **Wrapper:** `FunctionModel` - **Install:** `pip install validmind` @@ -326,8 +326,9 @@ The library accepts multiple data formats when initializing datasets with `vm.in ## Custom model wrappers -For models that don't fit standard framework patterns, use these flexible wrappers. +For models that don't fit standard framework patterns, use these flexible wrappers: +::: {.panel-tabset} ### FunctionModel Wrap any Python callable as a model: @@ -398,6 +399,7 @@ vm_model = vm.init_model( input_id="sentiment_classifier" ) ``` +::: ### LLM test suites From 6be94b7f180ac3c4bd55d82cff368b1b7be363d0 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Fri, 6 Mar 2026 08:17:06 -0800 Subject: [PATCH 14/30] Edits --- site/developer/supported-model-frameworks.qmd | 83 +++++++++---------- 1 file changed, 38 insertions(+), 45 deletions(-) diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-model-frameworks.qmd index 7e829e50ac..3fba300224 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-model-frameworks.qmd @@ -24,100 +24,93 @@ listing: The {{< var validmind.developer >}} supports a wide range of model frameworks for testing and documentation. This page explains which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. -## What is supported? +## What does "supported" mean? -::: {.column-margin} -::: {.feature} -Vendor models -: {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). -::: -::: +"Supported" here means the library provides dedicated wrappers, install options, and guidance for these models and frameworks. -"Supported" here means the library provides dedicated wrappers, install options, and guidance for these frameworks. +You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns. For example, you can wrap any Python callable with `FunctionModel` or pass precomputed predictions when you don't have a model object. -You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns: for example, wrap any Python callable with [FunctionModel](#functionmodel) or pass [precomputed predictions](#using-precomputed-predictions) when you don't have a model object. +Of note: -::: {.callout} -## The library is framework-agnostic +- The library is framework-agnostic — Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. -Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. -::: +- Vendor models are supported — {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). -### AI systems +## AI systems ::: {.panel-tabset} -#### LLM classifiers +## LLM classifiers LLM-based classification models via OpenAI-compatible APIs. - Test suite: `LLMClassifierFullSuite` - Includes prompt validation, text data quality, classifier metrics -#### NLP classifiers +## NLP classifiers Hugging Face and other NLP classification models. - Test suite: `NLPClassifierFullSuite` - Text classification, sentiment analysis, named entity recognition -#### Summarization models +## Summarization models Models that generate concise summaries from longer texts. - Test suite: `SummarizationMetrics` - BERT, BLEU, METEOR, ROUGE score evaluation -#### Embeddings models +## Embeddings models Text and vector embedding models. - Test suite: `EmbeddingsFullSuite` - Similarity metrics, stability analysis, visualization -#### RAG systems +## RAG systems Retrieval-augmented generation pipelines. - Tests: RAGAS integration - See [RAG evaluation](#rag-evaluation) for dataset requirements and available tests ::: -::: {.column-margin} + -### Machine learning models +## Machine learning models ::: {.panel-tabset} -#### Hugging Face-compatible +## Hugging Face-compatible NLP and tabular models using Hugging Face Transformers. - Test suite: `NLPClassifierFullSuite` (for NLP), `ClassifierFullSuite` or `RegressionFullSuite` (for tabular) - Text classification, tabular classification/regression -#### PyTorch +## PyTorch Any PyTorch neural network architecture, including LSTM, RNN, CNN, transformers, and custom architectures. - Test suite: `ClassifierFullSuite` or `RegressionFullSuite` (depending on task) - Model performance, feature importance, diagnosis -#### Tree-based +## Tree-based Gradient boosting and ensemble tree methods (XGBoost, CatBoost, random forest). - Test suite: `ClassifierFullSuite` or `RegressionFullSuite` - Confusion matrix, ROC/AUC, feature importance, SHAP values -#### K-nearest neighbors (KNN) +## K-nearest neighbors (KNN) Distance-based classification and regression. - Test suite: `ClassifierFullSuite` or `RegressionFullSuite` - Performance metrics, model comparison -#### Clustering +## Clustering Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hierarchical clustering. - Test suite: `ClusterFullSuite` - Silhouette score, homogeneity, completeness, cluster distribution -#### Embeddings +## Embeddings Text and vector embedding models. - Test suite: `EmbeddingsFullSuite` @@ -125,22 +118,22 @@ Text and vector embedding models. ::: -### Traditional statistical models +## Traditional statistical models ::: {.panel-tabset} -#### Linear regression +## Linear regression Models the relationship between a scalar response and one or more explanatory variables. - Test suite: `RegressionFullSuite` - R² score, regression errors, feature importance -#### Logistic regression +## Logistic regression Models the probability of a binary outcome based on one or more predictor variables. - Test suite: `ClassifierFullSuite` - Confusion matrix, ROC/AUC, precision/recall, feature importance -#### Time series and forecasting +## Time series and forecasting ARIMA, VAR, and other statsmodels time series models. - Test suite: `TimeSeriesModelValidation` @@ -148,19 +141,19 @@ ARIMA, VAR, and other statsmodels time series models. ::: -## Framework support +## Frameworks The {{< var validmind.developer >}} provides wrapper classes for common frameworks. When you call `vm.init_model()`, the library automatically detects your model's framework from its module, such as `sklearn`, `torch`, or `transformers`, and selects the appropriate wrapper. You don't need to specify the wrapper class manually. ::: {.panel-tabset} -### CatBoost +## CatBoost Gradient boosting on decision trees with categorical feature support. - **Wrapper:** `CatBoostModel` - **Install:** `pip install validmind` -### Hugging Face Transformers +## Hugging Face Transformers Pre-trained models for text classification, summarization, translation, and generation from the Hugging Face Transformers library. @@ -168,21 +161,21 @@ Pre-trained models for text classification, summarization, translation, and gene - **Install:** `pip install validmind[huggingface]` - **Supported tasks:** `text_classification`, `text2text_generation` (summarization), `feature_extraction` (embeddings) -### LLMs via API +## LLMs via API Access to OpenAI and Azure OpenAI models for text generation and analysis via OpenAI-compatible APIs. - **Wrapper:** `FoundationModel` - **Install:** `pip install validmind[llm]` -### PyTorch +## PyTorch An open-source machine learning library for computer vision and natural language processing. - **Wrapper:** `PyTorchModel` - **Install:** `pip install validmind[pytorch]` -### R models via rpy2 +## R models via rpy2 Integrates R's statistical computing with Python via rpy2. @@ -191,28 +184,28 @@ Integrates R's statistical computing with Python via rpy2. - **Supported model types:** `glm` (including logistic regression), `lm`, `xgb.Booster` - **Initialization:** Use `vm.init_r_model(model_path, input_id)` to load R model objects -### scikit-learn +## scikit-learn A Python library for machine learning, offering supervised and unsupervised learning algorithms. - **Wrapper:** `SKlearnModel` - **Install:** `pip install validmind` -### statsmodels +## statsmodels A Python module for statistical models, tests, and data exploration. - **Wrapper:** `StatsModelsModel` - **Install:** `pip install validmind[stats]` -### XGBoost +## XGBoost An optimized gradient boosting library for classification and regression. - **Wrapper:** `XGBoostModel` - **Install:** `pip install validmind[xgboost]` -### Python callable +## Python callable Wrap custom prediction functions or models from otherwise unsupported frameworks. @@ -226,13 +219,13 @@ To install all optional dependencies: pip install validmind[all] ``` -## Test input requirements +### Test input requirements Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case.[^1] ::: {.panel-tabset} -### When `predict()` is required +#### When `predict()` is required Most model tests call your model's `predict()` method to generate predictions. This includes: @@ -240,7 +233,7 @@ Most model tests call your model's `predict()` method to generate predictions. T - Error analysis tests - Robustness tests -### When `predict_proba()` is needed +#### When `predict_proba()` is needed Classification metrics that evaluate probability outputs require `predict_proba()`: From 5c1b48b0f12b4aa37c2442e17ed737f1aa30523b Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Fri, 6 Mar 2026 08:19:30 -0800 Subject: [PATCH 15/30] File rename again --- site/about/overview-model-documentation.qmd | 2 +- site/developer/_sidebar.yaml | 2 +- ...del-frameworks.qmd => supported-models-and-frameworks.qmd} | 3 ++- site/faq/faq-integrations.qmd | 4 ++-- site/releases/2024/2024-mar-27/highlights.qmd | 2 +- 5 files changed, 7 insertions(+), 6 deletions(-) rename site/developer/{supported-model-frameworks.qmd => supported-models-and-frameworks.qmd} (99%) diff --git a/site/about/overview-model-documentation.qmd b/site/about/overview-model-documentation.qmd index 955aa0e22c..59e462e4aa 100644 --- a/site/about/overview-model-documentation.qmd +++ b/site/about/overview-model-documentation.qmd @@ -146,7 +146,7 @@ How the {{< var validmind.developer >}} works: [^1]: [Model risk management](overview-model-risk-management.qmd) -[^2]: [Supported model frameworks](/developer/supported-model-frameworks.qmd) +[^2]: [Supported models and frameworks](/developer/supported-models-and-frameworks.qmd) [^3]: [Customize document templates](/guide/templates/customize-document-templates.qmd) diff --git a/site/developer/_sidebar.yaml b/site/developer/_sidebar.yaml index 5e6d3cc3c6..a4c11adc38 100644 --- a/site/developer/_sidebar.yaml +++ b/site/developer/_sidebar.yaml @@ -10,7 +10,7 @@ website: # USING THE VARIABLE IN THE LINK TEXT MESSES UP THE MOBILE VIEW - text: "ValidMind Library" file: developer/validmind-library.qmd - - developer/supported-model-frameworks.qmd + - developer/supported-models-and-frameworks.qmd - text: "---" - text: "Quickstart" - notebooks/quickstart/quickstart_model_documentation.ipynb diff --git a/site/developer/supported-model-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd similarity index 99% rename from site/developer/supported-model-frameworks.qmd rename to site/developer/supported-models-and-frameworks.qmd index 3fba300224..dfdce55458 100644 --- a/site/developer/supported-model-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -2,12 +2,13 @@ # Copyright © 2023-2026 ValidMind Inc. All rights reserved. # Refer to the LICENSE file in the root of this repository for details. # SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial -title: "Supported model frameworks" +title: "Supported models and frameworks" date: last-modified aliases: - /guide/supported-models.html - /developer/model-documentation/supported-models.html - /developer/supported-models.html + - /developer/supported-model-frameworks.html listing: - id: next-models type: grid diff --git a/site/faq/faq-integrations.qmd b/site/faq/faq-integrations.qmd index ad37b372d9..a8c86a0238 100644 --- a/site/faq/faq-integrations.qmd +++ b/site/faq/faq-integrations.qmd @@ -14,7 +14,7 @@ listing: sort: false fields: [title, description] contents: - - ../developer/supported-model-frameworks.qmd + - ../developer/supported-models-and-frameworks.qmd - ../about/overview-llm-features.qmd - ../about/deployment/deployment-options.qmd categories: ["supported libraries", "supported languages", "integrations", "images", "large language models", "explainability", "deployment options", "validmind library"] @@ -82,7 +82,7 @@ We will be implementing connector interfaces allowing extraction of relationship [^1]: [{{< var validmind.developer >}}](/developer/validmind-library.qmd) -[^2]: [Supported model frameworks](/developer/supported-model-frameworks.qmd) +[^2]: [Supported models and frameworks](/developer/supported-models-and-frameworks.qmd) [^3]: [Matplotlib](https://matplotlib.org/) diff --git a/site/releases/2024/2024-mar-27/highlights.qmd b/site/releases/2024/2024-mar-27/highlights.qmd index 4c4d929f7c..162ef46335 100644 --- a/site/releases/2024/2024-mar-27/highlights.qmd +++ b/site/releases/2024/2024-mar-27/highlights.qmd @@ -458,7 +458,7 @@ We improved our supported models documentation with additional information about ::: ::: {.w-30-ns .tc} -[Supported model frameworks](/developer/supported-model-frameworks.qmd){.button} +[Supported models and frameworks](/developer/supported-models-and-frameworks.qmd){.button} ::: From 0a741dd8b9c4ed90604e8514c741a27a5e1ab7b2 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Fri, 6 Mar 2026 08:23:42 -0800 Subject: [PATCH 16/30] Remove duplicated embeddings content --- site/developer/supported-models-and-frameworks.qmd | 7 ------- 1 file changed, 7 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index dfdce55458..12b9cb2b95 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -80,7 +80,6 @@ Tests such as bias, clarity, robustness, and toxicity can be applied to any text ## Machine learning models ::: {.panel-tabset} - ## Hugging Face-compatible NLP and tabular models using Hugging Face Transformers. @@ -110,12 +109,6 @@ Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hier - Test suite: `ClusterFullSuite` - Silhouette score, homogeneity, completeness, cluster distribution - -## Embeddings -Text and vector embedding models. - -- Test suite: `EmbeddingsFullSuite` -- Similarity metrics, stability analysis, visualization ::: From 73e2595464e50570e5a69e4601d31607a45dbd00 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Fri, 6 Mar 2026 08:29:58 -0800 Subject: [PATCH 17/30] Uncomment and move info about tests for text-generating models --- site/developer/supported-models-and-frameworks.qmd | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 12b9cb2b95..4a31456cbc 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -71,12 +71,6 @@ Retrieval-augmented generation pipelines. - See [RAG evaluation](#rag-evaluation) for dataset requirements and available tests ::: - - ## Machine learning models ::: {.panel-tabset} @@ -404,6 +398,8 @@ Available test suites for LLMs include: - Toxicity analysis - Bias evaluation +Tests such as bias, clarity, robustness, and toxicity can be applied to any text-generating model, regardless of type. + ## RAG evaluation For retrieval-augmented generation (RAG) systems, the {{< var validmind.developer >}} integrates with [RAGAS](https://docs.ragas.io/) for comprehensive evaluation. From 791915f832e0365750044a4a02d9381d371bdd52 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 15:36:09 -0700 Subject: [PATCH 18/30] Update site/developer/supported-models-and-frameworks.qmd Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com> --- site/developer/supported-models-and-frameworks.qmd | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 4a31456cbc..79df8f6359 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -25,7 +25,9 @@ listing: The {{< var validmind.developer >}} supports a wide range of model frameworks for testing and documentation. This page explains which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. -## What does "supported" mean? +::: {.attn} + +## What does _supported_ mean? "Supported" here means the library provides dedicated wrappers, install options, and guidance for these models and frameworks. From 3c26a38a211360f3a7eafc521f91390a521d6b43 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 15:36:41 -0700 Subject: [PATCH 19/30] Update site/developer/supported-models-and-frameworks.qmd Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com> --- site/developer/supported-models-and-frameworks.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 79df8f6359..6f9d803a43 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -29,7 +29,7 @@ The {{< var validmind.developer >}} supports a wide range of model frameworks fo ## What does _supported_ mean? -"Supported" here means the library provides dedicated wrappers, install options, and guidance for these models and frameworks. +_Supported_ here means the {{< var validmind.developer >}} provides dedicated wrappers, install options, and guidance for these models and frameworks. You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns. For example, you can wrap any Python callable with `FunctionModel` or pass precomputed predictions when you don't have a model object. From 585dce349de15680fef4e081037e367334e4edad Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 15:37:01 -0700 Subject: [PATCH 20/30] Update site/developer/supported-models-and-frameworks.qmd Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com> --- site/developer/supported-models-and-frameworks.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 6f9d803a43..54bca77b97 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -23,7 +23,7 @@ listing: - samples-jupyter-notebooks.qmd --- -The {{< var validmind.developer >}} supports a wide range of model frameworks for testing and documentation. This page explains which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. +Learn about the wide range of model frameworks for testing and documentation supported by the {{< var validmind.developer >}}. Understand which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. ::: {.attn} From 0bfd4f201b8cf7f570bed26ce08d4e97d0afc221 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 15:37:13 -0700 Subject: [PATCH 21/30] Update site/developer/supported-models-and-frameworks.qmd Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com> --- site/developer/supported-models-and-frameworks.qmd | 1 - 1 file changed, 1 deletion(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 54bca77b97..615c2672db 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -33,7 +33,6 @@ _Supported_ here means the {{< var validmind.developer >}} provides dedicated wr You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns. For example, you can wrap any Python callable with `FunctionModel` or pass precomputed predictions when you don't have a model object. -Of note: - The library is framework-agnostic — Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. From 8530a279cdd667370ea556e7d730de8a25b58458 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 15:40:50 -0700 Subject: [PATCH 22/30] Update site/developer/supported-models-and-frameworks.qmd Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com> --- site/developer/supported-models-and-frameworks.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 615c2672db..8aed05369c 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -69,7 +69,7 @@ Text and vector embedding models. Retrieval-augmented generation pipelines. - Tests: RAGAS integration -- See [RAG evaluation](#rag-evaluation) for dataset requirements and available tests +- Refer to [RAG evaluation](#rag-evaluation) for dataset requirements and available tests ::: ## Machine learning models From d03b58adf750bd5d96f5cc65f6fa72488f31c60c Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 15:42:25 -0700 Subject: [PATCH 23/30] Switch to definition list after removing lead-in text for bulleted list --- site/developer/supported-models-and-frameworks.qmd | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 8aed05369c..3a60694ab2 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -33,10 +33,11 @@ _Supported_ here means the {{< var validmind.developer >}} provides dedicated wr You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns. For example, you can wrap any Python callable with `FunctionModel` or pass precomputed predictions when you don't have a model object. +The library is framework-agnostic +: Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. -- The library is framework-agnostic — Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. - -- Vendor models are supported — {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). +Vendor models are supported +: {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). ## AI systems From 0fc9c621252da7dbd9172d85812ac4fcd65fa6dc Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 16:04:45 -0700 Subject: [PATCH 24/30] Two columns for a neater layout --- .../supported-models-and-frameworks.qmd | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 3a60694ab2..e6b837f41a 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -33,11 +33,19 @@ _Supported_ here means the {{< var validmind.developer >}} provides dedicated wr You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns. For example, you can wrap any Python callable with `FunctionModel` or pass precomputed predictions when you don't have a model object. -The library is framework-agnostic -: Any Python object with a `predict()` method works with {{< var vm.product >}}. Framework-specific wrappers provide enhanced functionality but aren't strictly required. +:::: {.columns} -Vendor models are supported -: {{< var vm.product >}} supports both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). +::: {.column width="50%" .pr3} +Framework-agnostic support +: Any Python object with a `predict()` method is compatible. Framework-specific wrappers provide extra features and convenience, but they are not required and you can use generic Python models without them. +::: + +::: {.column width="50%" .pl3} +Vendor model compatibility +: {{< var vm.product >}} works with both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model), enabling flexible integration with proprietary and external model sources. +::: + +:::: ## AI systems From 72c74e237f35641d301335f009c321b339e94551 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 16:23:02 -0700 Subject: [PATCH 25/30] Remove pointless link --- site/developer/supported-models-and-frameworks.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index e6b837f41a..8b468f150c 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -42,7 +42,7 @@ Framework-agnostic support ::: {.column width="50%" .pl3} Vendor model compatibility -: {{< var vm.product >}} works with both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model), enabling flexible integration with proprietary and external model sources. +: The {{< var vm.developer >}} works with both first-party models you create and third-party models from external vendors, enabling flexible integration with both proprietary and external sources. ::: :::: @@ -462,4 +462,4 @@ Optional dependencies for specific frameworks: -[^1]: [How to run tests and test suites](/developer/how-to/testing-overview.qmd). +[^1]: [How to run tests and test suites](/developer/how-to/testing-overview.qmd) From e7e98b4fd183430ca3161838e79e5bdf340c4594 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 17:01:47 -0700 Subject: [PATCH 26/30] Add links, text edits --- .../supported-models-and-frameworks.qmd | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 8b468f150c..029d4ff21b 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -37,12 +37,12 @@ You can also use other code with the {{< var vm.developer >}} for models that do ::: {.column width="50%" .pr3} Framework-agnostic support -: Any Python object with a `predict()` method is compatible. Framework-specific wrappers provide extra features and convenience, but they are not required and you can use generic Python models without them. +: Compatible with any Python object that implements a `predict()` method. Framework-specific wrappers provide extra features and convenience, but they are not required to use generic Python models. ::: ::: {.column width="50%" .pl3} Vendor model compatibility -: The {{< var vm.developer >}} works with both first-party models you create and third-party models from external vendors, enabling flexible integration with both proprietary and external sources. +: Works with both first-party models you create and third-party models from external vendors, enabling flexible integration with both proprietary and external sources. ::: :::: @@ -53,25 +53,25 @@ Vendor model compatibility ## LLM classifiers LLM-based classification models via OpenAI-compatible APIs. -- Test suite: `LLMClassifierFullSuite` +- Test suite: [`LLMClassifierFullSuite`](/validmind/validmind/test_suites/llm.qmd#llmclassifierfullsuite) - Includes prompt validation, text data quality, classifier metrics ## NLP classifiers Hugging Face and other NLP classification models. -- Test suite: `NLPClassifierFullSuite` +- Test suite: [`NLPClassifierFullSuite`](/validmind/validmind/test_suites/nlp.qmd#nlpclassifierfullsuite) - Text classification, sentiment analysis, named entity recognition ## Summarization models Models that generate concise summaries from longer texts. -- Test suite: `SummarizationMetrics` +- Test suite: [`SummarizationMetrics`](/validmind/validmind/test_suites/summarization.qmd#summarizationmetrics) - BERT, BLEU, METEOR, ROUGE score evaluation ## Embeddings models Text and vector embedding models. -- Test suite: `EmbeddingsFullSuite` +- Test suite: [`EmbeddingsFullSuite`](/validmind/validmind/test_suites/embeddings.qmd#embeddingsfullsuite) - Similarity metrics, stability analysis, visualization ## RAG systems @@ -87,31 +87,31 @@ Retrieval-augmented generation pipelines. ## Hugging Face-compatible NLP and tabular models using Hugging Face Transformers. -- Test suite: `NLPClassifierFullSuite` (for NLP), `ClassifierFullSuite` or `RegressionFullSuite` (for tabular) +- Test suite: [`NLPClassifierFullSuite`](/validmind/validmind/test_suites/nlp.qmd#nlpclassifierfullsuite) (for NLP), [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) (for tabular) - Text classification, tabular classification/regression ## PyTorch Any PyTorch neural network architecture, including LSTM, RNN, CNN, transformers, and custom architectures. -- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` (depending on task) +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) (depending on task) - Model performance, feature importance, diagnosis ## Tree-based Gradient boosting and ensemble tree methods (XGBoost, CatBoost, random forest). -- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) - Confusion matrix, ROC/AUC, feature importance, SHAP values ## K-nearest neighbors (KNN) Distance-based classification and regression. -- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) - Performance metrics, model comparison ## Clustering Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hierarchical clustering. -- Test suite: `ClusterFullSuite` +- Test suite: [`ClusterFullSuite`](/validmind/validmind/test_suites/cluster.qmd#clusterfullsuite) - Silhouette score, homogeneity, completeness, cluster distribution ::: @@ -122,19 +122,19 @@ Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hier ## Linear regression Models the relationship between a scalar response and one or more explanatory variables. -- Test suite: `RegressionFullSuite` +- Test suite: [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) - R² score, regression errors, feature importance ## Logistic regression Models the probability of a binary outcome based on one or more predictor variables. -- Test suite: `ClassifierFullSuite` +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) - Confusion matrix, ROC/AUC, precision/recall, feature importance -## Time series and forecasting +## Timeseries and forecasting ARIMA, VAR, and other statsmodels time series models. -- Test suite: `TimeSeriesModelValidation` +- Test suite: [`TimeSeriesModelValidation`](/validmind/validmind/test_suites/time_series.qmd#timeseriesmodelvalidation) - Stationarity tests, autocorrelation analysis, forecast evaluation ::: From 8ff7c5e0b60959be34e869d68c5b64edd44af21a Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 12 Mar 2026 17:06:02 -0700 Subject: [PATCH 27/30] Update site/developer/supported-models-and-frameworks.qmd Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com> --- site/developer/supported-models-and-frameworks.qmd | 1 + 1 file changed, 1 insertion(+) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 029d4ff21b..0dfd82d7cb 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -46,6 +46,7 @@ Vendor model compatibility ::: :::: +::: ## AI systems From 0c433168fc838dd18eb8a348f3d85b04e8ffadf7 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Fri, 13 Mar 2026 10:21:22 -0700 Subject: [PATCH 28/30] Whitespace --- site/developer/supported-models-and-frameworks.qmd | 1 + 1 file changed, 1 insertion(+) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 0dfd82d7cb..d8e05e2586 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -46,6 +46,7 @@ Vendor model compatibility ::: :::: + ::: ## AI systems From 1794ea4c72d473ad282b037abae4dd951bd78984 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Tue, 17 Mar 2026 12:06:31 -0700 Subject: [PATCH 29/30] Address review comment from Juan --- .../supported-models-and-frameworks.qmd | 73 +++++++++++++------ 1 file changed, 52 insertions(+), 21 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index d8e05e2586..8ea102692a 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -23,7 +23,7 @@ listing: - samples-jupyter-notebooks.qmd --- -Learn about the wide range of model frameworks for testing and documentation supported by the {{< var validmind.developer >}}. Understand which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. +Learn about the wide range of models and frameworks for testing and documentation supported by the {{< var validmind.developer >}}. Understand which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. ::: {.attn} @@ -341,31 +341,38 @@ vm_model = vm.init_model( ### PipelineModel -Chain multiple models or processing steps: +Chain multiple models or processing steps using the `|` operator between `VMModel` instances: ```python from validmind.models import PipelineModel vm_model = vm.init_model( - model=PipelineModel(models=[preprocessor, model]), + model=PipelineModel(vm_preprocessor | vm_model), input_id="my_pipeline" ) ``` +Or, pass the pipeline directly: + +```python +vm_model = vm.init_model(vm_preprocessor | vm_model, input_id="my_pipeline") +``` + ### MetadataModel -Use when you have model metadata but no inference capability — for example, when documenting a model deployed in an external system: +Use when you have model metadata but no inference capability — for example, when documenting a model deployed in an external system. Calling `vm.init_model()` with `attributes` creates a `MetadataModel` implicitly: ```python -from validmind.models import MetadataModel - vm_model = vm.init_model( - model=MetadataModel(), - input_id="external_model" + input_id="external_model", + attributes={ + "architecture": "ExternalModel", + "language": "Python", + }, ) ``` -With `MetadataModel`, you can still run dataset-only tests and use precomputed predictions. +With a metadata-only model, you can still run dataset-only tests and use precomputed predictions. ## Setting up for LLMs and GenAI @@ -373,23 +380,33 @@ This section covers how to configure and test large language models and generati ### FoundationModel -Use `FoundationModel` for LLM endpoints: +Use `FoundationModel` for LLM endpoints which wraps a callable `predict_fn` and a `Prompt`: ```python -from validmind.models import FoundationModel -from validmind.prompt import Prompt - -prompt = Prompt( - template="Classify the sentiment: {text}", - variables=["text"] -) +import validmind as vm +from openai import OpenAI +from validmind.models import FoundationModel, Prompt + +client = OpenAI() +def call_model(prompt: str) -> str: + return ( + client.chat.completions.create( + model="gpt-4", + messages=[{"role": "user", "content": prompt}], + ) + .choices[0] + .message.content + ) vm_model = vm.init_model( model=FoundationModel( - prompt=prompt, - model="gpt-4", # or your model endpoint + predict_fn=call_model, + prompt=Prompt( + template="Classify the sentiment: {text}", + variables=["text"], + ), ), - input_id="sentiment_classifier" + input_id="sentiment_classifier", ) ``` ::: @@ -418,7 +435,7 @@ For retrieval-augmented generation (RAG) systems, the {{< var validmind.develope ### Dataset requirements -RAG evaluation requires specific fields in your dataset: +RAG evaluation expects these logical fields, but you can map your own column names using test parameters: | Field | Type | Description | |-------|------|-------------| @@ -427,6 +444,20 @@ RAG evaluation requires specific fields in your dataset: | `retrieved_contexts` | List[str] | Retrieved context chunks | | `reference` | str | Ground truth (required for some tests) | +To map custom column names, pass them as parameters when running the test: + +```python +run_test( + "validmind.model_validation.ragas.Faithfulness", + inputs={"dataset": vm_test_ds}, + params={ + "user_input_column": "question", + "response_column": "rag_model_prediction", + "retrieved_contexts_column": "retrieval_model_prediction", + }, +).log() +``` + ### Available RAG tests - **Faithfulness** — Measures how well the response is grounded in retrieved contexts From 8e531961adae318ed46990a6c714c889ffab2b91 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Tue, 17 Mar 2026 14:27:15 -0700 Subject: [PATCH 30/30] Remove links I'd added --- .../supported-models-and-frameworks.qmd | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd index 8ea102692a..d1b25d81af 100644 --- a/site/developer/supported-models-and-frameworks.qmd +++ b/site/developer/supported-models-and-frameworks.qmd @@ -55,25 +55,25 @@ Vendor model compatibility ## LLM classifiers LLM-based classification models via OpenAI-compatible APIs. -- Test suite: [`LLMClassifierFullSuite`](/validmind/validmind/test_suites/llm.qmd#llmclassifierfullsuite) +- Test suite: `LLMClassifierFullSuite` - Includes prompt validation, text data quality, classifier metrics ## NLP classifiers Hugging Face and other NLP classification models. -- Test suite: [`NLPClassifierFullSuite`](/validmind/validmind/test_suites/nlp.qmd#nlpclassifierfullsuite) +- Test suite: `NLPClassifierFullSuite` - Text classification, sentiment analysis, named entity recognition ## Summarization models Models that generate concise summaries from longer texts. -- Test suite: [`SummarizationMetrics`](/validmind/validmind/test_suites/summarization.qmd#summarizationmetrics) +- Test suite: `SummarizationMetrics` - BERT, BLEU, METEOR, ROUGE score evaluation ## Embeddings models Text and vector embedding models. -- Test suite: [`EmbeddingsFullSuite`](/validmind/validmind/test_suites/embeddings.qmd#embeddingsfullsuite) +- Test suite: `EmbeddingsFullSuite` - Similarity metrics, stability analysis, visualization ## RAG systems @@ -89,31 +89,31 @@ Retrieval-augmented generation pipelines. ## Hugging Face-compatible NLP and tabular models using Hugging Face Transformers. -- Test suite: [`NLPClassifierFullSuite`](/validmind/validmind/test_suites/nlp.qmd#nlpclassifierfullsuite) (for NLP), [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) (for tabular) +- Test suite: `NLPClassifierFullSuite` (for NLP), `ClassifierFullSuite` or `RegressionFullSuite` (for tabular) - Text classification, tabular classification/regression ## PyTorch Any PyTorch neural network architecture, including LSTM, RNN, CNN, transformers, and custom architectures. -- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) (depending on task) +- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` (depending on task) - Model performance, feature importance, diagnosis ## Tree-based Gradient boosting and ensemble tree methods (XGBoost, CatBoost, random forest). -- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) +- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` - Confusion matrix, ROC/AUC, feature importance, SHAP values ## K-nearest neighbors (KNN) Distance-based classification and regression. -- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) +- Test suite: `ClassifierFullSuite` or `RegressionFullSuite` - Performance metrics, model comparison ## Clustering Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hierarchical clustering. -- Test suite: [`ClusterFullSuite`](/validmind/validmind/test_suites/cluster.qmd#clusterfullsuite) +- Test suite: `ClusterFullSuite` - Silhouette score, homogeneity, completeness, cluster distribution ::: @@ -124,19 +124,19 @@ Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hier ## Linear regression Models the relationship between a scalar response and one or more explanatory variables. -- Test suite: [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) +- Test suite: `RegressionFullSuite` - R² score, regression errors, feature importance ## Logistic regression Models the probability of a binary outcome based on one or more predictor variables. -- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) +- Test suite: `ClassifierFullSuite` - Confusion matrix, ROC/AUC, precision/recall, feature importance ## Timeseries and forecasting ARIMA, VAR, and other statsmodels time series models. -- Test suite: [`TimeSeriesModelValidation`](/validmind/validmind/test_suites/time_series.qmd#timeseriesmodelvalidation) +- Test suite: `TimeSeriesModelValidation` - Stationarity tests, autocorrelation analysis, forecast evaluation :::