Explain in 1-2 sentences what the dataset contains and what it is used for.
\n", + "\n", + "Summarize the dataset structure, target outcome, and the main types of input features in 2-3 sentences.
\n", + "\n", + "End with a short balanced conclusion on the dataset's suitability for model development and evaluation.
\n", + "\"\"\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fbf10ad9", + "metadata": {}, + "outputs": [], + "source": [ + "vm.run_text_generation(\n", + " content_id=\"dataset_summary_text\",\n", + " prompt=prompt,\n", + ").log()" + ] + }, + { + "cell_type": "markdown", + "id": "99a0740e", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Pass section-specific context\n", + "\n", + "Then, we’ll control the `context` used for generation by passing a selected set of content IDs to `vm.run_text_generation()`. Rather than relying on the full document, this lets you focus the model on the most relevant parts of the documentation for a given text block. In practice, that means you can generate more targeted qualitative content by choosing which existing test and text blocks should inform the output." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "43cf0e7d", + "metadata": {}, + "outputs": [], + "source": [ + "vm.get_content_ids(\"data_description\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1e1a919e", + "metadata": {}, + "outputs": [], + "source": [ + "vm.run_text_generation(\n", + " content_id=\"dataset_summary_text\",\n", + " context={\"content_ids\": vm.get_content_ids(\"data_description\")},\n", + ").log()" + ] + }, + { + "cell_type": "markdown", + "id": "701a0323", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Append a new text block to a section\n", + "\n", + "Sometimes you may want to generate text for a `content_id` that is not already defined in the template. In that case, you can still generate the text with `vm.run_text_generation()` and then use `.log(section_id=...)` to tell ValidMind where that new text block should be placed in the document. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6a9ba924", + "metadata": {}, + "outputs": [], + "source": [ + "vm.run_text_generation(\n", + " content_id=\"intended_use\",\n", + " section_id=\"intended_use\",\n", + ").log()" + ] + }, + { + "cell_type": "markdown", + "id": "6e032b79", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Generate text across the document\n", + "\n", + "At this stage, instead of generating one block at a time, we can populate multiple qualitative sections in a single pass.\n", + "\n", + "The [`vm.generate_documentation_text`](https://docs.validmind.ai/validmind/validmind.html#generate_documentation_text) function reads a configuration dictionary, generates content for each target block, logs the generated text to the ValidMind Platform, and returns a notebook summary grouped by section.\n", + "\n", + "- The function uses a `config` argument to describe which text blocks to generate and how each one should be handled.\n", + "- The `config` parameter is a dictionary with the following structure:\n", + "\n", + " ```python\n", + " config = {\n", + " \"Generated text
", + ) + def test_run_text_generation(self, mock_generate_text): + result = run_text_generation( + content_id="dataset_summary_text", + prompt="Summarize the dataset.", + section_id="data_description", + context={"content_ids": ["train_dataset"]}, + show=False, + ) + + self.assertIsInstance(result, TextGenerationResult) + self.assertEqual(result.content_id, "dataset_summary_text") + self.assertEqual(result.prompt, "Summarize the dataset.") + self.assertEqual(result.section_id, "data_description") + self.assertEqual(result.context, {"content_ids": ["train_dataset"]}) + self.assertEqual(result.description, "Generated text
") + self.assertTrue(result._was_description_generated) + self.assertIn("validmind", result.metadata) + self.assertIn("timestamp", result.metadata) + self.assertIn("duration_seconds", result.metadata) + mock_generate_text.assert_called_once_with( + "dataset_summary_text", + "Summarize the dataset.", + {"content_ids": ["train_dataset"]}, + section_id="data_description", + ) + + +class TestGenerateDocumentationText(TestCase): + @mock.patch( + "validmind.client_config.client_config.documentation_template", + MockedConfig.documentation_template, + ) + @mock.patch("validmind.client.vm_display") + @mock.patch("validmind.client.DocumentationTextSummary") + @mock.patch("validmind.client.HTMLLabel") + @mock.patch("validmind.client.HTMLProgressBar") + @mock.patch("validmind.client.run_text_generation") + def test_generate_documentation_text( + self, + mock_run_text_generation, + mock_progress_bar, + mock_label, + mock_summary, + mock_display, + ): + first_result = mock.Mock() + second_result = mock.Mock() + mock_run_text_generation.side_effect = [first_result, second_result] + + progress_instance = mock_progress_bar.return_value + progress_instance.value = 0 + progress_instance.update.side_effect = ( + lambda value, description=None: setattr(progress_instance, "value", value) + ) + + config = { + "monitoring_plan": { + "prompt": "Write the monitoring plan.", + "section_id": "monitoring", + "context": {"content_ids": ["model_summary"]}, + }, + "governance_plan": { + "prompt": "Write the governance plan.", + }, + } + + results = generate_documentation_text(config) + + self.assertEqual( + results, + { + "monitoring_plan": first_result, + "governance_plan": second_result, + }, + ) + self.assertEqual(mock_run_text_generation.call_count, 2) + mock_run_text_generation.assert_has_calls( + [ + mock.call( + content_id="monitoring_plan", + prompt="Write the monitoring plan.", + section_id="monitoring", + context={"content_ids": ["model_summary"]}, + show=False, + ), + mock.call( + content_id="governance_plan", + prompt="Write the governance plan.", + section_id=None, + context=None, + show=False, + ), + ] + ) + first_result.log.assert_called_once_with() + second_result.log.assert_called_once_with() + mock_label.assert_called_once_with(value="Generating documentation text...") + mock_progress_bar.assert_called_once_with( + max_value=4, + description="Generating documentation text...", + ) + progress_instance.display.assert_called_once_with() + progress_instance.close.assert_called_once_with() + mock_label.return_value.update.assert_any_call( + "Sending result to ValidMind: monitoring_plan..." + ) + mock_label.return_value.update.assert_any_call( + "Sending result to ValidMind: governance_plan..." + ) + mock_label.return_value.update.assert_any_call( + "Documentation text generation complete!" + ) + mock_summary.assert_called_once_with( + title="Binary classification", + description="Template for binary classification models.", + results={ + "monitoring_plan": first_result, + "governance_plan": second_result, + }, + template_sections=MockedConfig.documentation_template["sections"], + ) + mock_display.assert_called_once_with(mock_summary.return_value) + + # TODO: Fix this test # class TestPreviewTemplate(TestCase): # @mock.patch( diff --git a/tests/test_results.py b/tests/test_results.py index 0eafb0679..87b74c3c0 100644 --- a/tests/test_results.py +++ b/tests/test_results.py @@ -13,6 +13,10 @@ ResultTable, RawData, ) +from validmind.vm_models.result.utils import ( + AI_REVISION_NAME, + DEFAULT_REVISION_NAME, +) from validmind.vm_models.figure import Figure from validmind.errors import InvalidParameterError @@ -181,11 +185,141 @@ def test_text_generation_result(self): self.assertEqual(text_result.name, "Text Generation Result") self.assertEqual(text_result.title, "Text Test") self.assertEqual(text_result.description, "Generated text") + self.assertIsNone(text_result.doc) + self.assertIsNone(text_result.test_name) html = text_result.to_html() self.assertIsInstance(html, str) self.assertIn("Generated text", html) + @patch("validmind.vm_models.result.result.api_client.alog_text") + async def test_text_generation_result_log_async(self, mock_log_text): + """Test async logging of TextGenerationResult through alog_text""" + text_result = TextGenerationResult( + result_id="text_1", + content_id="dataset_summary_text", + description="Generated text", + ) + text_result._client_config_cache = type( + "MockConfig", + (), + { + "documentation_template": { + "sections": [ + { + "id": "data_description", + "contents": [ + { + "content_id": "dataset_summary_text", + "content_type": "text", + } + ], + } + ] + } + }, + )() + + await text_result.log_async() + + mock_log_text.assert_called_once_with( + content_id=f"dataset_summary_text::{DEFAULT_REVISION_NAME}", + text="Generated text", + section_id=None, + ) + + @patch("validmind.vm_models.result.result.api_client.alog_text") + async def test_text_generation_result_log_async_with_section_id(self, mock_log_text): + """Test async logging of TextGenerationResult forwards section_id""" + text_result = TextGenerationResult( + result_id="text_1", + content_id="intended_use_text", + description="Generated text", + section_id="intended_use", + ) + text_result._client_config_cache = type( + "MockConfig", + (), + {"documentation_template": {"sections": [{"id": "intended_use"}]}}, + )() + + await text_result.log_async() + + mock_log_text.assert_called_once_with( + content_id=f"intended_use_text::{DEFAULT_REVISION_NAME}", + text="Generated text", + section_id="intended_use", + ) + + @patch("validmind.vm_models.result.result.api_client.alog_text") + async def test_text_generation_result_log_async_uses_ai_revision_name( + self, mock_log_text + ): + """Test generated text logs with the AI revision name""" + text_result = TextGenerationResult( + result_id="text_1", + content_id="dataset_summary_text", + description="Generated text", + _was_description_generated=True, + ) + text_result._client_config_cache = type( + "MockConfig", + (), + { + "documentation_template": { + "sections": [ + { + "id": "data_description", + "contents": [ + { + "content_id": "dataset_summary_text", + "content_type": "text", + } + ], + } + ] + } + }, + )() + + await text_result.log_async() + + mock_log_text.assert_called_once_with( + content_id=f"dataset_summary_text::{AI_REVISION_NAME}", + text="Generated text", + section_id=None, + ) + + async def test_text_generation_result_log_async_requires_section_id_for_new_block(self): + """Test new generated text requires a section_id for placement""" + text_result = TextGenerationResult( + result_id="text_1", + content_id="model_overview_text", + description="Generated text", + ) + text_result._client_config_cache = type( + "MockConfig", + (), + {"documentation_template": {"sections": [{"id": "existing_section"}]}}, + )() + + with self.assertRaisesRegex( + ValueError, "New generated content requires `section_id` for placement" + ): + await text_result.log_async() + + async def test_text_generation_result_log_async_requires_content_id(self): + """Test TextGenerationResult requires a content_id when logging""" + text_result = TextGenerationResult( + result_id="text_1", + description="Generated text", + ) + + with self.assertRaisesRegex( + ValueError, "`content_id` must be provided to log generated text" + ): + await text_result.log_async() + def test_validate_log_config(self): """Test validation of log configuration""" test_result = TestResult(result_id="test_1") diff --git a/tests/unit_tests/test_documentation_text_summary.py b/tests/unit_tests/test_documentation_text_summary.py new file mode 100644 index 000000000..fcf117f0e --- /dev/null +++ b/tests/unit_tests/test_documentation_text_summary.py @@ -0,0 +1,62 @@ +"""Tests for documentation text summary rendering.""" + +from unittest import mock + +from validmind.vm_models.text_generation_summary import DocumentationTextSummary + + +def test_documentation_text_summary_renders_link_and_accordion(): + monitoring_result = mock.Mock() + monitoring_result.section_id = "monitoring_details" + monitoring_result.to_html.return_value = "