Add: AI-assisted payload generation to the Ability Explorer - Test Ability Screen by Arkenon · Pull Request #695 · WordPress/ai

Arkenon · 2026-06-10T11:07:58Z

What?

Adds AI-assisted payload generation to the Ability Test Runner. Users can describe a test scenario in natural language and automatically generate a valid JSON payload based on the selected ability's input schema.

Why?

Some abilities expose complex or deeply nested input schemas that are difficult to populate manually. Creating valid test payloads often requires understanding schema structures, enum values, nested objects, and validation constraints.

This makes testing abilities slower and more error-prone, especially for contributors who are unfamiliar with a specific ability's schema.

How?

ability_test_generate_with_ai.mp4

This PR introduces a new Generate with AI workflow in the Ability Test Runner:

Added a Generate with AI button to the test interface.
Added a modal that accepts a natural-language description of the desired test input.
Sends the ability's input schema and the user's description to the configured AI provider using wp_ai_client_prompt().
Parses the AI response, removes markdown code fencing if present, validates the JSON, and inserts the generated payload into the test input field.
The feature is only available when:
- An AI provider is configured.
- The selected ability exposes an input schema.

Implementation Details

Admin_Page.php
- Added ajax_generate_payload() AJAX handler.
- Retrieves the ability schema and user prompt.
- Calls the configured AI provider via wp_ai_client_prompt().
- Sanitizes and parses the response before returning JSON.
Abilities_Explorer.php
- Added generateNonce (ai_ability_explorer_generate_payload) for AJAX authentication.
GeneratePayloadModal.jsx
- New modal component for collecting user instructions and displaying errors.
index.js
- Added modal mounting/unmounting logic and integration with the Ability Test Runner UI.
index.scss
- Added styling for the modal footer and error notice area.

Use of AI Tools

AI assistance: Yes

Tool(s): ChatGPT

Model(s): GPT-5.5

Used for: Brainstorming the feature design, refining prompts, and drafting portions of documentation. Final implementation, review, testing, and code decisions were completed by me.

Testing Instructions

Navigate to Tools → Abilities Explorer.
Open an ability that exposes a non-trivial input schema.
Click Test Ability.
Click Generate with AI.
Enter a natural-language instruction such as:
- "Use the default values"
- "Search for posts with the title 'Hello'"
- "Query only the site URL information"
Click Generate.
Verify that the payload textarea is populated with valid JSON matching the schema.
Click Invoke Ability.
Verify that the ability executes successfully with the generated payload.

Note: An AI provider must be connected in the plugin settings for this feature to be available.

Screenshots or screencast

Before | After -- | -- N/A | Generate with AI button and payload generation modal

Changelog Entry

Added - AI-assisted payload generation for the Ability Test Runner.

…ility page.

github-actions · 2026-06-10T11:08:12Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: Arkenon <arkenon@git.wordpress.org>
Co-authored-by: dkotter <dkotter@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Arkenon · 2026-06-19T08:51:51Z

Any feedback or suggestions on this PR?

codecov · 2026-06-23T15:11:34Z

Codecov Report

❌ Patch coverage is 0.96154% with 103 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.66%. Comparing base (26ce3f0) to head (82cac3e).

Files with missing lines	Patch %	Lines
...udes/Experiments/Abilities_Explorer/Admin_Page.php	1.19%	83 Missing ⚠️
...eriments/Abilities_Explorer/Abilities_Explorer.php	0.00%	20 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #695      +/-   ##
=============================================
- Coverage      76.41%   75.66%   -0.75%     
- Complexity      1828     1840      +12     
=============================================
  Files             87       87              
  Lines           7764     7842      +78     
=============================================
+ Hits            5933     5934       +1     
- Misses          1831     1908      +77

Flag	Coverage Δ
unit	`75.66% <0.96%> (-0.75%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dkotter

I've not tested yet but a number of things that need cleaned up here. In addition, would be great to get test coverage here, unit tests potentially but definitely E2E tests.

While the overall idea is neat here, the overall flow I think could be improved a bit. For instance, the modal itself has a title of Generate Payload with AI but then in the textarea, you add a prompt that says Generate payload for ....

We've been trying to remove the use of with AI, so I think just Generate Payload both in the modal and the button that triggers the modal will be better. I'm also not sure about requiring a user to enter details about their desired payload. Could the generate function not just generate it's own payload without user input? I understand that gets trickier if a post is required, for example, but seems solvable. Just seems to make it less user friendly to click a generate button but then be required to manually enter in a bunch of information first

dkotter · 2026-06-23T15:06:58Z

+	/**
+	 * AJAX handler for AI-assisted payload generation.
+	 *
+	 * @since 1.0.1


All @since statements should be set to x.x.x

dkotter · 2026-06-23T15:07:10Z

+						<ol>
+							<li><?php esc_html_e( 'Edit the JSON input below with your test data', 'ai' ); ?></li>
+							<li><?php esc_html_e( 'Click "Validate Input" to check your JSON is correct', 'ai' ); ?></li>
+							<li><?php esc_html_e( 'Click "Invoke Ability" to execute the ability with your input', 'ai' ); ?></li>
+							<li><?php esc_html_e( 'View the results below', 'ai' ); ?></li>
+						</ol>


Why was spacing changed here?

dkotter · 2026-06-23T15:08:00Z

@@ -1,4 +1,5 @@
 <?php
+


There's quite a few of these spacing changes in this PR. Any particular reason for those? If not, reverting those keeps the diff here cleaner and easier to review

dkotter · 2026-06-23T15:08:43Z

+			);
+		}
+
+		if ( ! function_exists( 'wp_ai_client_prompt' ) ) {


This function should always exist as WP 7.0 is our minimum

dkotter · 2026-06-23T15:10:00Z

+		if ( ! $prompt_builder->is_supported_for_text_generation() ) {
+			wp_send_json_error(
+				array(
+					'message' => __( 'No AI provider available. Please connect one in the plugin settings.', 'ai' ),


Suggested change

'message' => __( 'No AI provider available. Please connect one in the plugin settings.', 'ai' ),

'message' => __( 'Generation failed. Please ensure you have a connected provider that supports text generation.', 'ai' ),

dkotter · 2026-06-23T15:11:13Z

 				'title'   => __( 'Overview', 'ai' ),
 				'content' =>
-					'<p>' . esc_html__( 'Abilities are a standardized way for WordPress core, plugins, and themes to expose discrete units of functionality. Each ability has a name, optional input/output schemas, and can be invoked programmatically.', 'ai' ) . '</p>' .
+				'<p>' . esc_html__( 'Abilities are a standardized way for WordPress core, plugins, and themes to expose discrete units of functionality. Each ability has a name, optional input/output schemas, and can be invoked programmatically.', 'ai' ) . '</p>' .


Another spacing removal

dkotter · 2026-06-23T15:11:19Z

+				'<p>' . esc_html__( 'Every ability is associated with a provider that indicates where it comes from:', 'ai' ) . '</p>' .
 					'<ul>' .
-						'<li>' . wp_kses( __( '<strong>Core</strong>: Built into WordPress itself.', 'ai' ), $provider_tags ) . '</li>' .
-						'<li>' . wp_kses( __( '<strong>Plugin</strong>: Registered by an active plugin.', 'ai' ), $provider_tags ) . '</li>' .
-						'<li>' . wp_kses( __( '<strong>Theme</strong>: Registered by the active theme.', 'ai' ), $provider_tags ) . '</li>' .
+					'<li>' . wp_kses( __( '<strong>Core</strong>: Built into WordPress itself.', 'ai' ), $provider_tags ) . '</li>' .
+					'<li>' . wp_kses( __( '<strong>Plugin</strong>: Registered by an active plugin.', 'ai' ), $provider_tags ) . '</li>' .
+					'<li>' . wp_kses( __( '<strong>Theme</strong>: Registered by the active theme.', 'ai' ), $provider_tags ) . '</li>' .


Another spacing removal

dkotter · 2026-06-23T15:11:25Z

+				'<p>' . esc_html__( 'You can test any ability directly from this screen:', 'ai' ) . '</p>' .
 					'<ol>' .
-						'<li>' . __( 'Click "Test" next to an ability in the list.', 'ai' ) . '</li>' .
-						'<li>' . __( 'Edit the pre-filled Input Data if the ability accepts JSON parameters.', 'ai' ) . '</li>' .
-						'<li>' . __( 'Use "Validate Input" to check your JSON against the schema.', 'ai' ) . '</li>' .
-						'<li>' . __( 'Click "Invoke Ability" to execute it and see the result.', 'ai' ) . '</li>' .
+					'<li>' . __( 'Click "Test" next to an ability in the list.', 'ai' ) . '</li>' .
+					'<li>' . __( 'Edit the pre-filled Input Data if the ability accepts JSON parameters.', 'ai' ) . '</li>' .
+					'<li>' . __( 'Use "Validate Input" to check your JSON against the schema.', 'ai' ) . '</li>' .
+					'<li>' . __( 'Click "Invoke Ability" to execute it and see the result.', 'ai' ) . '</li>' .


Another spacing removal

dkotter · 2026-06-23T15:11:38Z

 		$screen->set_help_sidebar(
 			'<p><strong>' . esc_html__( 'For more information:', 'ai' ) . '</strong></p>' .
-			'<p><a href="https://developer.wordpress.org/apis/abilities/" target="_blank" rel="noopener noreferrer">' . esc_html__( 'Abilities API Documentation', 'ai' ) . '</a></p>'
+				'<p><a href="https://developer.wordpress.org/apis/abilities/" target="_blank" rel="noopener noreferrer">' . esc_html__( 'Abilities API Documentation', 'ai' ) . '</a></p>'


Spacing addition that can be removed

dkotter · 2026-06-23T15:13:36Z

+					onClose: close,
+					onSuccess: handleSuccess,
+					abilitySlug,
+					strings: aiAbilityExplorer.strings,


We can use the client-side __ functions instead of passing in strings from the server-side

Arkenon and others added 2 commits June 10, 2026 13:39

feat: Add Generate with AI button into the Ability Explorer - Test Ab…

6cc4d73

…ility page.

Merge branch 'WordPress:develop' into develop

32ea4d7

dkotter assigned Arkenon Jun 23, 2026

Merge branch 'develop' into develop

82cac3e

dkotter requested changes Jun 23, 2026

View reviewed changes

dkotter added this to the Future Release milestone Jun 23, 2026

	'message' => __( 'No AI provider available. Please connect one in the plugin settings.', 'ai' ),
	'message' => __( 'Generation failed. Please ensure you have a connected provider that supports text generation.', 'ai' ),

Conversation

Arkenon commented Jun 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Implementation Details

Use of AI Tools

Testing Instructions

Screenshots or screencast

Changelog Entry

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Arkenon commented Jun 19, 2026

Uh oh!

codecov Bot commented Jun 23, 2026

Codecov Report

Uh oh!

dkotter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Arkenon commented Jun 10, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading