Optimize experiment pagination query performance

[zackcl]

Resolves #2859

FYI, I asked Copilot for input while investigating this issue, and it suggested removing database JOINs that aren’t required for the experiment root table. Applying this change significantly improved performance in my testing. Since I’m less familiar with this area of the codebase, I’d appreciate a careful review to confirm the changes are safe and correct.

---

Problem

The /api/experiments/paginated endpoint was experiencing severe performance degradation:

• Query execution time: 2815ms (backend logs)
• Total API response time: 3.34s in production
• 10x slower than legacy implementation (334ms)

Root cause: The findPaginated method was performing 20 LEFT JOINs to eagerly load deeply nested relationships that were not required for the experiment list table display.

Solution

1. Query Optimization (ExperimentService.ts)

Reduced database JOINs from 20 to 3 by loading only essential data for table rendering:

Removed unnecessary JOINs:

• queries, factors, levels, metric - not displayed in table
• experimentSegmentInclusion/Exclusion with nested segments - only needed in detail view
• levelCombinationElements, level - factorial experiment details
• conditionPayloads - runtime assignment data

Retained required JOINs:

• conditions - basic experiment structure
• partitions - needed for app context display and search functionality
• stratificationFactor - needed for post-experiment rule display

Additional improvements:

• Removed unnecessary data transformations (formattingConditionPayload, formattingPayload, reducedConditionPayload, inferListTypesForExperimentListForExperimentRedesignDataChange)

2. Database Indexing

Added strategic indexes to improve JOIN and filtering performance:

Experiment table:

• idx_experiment_updated_at - optimizes default sorting
• idx_experiment_state - optimizes state-based filtering
• idx_experiment_name - optimizes name search

Foreign key indexes:

• idx_decision_point_experiment - optimizes partition JOIN
• idx_experiment_condition_experiment - optimizes condition JOIN

Performance Impact

Expected improvement: 7-14x faster

• Query time: 2815ms → 200-400ms (estimated)
• API response: 3.34s → 300-500ms (estimated)

The paginated experiment list now loads only the minimal data required for display, with detailed information fetched separately when viewing individual experiments.

Testing

• Verify experiment list loads correctly
• Confirm search functionality works (name, status, tag, context)
• Check sorting by name, status, updatedAt
• Validate that experiment detail page still loads complete data
• Monitor backend logs for query execution time

Breaking Changes

None. The API response structure remains unchanged; we simply load less nested data that was never used by the frontend table component.

[Copilot]

Pull request overview

This PR optimizes the /api/experiments/paginated endpoint by reducing unnecessary database JOINs and adding strategic indexes, addressing severe performance degradation (3.34s response time down to an estimated 300-500ms).

Changes:

• Reduced database JOINs from 20 to 3 in the findPaginated method by removing eager loading of nested relationships not required for the experiment list view
• Added database indexes on frequently queried columns (updatedAt, state, name) and foreign keys
• Removed unnecessary data transformation methods that processed unused nested data

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
backend/packages/Upgrade/src/api/services/ExperimentService.ts	Removed 17 unnecessary LEFT JOINs and data transformations from the paginated query
backend/packages/Upgrade/src/api/models/Experiment.ts	Added indexes for updatedAt, state, and name columns to optimize sorting and filtering
backend/packages/Upgrade/src/api/models/ExperimentCondition.ts	Added index on experiment foreign key to optimize JOIN performance
backend/packages/Upgrade/src/api/models/DecisionPoint.ts	Added index on experiment foreign key to optimize JOIN performance

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bcb37]

This makes sense to me if there's really no need for that joined data in the frontend.
However, the indexes that were added to the typeorm models are to not added to the database until a migration is created and run. Since there's no migration file in this pr, the indexes were probably not added. Maybe they're not necessary if we're already seeing the performance improvements without them.

[zackcl]

I will open another PR later that merges into dev branch, since we decided to merge experiment-design-refresh into dev first.

Feel free to leave comments in the issue #2859, if anyone has any suggestions.