Match search operators against assetsSummary instead of per-asset metadata

[bendichter]

Problem

The advanced search operators released recently are slow. Timed against production:

query	time
species:mouse created_after:2026-01-01	14.7s
species:mouse	13.4s
created_after:2026-01-01	0.26s
mouse (free text)	0.63s

All of the cost is in species:, approach:, and technique:. Those matched per-asset metadata in the asset_search materialized view with jsonb_path_exists and a jsonpath built at runtime. A runtime-built jsonpath regex cannot use any of the view's GIN indexes, so each query sequential-scanned one row per asset and evaluated a regex against every row's metadata blob. The visible_to() embargo check added a correlated subquery per row on top of that. A comment in filters.py had flagged this risk when the operators were written.

Change

species, approach, and technique now match the version-level assetsSummary aggregation, which already rolls up exactly these three fields. The scan moves from one row per asset to one row per dandiset version, three to four orders of magnitude fewer rows. This is the same source the standard: operator was pointed at in a232b1f for a related reason.

While in here:

• Re-added standard:, which was removed in 05fda3c pending a decision on the right metadata source. That decision is now made, and it matches assetsSummary.dataStandard.
• Removed file_type:. Its remaining value is covered by standard:, and dropping it takes AssetSearch out of this code path entirely. Note this is only the search operator; the faceted-search file_type query parameter on DandisetSearchQueryParameterSerializer is a separate feature and is untouched.
• apply_search_filters no longer takes the requesting user. Visibility is enforced by the view's base queryset (get_visible_dandisets), and these filters only ever narrow it, so the per-row embargo subquery was redundant. The embargo test is retained and still passes.

One consequence to be aware of: assetsSummary is recomputed during metadata aggregation rather than on every asset write, so results can lag slightly for a dandiset whose assets changed very recently.

I have not re-measured latency against production data, since that requires the change to be deployed. The structural argument is the row-count reduction described above.

🤖 Generated with Claude Code

[bendichter]

demo on local seed data:

Screen.Recording.2026-08-05.at.3.33.30.PM.mov

[bendichter]

Added a variable: operator over assetsSummary.variableMeasured in 86dffab.

I surveyed the archive before picking this one. Of the fields left in assetsSummary, variableMeasured is populated on about 77% of dandisets and carries 22 distinct values in my sample, led by ProcessingModule (39), ElectrodeGroup (20), OpticalChannel (17), ImagingPlane (17), Units (15), and ElectricalSeries (14), with SpatialSeries, LFP, PupilTracking, and EyeTracking further down. It answers a genuinely different question from the operators we already have: approach: and technique: describe how the experiment was run, while this describes what is actually in the files, so "which dandisets contain sorted units" or "which have eye tracking" becomes answerable.

The one structural difference is that variableMeasured is an array of bare strings rather than objects with a name, so its jsonpath selects the elements themselves. The lookup table now holds full paths and _jsonpath_name_match became _jsonpath_match, since it no longer always matches a name.

On naming: I went with variable: rather than variable_measured: to match the existing convention, where technique: comes from measurementTechnique and standard: from dataStandard. Underscores show up only in the date operators, where the suffix is doing real work distinguishing before from after. Happy to rename if you would rather be explicit; it is a one-line change in the parser plus the frontend list.

For the record, the fields I looked at and skipped: numberOfSamples is populated on 8% of dandisets and numberOfCells on none of the 100 I sampled, so operators over either would return almost nothing. numberOfBytes and numberOfFiles are always present, but a size filter already exists in the faceted sidebar. numberOfSubjects is populated on 93% and is worth having, but it is a threshold operator rather than a substring match and is already written up as num_subjects: on the unmerged advanced-search-counts branch, so it belongs in its own PR.