diff --git a/README.md b/README.md index ae1d437d..c854f997 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ title: Generic Data Index # Pimcore Generic Data Index -The Pimcore Generic Data Index Bundle provides a centralized way to index and search elements (assets, data objects and documents) in Pimcore via indices (e.g OpenSearch, Elasticsearch). +The Pimcore Generic Data Index Bundle provides a centralized way to index and search elements (assets, data objects and documents) in Pimcore via indices (e.g. OpenSearch, Elasticsearch). It is shipped with the OpenSearch and Elasticsearch clients and provides a central configuration for them in order to be used in other bundles. This bundle can be extended and customized to fit your specific needs, for example if you would like to extend the search indices with custom attributes. diff --git a/doc/01_Installation/02_Upgrade.md b/doc/01_Installation/02_Upgrade.md index 87e2e845..833edf9b 100644 --- a/doc/01_Installation/02_Upgrade.md +++ b/doc/01_Installation/02_Upgrade.md @@ -1,86 +1,142 @@ -# Upgrade Information +--- +title: Upgrade Information +description: Version-specific upgrade instructions and breaking changes for the Generic Data Index bundle. +--- -Following steps are necessary during updating to newer versions. +# Upgrade Information ## Upgrade to 2026.1.0 -- [Searching] Added `forceReload` parameter to element search service interface `byId()`. -- The bundle installer now implements `PostInstallCommandsProviderInterface` from Pimcore's InstallBundle. This means the post-install command `generic-data-index:update:index -r` is automatically executed during `pimcore:install` when using Install Profiles. Manual execution of this command after `pimcore:bundle:install` is still required as before. -- The messenger transport DSN is now configurable via the `%pimcore.messenger.transport_dsn_prefix%` container parameter instead of being hardcoded to `doctrine://default`. This allows the installer to wire the transport DSN from environment variables (e.g. `PIMCORE_MESSENGER_TRANSPORT_DSN_PREFIX`). -- Added support to `PHP` `8.5`. -- Removed support to `PHP` `8.3` and Symfony `v6`. + +### PHP and Dependency Requirements + +- Added support for `PHP` `8.5`. +- Removed support for `PHP` `8.3` and Symfony `v6`. + +### Removed ExtJS / Admin Classic + +- `PimcoreGenericDataIndexBundle` no longer implements `PimcoreBundleAdminClassicInterface`. +- Removed `BundleAdminClassicTrait`. + +### Interface Changes + +- Added optional `bool $forceReload = false` parameter to the `byId()` method on the following interfaces: + - `AssetSearchServiceInterface::byId(int $id, ?User $user = null, bool $forceReload = false)` + - `DataObjectSearchServiceInterface::byId(int $id, ?User $user = null, bool $forceReload = false)` + - `DocumentSearchServiceInterface::byId(int $id, ?User $user = null, bool $forceReload = false)` + - `ElementSearchServiceInterface::byId(ElementType $elementType, int $id, ?User $user = null, bool $forceReload = false)` + +### Installer / Messenger Transport Changes + +- The bundle installer now implements `PostInstallCommandsProviderInterface`. The post-install command + `generic-data-index:update:index -r` is automatically executed during `pimcore:install` when using + Install Profiles. Manual execution after `pimcore:bundle:install` is still required. +- The messenger transport DSN is now configurable via the `%pimcore.messenger.transport_dsn_prefix%` + container parameter (env: `PIMCORE_MESSENGER_TRANSPORT_DSN_PREFIX`) instead of being hardcoded to + `doctrine://default`. ## Upgrade to 2.2.0 -- [Indexing] Added `id` column as new primary key to `generic_data_index_queue`. Please make sure to execute migrations. -- [Searching] Added `trackTotalHits` parameter to `DefaultSearchService` and `SearchExecutionService`. The default value is true, which means that total hits will always be computed accurately, even if they exceed the search engines threshold for accurate hit calculation. Change this parameter to `null`, to use the default threshold, pass an integer value to set a specific one. + +- **[Indexing]** Added `id` column as new primary key to `generic_data_index_queue`. + Run migrations after updating. +- **[Searching]** Added `trackTotalHits` parameter to `DefaultSearchService` and + `SearchExecutionService`. Default value: `true` (always compute accurate total hits, + even beyond the search engine's threshold). Set to `null` to use the engine's default + threshold, or pass an integer for a specific limit. ## Upgrade to 2.1.0 + - Added support for Symfony 7 -- [Indexing] Added sort index for documents -- [Indexing] Improved indexing of field collections to prevent mapping conflicts when properties have the same name but different types -- Execute the following command to reindex all elements to be able to use all new features: - ```bin/console generic-data-index:update:index -r``` +- **[Indexing]** Added sort index for documents +- **[Indexing]** Improved field collection indexing to prevent mapping conflicts when + properties share a name but differ in type +- Reindex all elements to use the new features: + ```bash + bin/console generic-data-index:update:index -r + ``` ## Upgrade to 2.0.0 -- [Indexing] Added inherited fields indicator to data object indexing -- [Indexing] Added functionality to enqueue dependent items -- [Indexing] Added class ID field for data object elements -- [Indexing] Added prefix for index names of data objects, these names changed from e.g. `pimcore_car` to `pimcore_data-object_car`. Old indexes are **not deleted** automatically. You need to delete them manually if necessary. -- [Searching] Added new `ClassIdsFilter` modifier to search for data object elements by class ID or class name -- Added a new method `isElementLocked()` to the `ElementLockService`, which provides functionality to retrieve element locked status based on the index data -- Execute the following command to reindex all elements to be able to use all new features: - ```bin/console generic-data-index:update:index -r``` - -### BC-Breaks -- Removed deprecated alias `generic-data-index.opensearch-client` and replaced it with `generic-data-index.search-client` -- Removed all deprecated classes from OpenSearch namespaces and replaced them with DefaultSearch namespace instead. - - `Pimcore\Bundle\GenericDataIndexBundle\Model\OpenSearch` -> `Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch` - - `Pimcore\Bundle\GenericDataIndexBundle\Enum\SearchIndex\OpenSearch` -> `Pimcore\Bundle\GenericDataIndexBundle\Enum\SearchIndex\DefaultSearch` -- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\Exception\OpenSearch\SearchFailedException` please use `Pimcore\Bundle\GenericDataIndexBundle\Exception\OpenSearch\SearchFailedException` instead -- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\Attribute\OpenSearch\AsSearchModifierHandler` please use `Pimcore\Bundle\GenericDataIndexBundle\Attribute\Search\AsSearchModifierHandler` instead -- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\OpenSearch\Asset\FieldDefinitionAdapter\AbstractAdapter` please use `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\DefaultSearch\Asset\FieldDefinitionAdapter\AbstractAdapter` instead -- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\OpenSearch\DataObject\FieldDefinitionAdapter\AbstractAdapter` please use `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\DefaultSearch\DataObject\FieldDefinitionAdapter\AbstractAdapter` instead -- Added default prefix `data-object_` prefix to all data object class definition index names. This change is necessary to avoid conflicts with other index names. -- Add element type to the `getIds` method of `Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Element\SearchResult\ElementSearchResult` -- Added `getSpecialPermissions` method to `Pimcore\Bundle\GenericDataIndexBundle\Service\Permission\ElementPermissionServiceInterface` to get special permissions workspace language permissions for elements -- Removed layout permission from `Pimcore\Bundle\GenericDataIndexBundle\Permission\DataObjectPermissions` as they are not index relevant -- Removed property `isLocked` from Index for elements as it needs to be dynamically calculated -- Changed workspace permissions evaluation in order to align more with the Pimcore Classic bundle permission system - -#### Interface changes -- Added `PermissionTypes $permissionType` parameter with default type `PermissionTypes::LIST` to -- `AssetSearchServiceInterface::search` method -- `DocumentSearchServiceInterface::search` method -- `DataObjectSearchServiceInterface::search` method -- `ElementSearchServiceInterface::search` method -- Search services `byId` methods now return elements based on the `PermissionTypes::VIEW` permission -- Added type specific interfaces for searches to avoid mixing up different search types in search services - - `AssetSearch` now implements `AssetSearchInterface` - - `DocumentSearch` now implements `DocumentSearchInterface` - - `ElementSearch` now implements `ElementSearchInterface` -- Search services now require the specific search type for the search - - `AssetSearchServiceInterface::search` now requires a `AssetSearchInterface` - - `DocumentSearchServiceInterface::search` now requires a `DocumentSearchInterface` - - `ElementSearchServiceInterface::search` now requires a `ElementSearchInterface` -- `SearchProviderInterface` now returns type specific search interfaces +- **[Indexing]** Added inherited fields indicator to data object indexing +- **[Indexing]** Added functionality to enqueue dependent items +- **[Indexing]** Added class ID field for data object elements +- **[Indexing]** Index name prefix changed from e.g. `pimcore_car` to + `pimcore_data-object_car`. Old indices are **not deleted** automatically - + delete them manually if necessary. +- **[Searching]** Added `ClassIdsFilter` modifier to filter data objects by class ID + or class name +- Added `isElementLocked()` to `ElementLockService` for retrieving element lock status + from the index +- Reindex all elements to use the new features: + ```bash + bin/console generic-data-index:update:index -r + ``` + +### Breaking Changes + +- Removed deprecated alias `generic-data-index.opensearch-client`. + Use `generic-data-index.search-client` instead. +- Replaced all deprecated OpenSearch namespace classes with DefaultSearch equivalents: + - `Pimcore\...\Model\OpenSearch` → `Pimcore\...\Model\DefaultSearch` + - `Pimcore\...\Enum\SearchIndex\OpenSearch` → `Pimcore\...\Enum\SearchIndex\DefaultSearch` +- Removed deprecated class `Pimcore\...\Exception\OpenSearch\SearchFailedException`. + Use `Pimcore\...\Exception\DefaultSearch\SearchFailedException` instead. +- Removed deprecated class `AsSearchModifierHandler` from OpenSearch namespace. + Use `Pimcore\...\Attribute\Search\AsSearchModifierHandler` instead. +- Removed deprecated `AbstractAdapter` classes from OpenSearch namespace. + Use `Pimcore\...\SearchIndexAdapter\DefaultSearch\...\AbstractAdapter` instead. +- Added default `data-object_` prefix to all data object class definition index names + to avoid conflicts with other index names. +- Added element type parameter to `getIds` method of `ElementSearchResult`. +- Added `getSpecialPermissions` method to `ElementPermissionServiceInterface` + for workspace language permissions. +- Removed layout permission from `DataObjectPermissions` (not index-relevant). +- Removed `isLocked` property from index elements (now dynamically calculated). +- Changed workspace permissions evaluation to align with Admin Classic permission system. + +#### Interface Changes + +- Added `PermissionTypes $permissionType` parameter (default: `PermissionTypes::LIST`) to: + - `AssetSearchServiceInterface::search` + - `DocumentSearchServiceInterface::search` + - `DataObjectSearchServiceInterface::search` + - `ElementSearchServiceInterface::search` +- Search service `byId` methods now return elements based on `PermissionTypes::VIEW` +- Added type-specific interfaces: + - `AssetSearch` → `AssetSearchInterface` + - `DocumentSearch` → `DocumentSearchInterface` + - `ElementSearch` → `ElementSearchInterface` +- Search services now require type-specific search objects: + - `AssetSearchServiceInterface::search` requires `AssetSearchInterface` + - `DocumentSearchServiceInterface::search` requires `DocumentSearchInterface` + - `ElementSearchServiceInterface::search` requires `ElementSearchInterface` +- `SearchProviderInterface` now returns type-specific search interfaces. ## Upgrade to 1.3.0 -- [Indexing] Added support for Elasticsearch in parallel to Opensearch. Opensearch remains the default search technology. If you are using Elasticsearch, you need to update your symfony configuration as follows: -```yml + +- **[Indexing]** Added Elasticsearch support alongside OpenSearch (OpenSearch remains + the default). To use Elasticsearch, update your Symfony configuration: + +```yaml pimcore_generic_data_index: index_service: client_params: client_name: default client_type: 'elasticsearch' ``` -- [Indexing] Introduced new service alias `generic-data-index.search-client`. This will replace deprecated alias `generic-data-index.opensearch-client` which will be removed in the next major version. -The new service alias can be used to inject the search client into your services. This search client is an instance of `Pimcore\SearchClient\SearchClientInterface` which is a common interface for OpenSearch and Elasticsearch clients. -- Classes under OpenSearch namespaces are now deprecated and will be removed in the next major version. Please use the classes under the DefaultSearch namespace instead. -- Execute the following command to reindex all elements to be able to use all new features or when switching between OpenSearch and Elasticsearch: - ```bin/console generic-data-index:update:index``` +- **[Indexing]** Introduced service alias `generic-data-index.search-client`, replacing + the deprecated `generic-data-index.opensearch-client` (removed in 2.0). The alias + provides a `Pimcore\SearchClient\SearchClientInterface` instance compatible with both + OpenSearch and Elasticsearch. +- OpenSearch namespace classes are deprecated. Use DefaultSearch namespace instead. +- Reindex all elements after upgrading or switching search engines: + ```bash + bin/console generic-data-index:update:index + ``` ## Upgrade to 1.1.0 -- Execute the following command to reindex all elements to be able to use all new features: - ```bin/console generic-data-index:update:index``` +- Reindex all elements to use the new features: + ```bash + bin/console generic-data-index:update:index + ``` diff --git a/doc/01_Installation/README.md b/doc/01_Installation/README.md index 4c4cf415..827e40f9 100644 --- a/doc/01_Installation/README.md +++ b/doc/01_Installation/README.md @@ -1,22 +1,26 @@ -# Installation of Generic Data Index +--- +title: Installation +description: Install and configure the Generic Data Index bundle with OpenSearch or Elasticsearch. +--- + +# Installation :::info - This bundle requires minimum version of OpenSearch 2.7. or Elasticsearch 8.0.0. +This bundle requires OpenSearch >= 2.7 or Elasticsearch >= 8.0.0. ::: - ## Bundle Installation - -To install the Generic Data Index bundle, follow the steps below: +## Bundle Installation -1) Install the required dependencies: +1. Install the required dependencies: ```bash composer require pimcore/generic-data-index-bundle ``` -2) Make sure the bundle is enabled in the `config/bundles.php` file. The following lines should be added: +2. Enable the bundle in `config/bundles.php`: + ```php use Pimcore\Bundle\GenericDataIndexBundle\PimcoreGenericDataIndexBundle; // ... @@ -24,29 +28,33 @@ return [ // ... PimcoreGenericDataIndexBundle::class => ['all' => true], // ... -]; +]; ``` -3) Install the bundle: +3. Install the bundle: ```bash bin/console pimcore:bundle:install PimcoreGenericDataIndexBundle ``` -4) Setup search client configuration in your Symfony configuration files (e.g. `config.yaml`): - -See [OpenSearch Client Setup](../02_Configuration/04_Opensearch.md) or [Elasticsearch Client Setup](../02_Configuration/05_Elasticsearch.md) for more information. +4. Configure the search client in your Symfony configuration (e.g. `config.yaml`). + See [OpenSearch Client Setup](../02_Configuration/04_Opensearch.md) + or [Elasticsearch Client Setup](../02_Configuration/05_Elasticsearch.md). -5) Setup one or multiple Symfony messenger workers for the indexing queue processing. It is recommended to use a tool like Supervisor to manage the workers. - For more information, see the [Symfony Messenger documentation](https://symfony.com/doc/current/messenger.html). +5. Start one or more Symfony Messenger workers for index queue processing. + Use a process manager like Supervisor to keep workers running. + See the [Symfony Messenger documentation](https://symfony.com/doc/current/messenger.html) + for details. ```bash -./bin/console messenger:consume pimcore_generic_data_index_queue scheduler_generic_data_index +bin/console messenger:consume pimcore_generic_data_index_queue scheduler_generic_data_index ``` -**Deployment hint:** +:::tip Deployment hint -For deployments of applications with this bundle via deployment pipelines without actual database access, the Symfony cache warming process could fail as doctrine ORM tries to determine the database version on cache warm-up to build its cache. Therefore, it is recommended to configure the database server version in the default DBAL connection like this: +For deployments without database access (e.g. CI pipelines), Doctrine ORM cache warm-up +fails because it tries to detect the database version. Configure the server version +explicitly in the default DBAL connection: ```yaml doctrine: @@ -58,9 +66,12 @@ doctrine: server_version: mariadb-10.11.0 ``` -## Commands after Installation +::: + +## Post-Installation + +After installation, create the indices and queue all elements for indexing: -It is needed to run following command after installation (at least) once to create the indices and add all assets and data objects to the index queue: ```bash -./bin/console generic-data-index:update:index -r +bin/console generic-data-index:update:index -r ``` diff --git a/doc/02_Configuration/03_Index_Management.md b/doc/02_Configuration/03_Index_Management.md index 83741e3d..13e72939 100644 --- a/doc/02_Configuration/03_Index_Management.md +++ b/doc/02_Configuration/03_Index_Management.md @@ -1,64 +1,83 @@ +--- +title: Index Management +description: Manage search indices for the Generic Data Index, including creation, updates, queue processing, and deployment. +--- + # Index Management -It is important to index all assets and data object in Pimcore in order to be able to use the search and listing features powered by the Generic Data Index bundle. +The Generic Data Index must index all assets, data objects, and documents +to power search and listing features in Pimcore. + +## Console Commands Overview -## Define Index Prefix +| Command | Description | +|---------|-------------| +| `generic-data-index:update:index` | Update index mappings and queue all elements for reindex from the database | +| `generic-data-index:update:index -r` | Delete and recreate indices, then queue all elements | +| `generic-data-index:reindex` | Native search engine reindex (reorganizes data within existing indices, no database read) | +| `generic-data-index:deployment:reindex` | Update indices only for class definitions changed since the last deployment | -To avoid duplicate names or index interferences in your search engine, it is necessary to define an index name prefix, which is added to all indices created by Generic Data Index. -Default one is `pimcore_`. +## Index Prefix -This can be done by defining following configuration: +Define an index name prefix to avoid naming collisions in shared search engine clusters. +The default prefix is `pimcore_`. -```yaml +```yaml pimcore_generic_data_index: index_service: client_params: - index_prefix: 'my_prefix' # Prefix for all index names created by Generic Data Index + index_prefix: 'my_prefix' ``` ## Created Indices -The Generic Data Index generates indices for the following entities: +The Generic Data Index creates the following indices: -* Assets search index (one alias and one index) -* Data objects search index (one alias and one index per class definition) +- **Assets** - one alias and one index +- **Data objects** - one alias and one index per class definition -For the asset and data object indices the Generic Data Index uses an alias (e.g. `_asset`) that points to the -most current index (e.g. `_asset-odd`). The alias name always stays the same, the index names alternate -between `-odd` and `-even` suffix. For more details also see 'Updating index structure for data indices' in the next section. +Each index uses an alias (e.g. `_asset`) pointing to the current index +(e.g. `_asset-odd`). The alias name stays constant; the backing index alternates +between `-odd` and `-even` suffixes during reindexing (see +[Updating index structure](#updating-index-structure) below). ## Keeping Indices Up to Date -The element search indices need to be created via the following console commands: +Create and update indices with: -``` -# create/update all indices + their mappings and add all items to the index queue +```bash bin/console generic-data-index:update:index ``` -The command will create the indices and add all assets and data objects to the index queue. The queue will be processed by Symfony messenger workers (`pimcore_generic_data_index_queue` queue). +This command creates the indices and queues all assets and data objects for indexing. +The Symfony Messenger `pimcore_generic_data_index_queue` transport processes the queue. -### Refreshing of the index +### Index Refresh -By default, the index queue is refreshed after each bulk operation as the items are processed asynchronously with the Symfony messenger. -If you want to perform index refresh immediately you can use enable synchronous processing by injecting the `SynchronousProcessingServiceInterface` and calling `enable()` method. +By default, the index refreshes after each bulk operation since items are processed +asynchronously via Symfony Messenger. -Available methods: -- `enable()`: enable synchronous processing -- `disable()`: disable synchronous processing -- `isEnabled()`: check if synchronous processing is enabled +To force synchronous processing (immediate refresh), inject +`SynchronousProcessingServiceInterface` and call `enable()`: -### Index Queue Options +| Method | Description | +|--------|-------------| +| `enable()` | Enable synchronous processing | +| `disable()` | Disable synchronous processing | +| `isEnabled()` | Check current mode | -The indexing queue considers the following options: +### Queue Options -- **worker_count** (default 1): number of messenger workers to process the queue. Set this to the actual used parallel number of `messenger:consume` workers to improve the calculation of items per batch. -- **min_batch_size** (default 5): minimum number of items to process in one batch (when using multiple workers) -- **max_batch_size** (default 400): maximum number of items to process in one batch +Configure the indexing queue batch behavior: -Based on this configuration, the queue will be processed in batches of `min_batch_size` to `max_batch_size` items. The number of items per batch is calculated based on the number of workers and the number of items in the queue. +| Option | Default | Description | +|--------|---------|-------------| +| `worker_count` | 1 | Number of parallel `messenger:consume` workers. Improves batch size calculation. | +| `min_batch_size` | 5 | Minimum items per batch (relevant with multiple workers) | +| `max_batch_size` | 400 | Maximum items per batch | -Sample configuration: +The queue calculates batch sizes dynamically between `min_batch_size` and `max_batch_size` +based on the number of workers and queue depth. ```yaml pimcore_generic_data_index: @@ -69,76 +88,67 @@ pimcore_generic_data_index: max_batch_size: 400 ``` -#### Related elements +### Related Elements -The indexing queue is automatically populated whenever an element undergoes an update operation. This process includes not only the modified element itself but also any related elements. By default, this indexing occurs asynchronously through Symfony Messenger. +Updating an element automatically enqueues its related elements for reindexing. +By default, this runs asynchronously through Symfony Messenger. -For scenarios requiring immediate processing, you can temporarily switch to synchronous mode by utilizing the `SynchronousProcessingRelatedIdsServiceInterface`. - -Available methods are: +For immediate processing, use `SynchronousProcessingRelatedIdsServiceInterface`: | Method | Description | |--------|-------------| -| `enable()` | Activates synchronous processing mode | -| `disable()` | Reverts to asynchronous processing mode | -| `isEnabled()` | Returns the current processing mode status | +| `enable()` | Activate synchronous processing | +| `disable()` | Revert to asynchronous processing | +| `isEnabled()` | Return current processing mode | :::info -Currently the `SynchronousProcessingRelatedIdsServiceInterface` interface does not influence the behavior of delete operations. They are always processed synchronously. +`SynchronousProcessingRelatedIdsServiceInterface` does not affect delete operations. +Deletes always process synchronously. ::: - ### Repairing Indices -Sometimes it might be needed to delete and recreate the index from the Pimcore database -(for example if the mapping changed and cannot be updated). +To delete and recreate an index from the Pimcore database (e.g. after an incompatible +mapping change), pass the `-r` option: -Do this with the index update command and pass `-r` option (which deletes and recreates the index). -``` -# delete index and recreate it +```bash bin/console generic-data-index:update:index -r ``` -Without the `-r` option, the index mapping is just updated and all items are added into the queue -for a reindex from the Pimcore database. - -### Updating Index Structure for Data Indices +Without `-r`, the command only updates the index mapping and queues all items for reindex. -Index mapping is updated automatically e.g. when adding system languages or new fields to the class definition. -Sometimes it might be necessary to update the index structure manually. +### Updating Index Structure -Do this with the reindex command. This command does native opensearch/elasticsearch re-indexing. So it does not -index data from the database but reindexes data within the search indices. +Index mappings update automatically when system languages or class definition fields change. +For manual updates, run the reindex command. This performs a native OpenSearch/Elasticsearch +reindex within the search indices (no database read): -``` -# updates index mapping with native reindexing +```bash bin/console generic-data-index:reindex ``` ### Handling Failed Messages -By default, the messenger will retry failed messages 3 times and then send them into the failed queue `pimcore_generic_data_index_failed`. -If you want to retry failed messages, you can use the following command: +The messenger retries failed messages 3 times, then routes them to the +`pimcore_generic_data_index_failed` transport. Retry failed messages with: -``` -php bin/console messenger:failed:retry -vv +```bash +bin/console messenger:failed:retry -vv ``` -For the further commands please refer to the [Symfony Messenger documentation](https://symfony.com/doc/current/messenger.html#saving-retrying-failed-messages). +See the [Symfony Messenger documentation](https://symfony.com/doc/current/messenger.html#saving-retrying-failed-messages) +for additional commands. -## Configuring index options (Maximum Item Limit, ...) +## Index Options -You can configure different options to use with your indices. The available options can differ depending on which -engine you are using. Make sure to check the corresponding documentation, before using any options. +Configure search engine-specific index options. Check your engine's documentation +for available settings. -See the `Maxium Item Limit` and `Total fields limit` section for examples. +### Maximum Result Window -### Maximum Item limit - -A maximum of 10000 items can be retrieved and viewed, because of the maximum default item limit. -To increase this limit, configuration can be adjusted as follows: +The default limit of retrievable items is 10,000. Increase it with: ```yaml pimcore_generic_data_index: @@ -147,10 +157,9 @@ pimcore_generic_data_index: max_result_window: 20000 ``` -### Total fields limit +### Total Fields Limit -A maximum of 1000 fields can be used with your indces. -To increase this limit, configuration can be adjusted as follows: +The default field limit per index is 1,000. Increase it with: ```yaml pimcore_generic_data_index: @@ -161,9 +170,9 @@ pimcore_generic_data_index: :::info -If an index was already created before setting this parameter, the index needs to be recreated. +If the index already exists, recreate it after changing this setting: -``` +```bash bin/console generic-data-index:update:index -r ``` @@ -171,12 +180,13 @@ bin/console generic-data-index:update:index -r ## Deployment and Index Management -### Pimcore Class Definitions +### Class Definition Changes -After every class definition update you should run the following command to update the index structure: +After updating class definitions during deployment, run: -``` -php bin/console generic-data-index:deployment:reindex +```bash +bin/console generic-data-index:deployment:reindex ``` -This command will update the index structure for all data object classes which were created/updated since the last deployment and reindex all data objects for relevant classes. +This updates the index structure for all class definitions modified since the last +deployment and reindexes data objects for affected classes. diff --git a/doc/02_Configuration/04_Opensearch.md b/doc/02_Configuration/04_Opensearch.md index 77757822..8cb66124 100644 --- a/doc/02_Configuration/04_Opensearch.md +++ b/doc/02_Configuration/04_Opensearch.md @@ -1,17 +1,23 @@ +--- +title: OpenSearch Client Setup +description: Configure OpenSearch as the search engine for the Generic Data Index bundle. +--- + # OpenSearch Client Setup :::info -Supported versions of OpenSearch are 2.7. to 2.19 +Supported OpenSearch versions: 2.7 to 2.19 ::: -Following configuration is required to set up OpenSearch. The OpenSearch client configuration takes place via [Pimcore Opensearch Client](https://github.com/pimcore/opensearch-client) and has two parts: -1) Configuring an OpenSearch client. -2) Define the client to be used by Generic Data Index bundle. +Configuration requires two steps: +1. Configure an OpenSearch client via + [Pimcore OpenSearch Client](https://github.com/pimcore/opensearch-client). +2. Assign the client to the Generic Data Index bundle. ```yaml -# Configuring an OpenSearch client +# 1. OpenSearch client configuration pimcore_open_search_client: clients: default: @@ -20,15 +26,25 @@ pimcore_open_search_client: username: 'admin' ssl_verification: false -# Define the client to be used by your bundle +# 2. Assign client to Generic Data Index pimcore_generic_data_index: index_service: client_params: client_name: default ``` -For the further configuration of the client, please refer to the [Pimcore OpenSearch Client documentation](https://github.com/pimcore/opensearch-client/blob/1.x/doc/02_Configuration.md). +For additional client options, see the +[Pimcore OpenSearch Client documentation](https://github.com/pimcore/opensearch-client/blob/1.x/doc/02_Configuration.md). + +## Disable Auto-Index Creation + +OpenSearch creates indices automatically when storing data to a nonexistent index. +This causes incorrect indices and missing aliases. +Disable this in your OpenSearch configuration: -## Important OpenSearch Configuration +``` +action.auto_create_index=false +``` -OpenSearch automatically creates indices on storing data if the index does not yet exist. This will cause issues with wrong indices and missing aliases. To overcome this issue, you need to disable that feature with the configuration `action.auto_create_index=false`. (see here for more information on this https://github.com/pimcore/generic-data-index-bundle/issues/165 and https://github.com/pimcore/generic-data-index-bundle/issues/202) +See [#165](https://github.com/pimcore/generic-data-index-bundle/issues/165) and +[#202](https://github.com/pimcore/generic-data-index-bundle/issues/202) for details. diff --git a/doc/02_Configuration/05_Elasticsearch.md b/doc/02_Configuration/05_Elasticsearch.md index dcfe3835..693f1d08 100644 --- a/doc/02_Configuration/05_Elasticsearch.md +++ b/doc/02_Configuration/05_Elasticsearch.md @@ -1,17 +1,23 @@ +--- +title: Elasticsearch Client Setup +description: Configure Elasticsearch as the search engine for the Generic Data Index bundle. +--- + # Elasticsearch Client Setup :::info -This bundle requires minimum version of Elasticsearch 8.0. +Requires Elasticsearch >= 8.0 ::: -Following configuration is required to set up Elasticsearch. The Elasticsearch client configuration takes place via [Pimcore Elasticsearch Client](https://github.com/pimcore/elasticsearch-client) and has two parts: -1) Configuring an Elasticsearch client. -2) Define the client to be used by Generic Data Index bundle. +Configuration requires two steps: +1. Configure an Elasticsearch client via + [Pimcore Elasticsearch Client](https://github.com/pimcore/elasticsearch-client). +2. Assign the client to the Generic Data Index bundle with `client_type: 'elasticsearch'`. ```yaml -# Configuring an Elasticsearch client +# 1. Elasticsearch client configuration pimcore_elasticsearch_client: es_clients: default: @@ -20,7 +26,7 @@ pimcore_elasticsearch_client: password: 'somethingsecret' logger_channel: 'pimcore.elasticsearch' -# Define the client to be used by your bundle (default client_type is 'openSearch') +# 2. Assign client to Generic Data Index (default client_type is 'openSearch') pimcore_generic_data_index: index_service: client_params: @@ -28,8 +34,15 @@ pimcore_generic_data_index: client_type: 'elasticsearch' ``` -For the further configuration of the client, please refer to the [Pimcore Elasticsearch Client documentation](https://github.com/pimcore/elasticsearch-client/blob/1.x/README.md). +For additional client options, see the +[Pimcore Elasticsearch Client documentation](https://github.com/pimcore/elasticsearch-client/blob/1.x/README.md). + +## Disable Auto-Index Creation -## Important Elasticsearch Configuration +Elasticsearch creates indices automatically when storing data to a nonexistent index. +This causes incorrect indices and missing aliases. +Disable this in your Elasticsearch configuration: -Elasticsearch automatically creates indices on storing data if the index does not yet exist. This will cause issues with wrong indices and missing aliases. To overcome this issue, you need to disable that feature with the configuration `action.auto_create_index=false`. +``` +action.auto_create_index=false +``` diff --git a/doc/02_Configuration/README.md b/doc/02_Configuration/README.md index 344bdc8f..f4032fc5 100644 --- a/doc/02_Configuration/README.md +++ b/doc/02_Configuration/README.md @@ -1,9 +1,23 @@ -# Configuration of the Generic Data Index Bundle +--- +title: Configuration +description: Configure the Generic Data Index bundle, including search client, index prefix, queue settings, and index options. +--- -The Generic Data Index Bundle provides a generic way to index and search data in Pimcore. It is shipped with the OpenSearch and Elasticsearch clients and provides a central configuration for it in order to be used in other bundles. -This bundle can be extended and customized to fit your specific needs, for example if you would like to use a custom search engine. +# Configuration + +The Generic Data Index Bundle indexes and searches Pimcore elements (assets, data objects, +documents) through OpenSearch or Elasticsearch. It provides a central search client +configuration shared across bundles and supports custom search engine integrations. + +Configuration covers: + +- **Search client** - which OpenSearch or Elasticsearch instance to connect to +- **Index prefix** - namespace for index names to avoid collisions in shared clusters +- **Queue settings** - batch sizes and worker count for asynchronous indexing +- **Index settings** - engine-specific options like result window size and field limits ## Further Reading -- [Index management](./03_Index_Management.md) -- [OpenSearch setup](./04_Opensearch.md) -- [Elasticsearch setup](./05_Elasticsearch.md) \ No newline at end of file + +- [Index Management](./03_Index_Management.md) +- [OpenSearch Setup](./04_Opensearch.md) +- [Elasticsearch Setup](./05_Elasticsearch.md) diff --git a/doc/04_Searching_For_Data_In_Index/05_Search_Modifiers/README.md b/doc/04_Searching_For_Data_In_Index/05_Search_Modifiers/README.md index db82d95b..1d3beb13 100644 --- a/doc/04_Searching_For_Data_In_Index/05_Search_Modifiers/README.md +++ b/doc/04_Searching_For_Data_In_Index/05_Search_Modifiers/README.md @@ -1,107 +1,124 @@ -# Search Modifiers +--- +title: Search Modifiers +description: Filter, sort, and aggregate search results using built-in and custom search modifiers. +--- -Search modifiers can influence the search results by modifying the search query. They can be used to filter, sort or aggregate the search results. +# Search Modifiers -Search modifiers can be added to the search via the `addModifier()` method of the search object. +Search modifiers filter, sort, and aggregate search results by altering the +underlying query. Add them via the `addModifier()` method: ```php -$search->addModifier(new ParentIdFilter(1)) +$search->addModifier(new ParentIdFilter(1)); ``` ## Available Search Modifiers ### Filters -| Modifier | Modifier Category | Description | -|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [IdFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Basic/IdFilter.php) | Basic filters | Filter by element ID | -| [IdsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Basic/IdsFilter.php) | Basic filters | Filter by multiple element IDs | -| [BooleanFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.x/src/Model/Search/Modifier/Filter/Basic/BooleanFilter.php) | Basic filters | Filter boolean fields based on the value with [PQL field name resolution support](#pql-field-name-resolution) | -| [IntegerFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Basic/IntegerFilter.php) | Basic filters | Filter integer fields based on the value with [PQL field name resolution support](#pql-field-name-resolution) | -| [NumberFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.x/src/Model/Search/Modifier/Filter/Basic/NumberFilter.php) | Basic filters | Filter number fields based on the value with [PQL field name resolution support](#pql-field-name-resolution) | -| [ExcludeFoldersFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Basic/ExcludeFoldersFilter.php) | Basic filters | Exclude folders from search result | -| [ExcludeVariantsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.x/src/Model/Search/Modifier/Filter/Basic/ExcludeVariantsFilter.php) | Basic filters | Exclude data object variants from search result | -| [ParentIdsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Tree/ParentIdsFilter.php) | Tree related filters | Filter by parent ID | -| [PathFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Tree/PathFilter.php) | Tree related filters | Filter by path (depending on use case for all levels or direct children only and with or without the parent item included) | -| [ClassIdsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Tree/ClassIdsFilter.php) | Tree related filters | Filter object items by class IDs (depending on use case the folders can be included in the result). Setting parameter `$useClassName` to `true` allows filtering based on the classNames instead | -| [TagFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Tree/TagFilter.php) | Tree related filters | Filter by tag IDs (it is also possible to include child tags) | -| [AssetMetaDataFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Asset/AssetMetaDataFilter.php) | Asset filters | Filter by asset meta data attribute. The format of the `$data` which needs to be passed depends on the type of the meta data attribute and is handled by its [field definition adapter](https://github.com/pimcore/generic-data-index-bundle/tree/1.x/src/SearchIndexAdapter/DefaultSearch/Asset/FieldDefinitionAdapter). | -| [WorkspaceQuery](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Workspaces/WorkspaceQuery.php) | Workspace related filters | Filter based on the user workspaces and permissions for a defined element type (this query is added to the asset/document/data object search by default) | -| [ElementWorkspacesQuery](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Workspaces/WorkspaceQuery.php) | Workspace related filters | Filter based on the user workspaces and permissions respecting all element types (this query is added to the element search by default) | -| [MultiSelectFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/FieldType/MultiSelectFilter.php) | Field type filters | Filter text fields by a list of exact strings. Supports [PQL field name resolution](#pql-field-name-resolution). | -| [BooleanMultiSelectFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.x/src/Model/Search/Modifier/Filter/FieldType/BooleanMultiSelectFilter.php) | Field type filters | Filter boolean fields by a list of values (true, false, null). Supports [PQL field name resolution](#pql-field-name-resolution). | -| [DateFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/FieldType/DateFilter.php) | Field type filters | Filter date fields based on an exact date or a range of dates. Supports [PQL field name resolution](#pql-field-name-resolution). | -| [ClassificationStoreFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.x/src/Model/Search/Modifier/Filter/FieldType/ClassificationStoreFilter.php) | Nested filters | Filter based on the classification store field values. Requires sub-modifier based on the filtered field type. Only fields types, which are supported by classificaiton store can be used for sub-modifier. | -| [NestedFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.x/src/Model/Search/Modifier/Filter/FieldType/NestedFilter.php) | Nested filters | Filter for nested fields. Requires sub-modifier based on the field type of nested field. | +| Modifier | Category | Description | +|----------|----------|-------------| +| [IdFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/IdFilter.php) | Basic | Filter by element ID | +| [IdsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/IdsFilter.php) | Basic | Filter by multiple element IDs | +| [BooleanFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/BooleanFilter.php) | Basic | Filter boolean fields. Supports [PQL field name resolution](#pql-field-name-resolution). | +| [IntegerFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/IntegerFilter.php) | Basic | Filter integer fields. Supports [PQL field name resolution](#pql-field-name-resolution). | +| [NumberFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/NumberFilter.php) | Basic | Filter number fields. Supports [PQL field name resolution](#pql-field-name-resolution). | +| [ExcludeFoldersFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/ExcludeFoldersFilter.php) | Basic | Exclude folders from results | +| [ExcludeVariantsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/ExcludeVariantsFilter.php) | Basic | Exclude data object variants from results | +| [ParentIdsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Tree/ParentIdFilter.php) | Tree | Filter by one or more parent IDs | +| [PathFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Tree/PathFilter.php) | Tree | Filter by path (all levels or direct children, with or without parent) | +| [ClassIdsFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Tree/ClassIdsFilter.php) | Tree | Filter objects by class IDs (optionally include folders). Set `$useClassName` to `true` to filter by class name instead. | +| [TagFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Tree/TagFilter.php) | Tree | Filter by tag IDs (optionally include child tags) | +| [AssetMetaDataFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Asset/AssetMetaDataFilter.php) | Asset | Filter by asset metadata attribute. The `$data` format depends on the metadata type and its [field definition adapter](https://github.com/pimcore/generic-data-index-bundle/tree/2026.x/src/SearchIndexAdapter/DefaultSearch/Asset/FieldDefinitionAdapter). | +| [WorkspaceQuery](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Workspaces/WorkspaceQuery.php) | Workspace | Filter by user workspaces and permissions for a single element type (added to asset/document/data object searches by default) | +| [ElementWorkspacesQuery](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Workspaces/ElementWorkspacesQuery.php) | Workspace | Filter by user workspaces across all element types (added to element search by default) | +| [MultiSelectFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/FieldType/MultiSelectFilter.php) | Field type | Filter text fields by exact string list. Supports [PQL field name resolution](#pql-field-name-resolution). | +| [BooleanMultiSelectFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/FieldType/BooleanMultiSelectFilter.php) | Field type | Filter boolean fields by value list (true, false, null). Supports [PQL field name resolution](#pql-field-name-resolution). | +| [DateFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/FieldType/DateFilter.php) | Field type | Filter date fields by exact date or date range. Supports [PQL field name resolution](#pql-field-name-resolution). | +| [ClassificationStoreFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/FieldType/ClassificationStoreFilter.php) | Nested | Filter by classification store field values. Requires a sub-modifier matching the field type. | +| [NestedFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/FieldType/NestedFilter.php) | Nested | Filter nested fields. Requires a sub-modifier matching the nested field type. | ### Full Text Search Queries -| Modifier | Modifier Category | Description | -|-------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [ElementKeySearch](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/FullTextSearch/ElementKeySearch.php) | Full text search | Search by element key like in the studio UI with [wildcard support](#wildcard-support). | -| [FullTextSearch](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/FullTextSearch/FullTextSearch.php) | Full text search | Search on all element fields by value with simple query string syntax for [OpenSearch](https://opensearch.org/docs/latest/query-dsl/full-text/simple-query-string/#simple-query-string-syntax) or [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax). | -| [MultiMatchSearch](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/FullTextSearch/MultiMatchSearch.php) | Full text search | Search using multi_match query with configurable fields, match type (best_fields, most_fields, cross_fields, phrase, phrase_prefix) and operator (or/and). See [OpenSearch](https://opensearch.org/docs/latest/query-dsl/full-text/multi-match/) or [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html) documentation. | -| [WildcardSearch](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/FullTextSearch/WildcardSearch.php) | Full text search | Filter text fields based on search terms with [wildcard support](#wildcard-support) and [PQL field name resolution support](#pql-field-name-resolution). | - +| Modifier | Description | +|----------|-------------| +| [ElementKeySearch](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/FullTextSearch/ElementKeySearch.php) | Search by element key (as in Pimcore Studio) with [wildcard support](#wildcard-support) | +| [FullTextSearch](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/FullTextSearch/FullTextSearch.php) | Search all element fields using simple query string syntax ([OpenSearch](https://opensearch.org/docs/latest/query-dsl/full-text/simple-query-string/#simple-query-string-syntax) / [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax)) | +| [MultiMatchSearch](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/FullTextSearch/MultiMatchSearch.php) | Search with configurable fields, match type (best_fields, most_fields, cross_fields, phrase, phrase_prefix), and operator (or/and). See [OpenSearch](https://opensearch.org/docs/latest/query-dsl/full-text/multi-match/) / [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html) docs. | +| [WildcardSearch](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/FullTextSearch/WildcardSearch.php) | Filter text fields with [wildcard support](#wildcard-support) and [PQL field name resolution](#pql-field-name-resolution) | ### Dependencies -| Modifier | Modifier Category | Description | -|------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------------------------------------| -| [RequiresFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Dependency/RequiresFilter.php) | Dependencies | Get all elements which the given element requires. | -| [RequiredByFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Dependency/RequiredByFilter.php) | Dependencies | Get all elements which are required by the given element. | - +| Modifier | Description | +|----------|-------------| +| [RequiresFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Dependency/RequiresFilter.php) | Get all elements that the given element requires | +| [RequiredByFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Dependency/RequiredByFilter.php) | Get all elements required by the given element | ### Query Language -| Modifier | Modifier Category | Description | -|----------------------------------------------------------------------------------------------------------------------------------|-------------------|-------------------------------------------------------------------------------------------| -| [PqlFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/QueryLanguage/PqlFilter.php) | Query Language | Apply a [Pimcore Query Language (PQL)](../09_Pimcore_Query_Language/README.md) condition. | +| Modifier | Description | +|----------|-------------| +| [PqlFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/QueryLanguage/PqlFilter.php) | Apply a [Pimcore Query Language (PQL)](../09_Pimcore_Query_Language/README.md) condition | ### Sort Modifiers -If multiple sort modifiers are added to the search, the order of the modifiers is important. The search result will be sorted by the first added modifier first, then by the second added modifier and so on. +When multiple sort modifiers are added, they apply in order: the first modifier +is the primary sort, the second is the secondary sort, and so on. -| Modifier | Modifier Category | Description | -|----------------------------------------------------------------------------------------------------------------------------------------------|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [OrderByFullPath](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Sort/Tree/OrderByFullPath.php) | Tree related sorting | Order by full path (including element key) | -| [OrderByField](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Sort/OrderByField.php) | Field based sorting | Order by given field name.
If `$enablePqlFieldNameResolution` is set to true (default) [Pimcore Query Language](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Sort/OrderByField.php) field name resolution logic is enabled. Therefore it's possible to use short field names then instead of specifying the full indexed path. | -| [OrderByPageNumber](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Sort/Tree/OrderByPageNumber.php) | Search related sorting | Use inverted search for large amounts of data (this modifier is added to the search when there are at least 1000 results by default, and page number is above the half of total pages. Furthermore, existing sorting has to be already applied.) | -| [OrderByIndexField](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Sort/Tree/OrderByIndexField.php) | Search related sorting | Order by element tree index for custom tree sorting. This modifier is currently applied only for data objects and documents! | +| Modifier | Category | Description | +|----------|----------|-------------| +| [OrderByFullPath](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Sort/Tree/OrderByFullPath.php) | Tree | Sort by full path (including element key) | +| [OrderByField](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Sort/OrderByField.php) | Field | Sort by field name. With `$enablePqlFieldNameResolution` set to `true` (default), short field names resolve automatically via PQL logic. | +| [OrderByPageNumber](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Sort/OrderByPageNumber.php) | Search | Inverted search for large result sets. Applied automatically when results exceed 1,000 and the current page is past the halfway point. Requires existing sorting. | +| [OrderByIndexField](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Sort/Tree/OrderByIndexField.php) | Search | Sort by element tree index for custom tree ordering. Applies to data objects and documents only. | ### Aggregations +| Modifier | Category | Description | +|----------|----------|-------------| +| [ChildrenCountAggregation](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Aggregation/Tree/ChildrenCountAggregation.php) | Tree | Get children counts for given element IDs | +| [AssetMetaDataAggregation](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Aggregation/Asset/AssetMetaDataAggregation.php) | Asset | Aggregate filter options for supported metadata types (used in asset grid filters) | +| [FileSizeSumAggregation](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Aggregation/Asset/FileSizeSumAggregation.php) | Asset | Sum file sizes across assets for a search. Use `FileSizeAggregationServiceInterface` for a simplified API. | + +## Implementation Details + +### Wildcard Support -| Modifier | Modifier Category | Description | -|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [ChildrenCountAggregation](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Aggregation/Tree/ChildrenCountAggregation.php) | Tree related aggregation | Get children counts for given element IDs. | -| [AssetMetaDataAggregation](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Aggregation/Asset/AssetMetaDataAggregation.php) | Assets | Used for the filters in the asset grid to aggregate the filter options for supported meta data types. | -| [FileSizeSumAggregation](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Aggregation/Asset/FileSizeSumAggregation.php) | Assets | Aggregates the sum of file sizes for all assets for a given search. The `FileSizeAggregationServiceInterface` internally uses this aggregation and provides an easy way to use this functionality. | +Some modifiers support wildcard characters in search terms: -## Search Modifier Implementation Details +- `*` matches any character sequence (e.g. `Car*` matches "Car", "Carbon", "Carpet") +- `?` matches exactly one character (e.g. `Car?` matches "Card", "Cars") -### Wildcard support +### PQL Field Name Resolution -For some search modifiers, wildcard support is available. Wildcards support the following characters: -- `*` can be used to match any sequence of characters, regardless of length - for example "Car*" to find all items starting with "Car". -- `?` can be used to match exactly one character - for example "Car?" to find all items starting with "Car" and having one more character. +Some modifiers support +[Pimcore Query Language (PQL)](../09_Pimcore_Query_Language/README.md) +field name resolution via `$enablePqlFieldNameResolution` (enabled by default). +This allows using short field names instead of full indexed paths. -### PQL field name resolution +## Creating Custom Search Modifiers -Some modifiers support [Pimcore Query Language (PQL)](../09_Pimcore_Query_Language/README.md) field name resolution by setting `$enablePqlFieldNameResolution` to `true` (enabled by default). Therefore, it's possible to use short field names then instead of specifying the full indexed path. +Creating a custom search modifier requires two steps: -## Add your own search modifier +### 1. Define the Modifier Model -To add a custom search modifier implementation two steps are necessary: +Create a class implementing `ModifierInterface`. This model holds the modifier's +configurable attributes. See +[IdFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/Filter/Basic/IdFilter.php) +for an example. -1. Create a new class that implements the `Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\ModifierInterface` interface. -This model class should contain all configurable attributes for the modifier. Take a look at the [IdFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/Filter/Basic/IdFilter.php) for an example. +### 2. Implement the Handler -2. Create a service to implement the logic behind the modifier and add the [AsSearchModifierHandler](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Attribute/DefaultSearch/AsSearchModifierHandler.php) attribute. -The attribute can either be directly added to the method which implements to logic or to a class. If added to a class the ´__invoke` method will be used as the handler. +Create a service with the +[AsSearchModifierHandler](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Attribute/Search/AsSearchModifierHandler.php) +attribute. Apply the attribute to either a method or a class (uses `__invoke`). -The implemented method needs exactly two arguments.: -* First argument: the modifier model (see step 1). -* Second argument: [SearchModifierContextInterface](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/DefaultSearch/Modifier/SearchModifierContextInterface.php) $context +The handler method requires exactly two parameters: +- The modifier model (from step 1) +- `SearchModifierContextInterface $context` -Take a look at the [BasicFilters](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/SearchIndexAdapter/DefaultSearch/Search/Modifier/Filter/BasicFilters.php) for an example and the [Default search models documentation](../06_Default_Search_Models/README.md) for more details about the search models to manipulate the search. \ No newline at end of file +See +[BasicFilters](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/SearchIndexAdapter/DefaultSearch/Search/Modifier/Filter/BasicFilters.php) +for an example and the +[Default Search Models documentation](../06_Default_Search_Models/README.md) +for manipulating the search query. diff --git a/doc/04_Searching_For_Data_In_Index/06_Default_Search_Models/README.md b/doc/04_Searching_For_Data_In_Index/06_Default_Search_Models/README.md index 8d22d628..91a5f93a 100644 --- a/doc/04_Searching_For_Data_In_Index/06_Default_Search_Models/README.md +++ b/doc/04_Searching_For_Data_In_Index/06_Default_Search_Models/README.md @@ -1,25 +1,25 @@ -# Default Search Models - -:::info - -All models under namespace `Pimcore\Bundle\GenericDataIndexBundle\Model\OpenSearch` are deprecated and will be removed in version 2.0 - -Please use models from `Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch` instead. +--- +title: Default Search Models +description: Build custom OpenSearch and Elasticsearch queries using default search models for the Generic Data Index. +--- -::: - -Default search models can be used when individual search queries are needed to streamline the creation of Elasticsearch or OpenSearch search JSONs. +# Default Search Models -This is especially useful when you want to create your own [search modifiers](../05_Search_Modifiers/README.md) or when you would like to create services which should directly execute searches through the search client. They are used by the Generic Data Index and its search services internally to handle the execution of search queries on a lower level. +Default search models build OpenSearch/Elasticsearch query JSON programmatically. +Use them when building custom [search modifiers](../05_Search_Modifiers/README.md) +or when executing searches directly through the search client. The Generic Data Index +search services use these models internally. -## Example usage in search modifier +## Example: Custom Search Modifier -This example shows how to use a custom search modifier to add a term filter to the search query. +Add a term filter to the search query in a custom search modifier handler: ```php #[AsSearchModifierHandler] -public function handleCustomFilter(CustomFilter $customFilter, SearchModifierContextInterface $context): void -{ +public function handleCustomFilter( + CustomFilter $customFilter, + SearchModifierContextInterface $context +): void { $context->getSearch()->addQuery( new TermFilter( field: $customFilter->getField(), @@ -31,33 +31,34 @@ public function handleCustomFilter(CustomFilter $customFilter, SearchModifierCon ## Search Model -The search model is the main model to create a search query. It can be used to add queries, filters, aggregations and sorting to the search. +The `Search` model is the top-level container for building a search query. +Add queries, aggregations, sorting, and control pagination: ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch\Search; $search = (new Search()) - ->setSize(10) // set the number of results to return - ->setFrom(0) // set the offset of the results - ->setSource(['field']) // set the fields to return - ->addSort(new FieldSort('field', 'asc')) // add a sort - ->addQuery(new TermQuery('field', 'value')) // add a query - ->addAggregation(new Aggregation('test-aggregation',[...])) // add an aggregation -; - -$result = $searchClient->search( [ - 'index' => $indexName, - 'body' => $search->toArray() + ->setSize(10) + ->setFrom(0) + ->setSource(['field']) + ->addSort(new FieldSort('field', 'asc')) + ->addQuery(new TermQuery('field', 'value')) + ->addAggregation(new Aggregation('test-aggregation', [...])); + +$result = $searchClient->search([ + 'index' => $indexName, + 'body' => $search->toArray() ]); ``` ## Query Models -The query models are used to create a query for the search. They can be used to create any query which is supported by OpenSearch or Elasticsearch. - ### BoolQuery -Represents a boolean query. It can be used to combine multiple queries with boolean operators. See documentation for [OpenSearch](https://opensearch.org/docs/latest/query-dsl/compound/bool/) or [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) for more details. +Combines multiple queries with boolean operators. +See [OpenSearch](https://opensearch.org/docs/latest/query-dsl/compound/bool/) +or [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) +documentation. #### Basic usage @@ -72,7 +73,8 @@ $boolQuery = new BoolQuery([ ]); ``` -#### Add additional conditions +#### Adding conditions + ```php $boolQuery = new BoolQuery(); $boolQuery->addCondition('must', [ @@ -80,8 +82,8 @@ $boolQuery->addCondition('must', [ ]); ``` +#### Merging queries -#### Merge multiple queries ```php $boolQueryA = new BoolQuery([ 'should' => [ @@ -95,11 +97,12 @@ $boolQueryB = new BoolQuery([ ], ]); -// this will result in a query with two "should" conditions +// Results in a query with two "should" conditions $boolQueryA->merge($boolQueryB); ``` -#### Use other queries in sub queries +#### Using query objects in sub-queries + ```php $boolQuery = new BoolQuery([ 'should' => [ @@ -111,54 +114,55 @@ $boolQuery = new BoolQuery([ ### TermFilter -The term filter combines a boolean query with a term query. It can be used to filter the search results by a term. +Combines a boolean query with a term query to filter results by an exact term: ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch\Query\TermFilter; + $termFilter = new TermFilter('field', 'value'); ``` ### TermsFilter -The terms filter combines a boolean query with a terms query. It can be used to filter the search results by multiple term. +Combines a boolean query with a terms query to filter by multiple terms. ### WildcardFilter -The wildcard filter combines a boolean query with a wildcard query. It can be used to filter the search results by terms using * as wildcard. +Combines a boolean query with a wildcard query. Use `*` as wildcard: ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch\Query\WildcardFilter; + $wildcardFilter = new WildcardFilter('field', 'value*'); ``` -It is possible to influence the wildcard filter behaviour by setting additional options. Take a look at the constructor of the `WildcardFilter` class for more details. +See the `WildcardFilter` constructor for additional options. ### DateFilter -The date filter can be used to filter the search results by a date range or exact date. +Filter results by a date range or exact date: ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch\Query\DateFilter; -// date range +// Date range $dateFilter = new DateFilter('datefield', strtotime('2000-01-01'), strtotime('2099-12-31')); -// exact date +// Exact date $dateFilter = new DateFilter('datefield', null, null, strtotime('2000-01-01')); ``` -The date filter rounds the timestamps to full days by default. If you want to use exact timestamps, you can set the `roundToDay` option to `false`. +By default, timestamps are rounded to full days. Disable rounding for exact timestamps: ```php -// exact timestamp -$dateFilter = new DateFilter('datefield', null, null, strtotime('2000-01-01 12:00:00'), false); +$dateFilter = new DateFilter( + 'datefield', null, null, strtotime('2000-01-01 12:00:00'), false +); ``` - - ### Generic Query -The generic `Query` model can be used to create any query which is supported by OpenSearch or Elasticsearch. It can be used to create custom queries which are not covered by the other query models. +Build any query supported by OpenSearch or Elasticsearch: ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch\Query\Query; @@ -175,10 +179,9 @@ $rangeQuery = new Query('range', [ ]); ``` - ## Aggregation Model -The aggregation model is used to create an aggregation for the search. It can be used to create any aggregation which is supported by OpenSearch or Elasticsearch. It's just a simple wrapper class without any special logic. +Build any aggregation supported by OpenSearch or Elasticsearch: ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch\Aggregation\Aggregation; diff --git a/doc/04_Searching_For_Data_In_Index/08_Permissions_Workspaces/README.md b/doc/04_Searching_For_Data_In_Index/08_Permissions_Workspaces/README.md index 966afdd0..8e53dafb 100644 --- a/doc/04_Searching_For_Data_In_Index/08_Permissions_Workspaces/README.md +++ b/doc/04_Searching_For_Data_In_Index/08_Permissions_Workspaces/README.md @@ -1,21 +1,33 @@ +--- +title: Permissions and Workspaces +description: How the Generic Data Index search services enforce user permissions and workspace restrictions. +--- + # Permissions and Workspaces -The Generic Data Index bundle respects the user permissions and user workspaces in connection to his roles. -User workspace permissions are respected by the search service and are attached by using the search modifier to the search query. -These permissions are then returned as a part of the search result item. +The Generic Data Index search services respect user permissions and workspace +configurations associated with their roles. + +The search services attach workspace permissions to queries via search modifiers +and include them in each result item. ## Permission Manipulation -If there is a need to manipulate the workspace permissions, e.g. for specific asset path, it is possible to do so via events: -- `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\PermisisonEvent` for data objects -- `Pimcore\Bundle\GenericDataIndexBundle\Event\Asset\PermisisonEvent` for assets -- `Pimcore\Bundle\GenericDataIndexBundle\Event\Document\PermisisonEvent` for documents -You can define your event listener and adapt the permissions as needed. In this example we want to restrict `view` and `list` permissions for a specific asset path: +Override workspace permissions for specific elements via events: + +- `Pimcore\Bundle\GenericDataIndexBundle\Event\Asset\PermissionEvent` (assets) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\PermissionEvent` (data objects) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\Document\PermissionEvent` (documents) + +### Example: Restrict Asset Permissions by Path + +This event subscriber restricts `view` and `list` permissions for a specific asset path: + ```php getPermissions(); - $asset = $event->getElement(); + if ($asset->getFullPath() === '/path/to/your/asset') { $permissions->setView(false); $permissions->setList(false); @@ -45,5 +57,6 @@ class AssetPermissionSubscriber implements EventSubscriberInterface ``` ## User Permissions -Additionally, to the workspace permissions, the user permissions are also respected by the search service. -The user permissions (assets, objects, documents) are checked before the search query is constructed. \ No newline at end of file + +In addition to workspace permissions, the search services check user-level permissions +(assets, objects, documents) before constructing the search query. diff --git a/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/03_Use_PQL_as_Developer.md b/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/03_Use_PQL_as_Developer.md index 4f79fb54..fa535da0 100644 --- a/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/03_Use_PQL_as_Developer.md +++ b/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/03_Use_PQL_as_Developer.md @@ -1,42 +1,54 @@ -# Use Pimcore Query Language (PQL) as a Developer +--- +title: Use PQL as a Developer +description: Execute PQL queries programmatically using search modifiers or the PQL processor. +keywords: + - PqlFilter + - query processor + - developer +--- -## Execute searches based on PQL queries +# Use PQL as a Developer -If you want to use the Pimcore Query Language (PQL) as a developer to search for data in the Pimcore Generic Data Index, you can use one of the following methods: +## Execute Searches with PQL -#### 1. Search Modifier for the Generic Data Index search services +### Option 1: PqlFilter Search Modifier -You can use the [PqlFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Model/Search/Modifier/QueryLanguage/PqlFilter.php) search modifier to filter search results based on a PQL query. The `PqlFilter` search modifier can be used with the search services provided by the Generic Data Index bundle. Take a look at the [Search Services](../README.md) documentation for details. +Use the +[PqlFilter](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Model/Search/Modifier/QueryLanguage/PqlFilter.php) +search modifier with the Generic Data Index search services. +See the [Search Services](../README.md) documentation. -#### 2. Direct use of the PQL processor to get the search query +### Option 2: PQL Processor (Direct Query) -Use the `Pimcore\Bundle\GenericDataIndexBundle\QueryLanguage\ProcessorInterface` together with the `Pimcore\Bundle\GenericDataIndexBundle\Service\SearchIndex\IndexEntityServiceInterface` to process a PQL query. +Use `ProcessorInterface` together with `IndexEntityServiceInterface` +to process a PQL query directly: ```php -// inject both services via Symfony dependency injection /** @var \Pimcore\Bundle\GenericDataIndexBundle\QueryLanguage\ProcessorInterface $queryLanguageProcessor */ /** @var \Pimcore\Bundle\GenericDataIndexBundle\Service\SearchIndex\IndexEntityServiceInterface $indexEntityService */ $query = $queryLanguageProcessor->process( - 'color = "red" or color = "blue"', // The PQL query - $indexEntityService->getByEntityName('Car') // 'Asset', 'Document' or the name of the data object class + 'color = "red" or color = "blue"', + $indexEntityService->getByEntityName('Car') // 'Asset', 'Document', or data object class name ); -// $query is now a valid search index query array which can be used to search in the index +// $query is a valid search index query array ``` ## Exception Handling -In both cases, the PQL processor will throw an exception if the PQL query is invalid. The exception message will contain detailed information about the error. Especially when you would like to allow users to enter PQL queries, you should catch the exception and provide a user-friendly error feedback. +The PQL processor throws a `ParsingException` for invalid queries. +The exception contains the error details and the position of the syntax error. +Catch this exception to provide user-friendly error feedback, especially when +allowing end users to enter PQL queries. -##### Example +### Example -This example will produce a error message like this: +This invalid query produces a syntax error: ![PQL Syntax Error](../../img/pql-syntax-error.png) ```php -## Catching the exception use Pimcore\Bundle\GenericDataIndexBundle\Exception\QueryLanguage\ParsingException; try { @@ -44,11 +56,10 @@ try { and color "red"'; $query = $queryLanguageProcessor->process( - $pqlQuery, // The PQL query - $indexEntityService->getByEntityName('Car') // 'Asset', 'Document' or the name of the data object class + $pqlQuery, + $indexEntityService->getByEntityName('Car') ); } catch (ParsingException $e) { - // Provide user-friendly error feedback return $twig->render('pql-syntax-error.html.twig', [ 'error' => $e->getMessage(), 'syntaxBeforeError' => substr($e->getQuery(), 0, $e->getPosition()), @@ -57,15 +68,16 @@ try { } ``` - ```twig {# pql-syntax-error.html.twig #} - - + -
@@ -92,7 +103,7 @@ try {
{{ syntaxBeforeError|nl2br }} - ⇧ + ⇧ {{ syntaxAfterError|nl2br }}
diff --git a/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/README.md b/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/README.md index 1d11727b..f0668ac5 100644 --- a/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/README.md +++ b/doc/04_Searching_For_Data_In_Index/09_Pimcore_Query_Language/README.md @@ -1,18 +1,30 @@ +--- +title: Pimcore Query Language (PQL) +description: PQL syntax reference for querying data objects, assets, and documents in the Generic Data Index. +keywords: + - PQL + - Pimcore Query Language + - query language + - search syntax + - filter + - relation filter + - query string +--- + # Pimcore Query Language (PQL) -Pimcore Query Language (PQL) is a query language that allows you to search for data in the Pimcore Generic Data Index. It is a simple and powerful query language that allows you to search for data using a wide range of search criteria. +Pimcore Query Language (PQL) defines a query syntax for searching data objects, assets, +and documents in the Generic Data Index. ## Syntax -Description of the PQL syntax: - ``` CONDITION = EXPRESSION | CONDITION ("AND" | "OR") CONDITION EXPRESSION = "(" CONDITION ")" | COMPARISON | QUERY_STRING_QUERY COMPARISON = FIELDNAME OPERATOR VALUE | RELATION_COMPARISON RELATION_COMPARISON = RELATION_FIELD_NAME OPERATOR VALUE -FIELDNAME = IDENTIFIER{.IDENTIFIER} -RELATION_FIELD_NAME = FIELDNAME:ENTITYNAME.FIELDNAME +FIELDNAME = IDENTIFIER{.IDENTIFIER} +RELATION_FIELD_NAME = FIELDNAME:ENTITYNAME.FIELDNAME IDENTIFIER = [a-zA-Z_]\w* ENTITYNAME = [a-zA-Z_]\w* OPERATOR = "="|"!="|"<"|">"|">="|"<="|"LIKE"|"NOT LIKE" @@ -24,35 +36,33 @@ QUERY_STRING_QUERY = "QUERY('" STRING "')" ### Operators -| Operator | Description | Examples | -|------------|-------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------| -| `=` | equal (case-sensitive) | `field = "value"` | -| `!=` | not equal (case-sensitive) | `field != "value"` | -| `<` | smaller than | `field < 100` | -| `<=` | smaller or equal than | `field <= 100` | -| `=>` | bigger or equal than | `field >= 100` | -| `>` | bigger than | `field > 100` | -| `LIKE` | equal with wildcard support (case-insensitive)
* matches zero or more characters
? matches any single character | `field like "val*"`
`field like "val?e"` | -| `NOT LIKE` | not equal with wildcard support (case-insensitive)
* matches zero or more characters
? matches any single character | `field not like "val*"`
`field not like "val?e"` | +| Operator | Description | Examples | +|----------|-------------|----------| +| `=` | Equal (case-sensitive) | `field = "value"` | +| `!=` | Not equal (case-sensitive) | `field != "value"` | +| `<` | Less than | `field < 100` | +| `<=` | Less than or equal to | `field <= 100` | +| `>=` | Greater than or equal to | `field >= 100` | +| `>` | Greater than | `field > 100` | +| `LIKE` | Equal with wildcard support (case-insensitive). `*` matches zero or more characters, `?` matches one. | `field like "val*"` | +| `NOT LIKE` | Not equal with wildcard support (case-insensitive). `*` matches zero or more characters, `?` matches one. | `field not like "val*"` | ### Null/Empty Values -To search for null and empty values use the `NULL`/`EMPTY` keywords. Those can be used together with the `=` and `!=` operators to search for fields without value. Keep in mind that there can be a difference between `NULL` and an empty string. The `EMPTY` keyword is a shortcut for `NULL` or an empty string. - -**Examples:** +Use the `NULL` and `EMPTY` keywords with `=` and `!=` operators to search for +fields without values. `NULL` matches null values; `EMPTY` matches both null +and empty strings. ``` field = NULL field != NULL -field = EMPTY # same as: field = NULL OR field = '' -field != EMPTY # same as: field != NULL AND field != '' +field = EMPTY # equivalent to: field = NULL OR field = '' +field != EMPTY # equivalent to: field != NULL AND field != '' ``` ### AND / OR / Brackets -You can combine multiple conditions using the `AND` and `OR` operators. You can also use brackets to group conditions. - -**Examples:** +Combine conditions with `AND` and `OR` operators. Use brackets to group conditions: ``` field1 = "value1" AND field2 = "value2" @@ -60,14 +70,10 @@ field1 = "value1" AND (field2 = "value2" OR field3 = "value3") (field1 = "value1" AND (field2 = "value2" OR field3 = "value3")) OR field4 = "value4" ``` - ### Relation Filters -Supports filtering along relations with following notation: - -`:.` - -**Examples:** +Filter along relations using the notation +`:.`: ``` main_image:Asset.type @@ -75,15 +81,16 @@ category:Category.name manufacturer:Company.country ``` -The entity name can be either 'Asset', 'Document' or the name of the data object class. +The entity name must be `Asset`, `Document`, or a data object class name. ### Field Names -The field names are named and structured the same way as in the search index. Nested field names are supported with a dot ('.') notation. -As described [here](../../05_Extending_Data_Index/06_Extend_Search_Index.md) the fields are separated into three sections (system_fields, standard_fields and custom_fields) and depending on the data type of a attribute the attribute value could be a nested structure with sub-attributes. +Field names match the search index structure. Nested fields use dot (`.`) notation. +Fields are organized into three sections (`system_fields`, `standard_fields`, +`custom_fields`) as described in [Extending Search Index](../../05_Extending_Data_Index/06_Extend_Search_Index.md). +Depending on the data type, attribute values may contain nested sub-attributes. - -**Examples for field names with their full path in the index:** +**Full path examples:** ``` system_fields.id @@ -92,9 +99,12 @@ standard_fields.my_relation_field.asset standard_fields.description.de ``` -To simplify the usage of the PQL the field names can be used without the full path in most of the cases. The PQL will automatically search in the index structure and try to detect the correct field. So normally it's enough to use the technical field name like used for example in the data object class or asset metadata attribute. +PQL automatically resolves short field names to their full indexed path. +This works for all unambiguous field names. If a field name exists in multiple sections +(e.g. both `system_fields` and `standard_fields`), use the full path. +Query string queries (`QUERY()`) also require full paths. -**Above examples for field names without the full path:** +Use the technical field name as defined in the data object class or asset metadata: ``` id @@ -103,35 +113,60 @@ my_relation_field description.de ``` -Localized fields can be accessed in the form 'field_name.locale' (e.g. description.de). +Access localized fields with `field_name.locale` (e.g. `description.de`). + +### Query String Queries + +Pass +[OpenSearch](https://opensearch.org/docs/latest/query-dsl/full-text/query-string) +or +[Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html) +query string queries directly to the index using `QUERY()`: + +``` +QUERY("standard_fields.color:(red OR blue)") +``` + +See the +[OpenSearch query string syntax](https://opensearch.org/docs/latest/query-dsl/full-text/query-string/#query-string-syntax) +or +[Elasticsearch query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax) +for details. -### Query String Query Filters +:::caution -The PQL allows passing `query string queries` of [OpenSearch](https://opensearch.org/docs/latest/query-dsl/full-text/query-string) or [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html) directly to the index. -The query string query syntax provides even more flexibility to define the search criteria. Take a look at the [OpenSearch documentation](https://opensearch.org/docs/latest/query-dsl/full-text/query-string/#query-string-syntax) or [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax) respectively for more details. +Automatic field name detection is not supported for query string queries. +Use full field paths. -**Caution**: The automatic field detection is not supported for query string queries. So you have to use the full path for the field names. +::: ### Example PQL Queries -All examples are based on the `Car` data object class of the [Pimcore Demo](https://pimcore.com/en/try). +All examples reference the `Car` data object class from the +[Pimcore Demo](https://pimcore.com/en/try). -| Query | Description | -|---------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------| -| `series = "E-Type" AND (color = "green" OR productionYear < 1965)` | All E-Type models which are green or produced before 1965. | -| `manufacturer:Manufacturer.name = "Alfa" AND productionYear > 1965` | All Alfa cars produced after 1965. | -| `genericImages:Asset.fullPath LIKE "/Car Images/vw/*"` | All cars with a image linked in the `genericImages` image gallery which is contained in the asset folder `/Car Images/vw`. | -| `color = "red" OR color = "blue"` | All red or blue cars using standard PQL syntax. | -| `series = empty AND color="red"` | All models where the series is empty and the color is red. | -| `License.expiryDate <= 'now+2d/d'` | All elements whose license expires in two days - if the field is a date field you can use [date-math](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math) syntax of elastic search/opensearch.| -| `Query("standard_fields.color:(red OR blue)")` | All red or blue cars using simple query string syntax. | +| Query | Description | +|-------|-------------| +| `series = "E-Type" AND (color = "green" OR productionYear < 1965)` | All E-Type models that are green or produced before 1965 | +| `manufacturer:Manufacturer.name = "Alfa" AND productionYear > 1965` | All Alfa cars produced after 1965 | +| `genericImages:Asset.fullPath LIKE "/Car Images/vw/*"` | All cars with images in the `/Car Images/vw` asset folder | +| `color = "red" OR color = "blue"` | All red or blue cars | +| `series = empty AND color="red"` | All models with empty series and red color | +| `License.expiryDate <= 'now+2d/d'` | All elements whose license expires within two days (uses [date math](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math) syntax) | +| `Query("standard_fields.color:(red OR blue)")` | All red or blue cars using query string syntax | ## Limitations -* When searching for related elements the maximum possible results amount of sub queries is 65.000, see also `terms query` [OpenSearch](https://opensearch.org/docs/latest/query-dsl/term/terms/) or [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html) documentation respectively. -* Filtering for asset metadata fields is only possible if they are defined as predefined asset metadata or via the asset metadata class definitions bundle. Custom asset metadata fields directly defined on single assets are not supported. -* Reserved keywords (`AND`, `OR`, `LIKE`, `NOT LIKE`, `NULL`, `EMPTY`) cannot be used as field names. +- **Relation sub-query limit:** Maximum 65,000 results for related element sub-queries. + See `terms query` docs for + [OpenSearch](https://opensearch.org/docs/latest/query-dsl/term/terms/) / + [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html). +- **Asset metadata filtering:** Only predefined asset metadata and metadata from the + asset metadata class definitions bundle are supported. Custom metadata defined + directly on individual assets is not filterable. +- **Reserved keywords:** `AND`, `OR`, `LIKE`, `NOT LIKE`, `NULL`, `EMPTY` cannot + be used as field names. ## Further Reading -- [Use PQL as a Developer](./03_Use_PQL_as_Developer.md). +- [Use PQL as a Developer](./03_Use_PQL_as_Developer.md) diff --git a/doc/04_Searching_For_Data_In_Index/README.md b/doc/04_Searching_For_Data_In_Index/README.md index 0317afd1..ed46bad8 100644 --- a/doc/04_Searching_For_Data_In_Index/README.md +++ b/doc/04_Searching_For_Data_In_Index/README.md @@ -1,45 +1,59 @@ -# Searching For Data In Index +--- +title: Searching For Data In Index +description: Search assets, data objects, and documents using the Generic Data Index search services. +--- -The Generic Data Index bundle adds standardized and flexible services to search data from the search indices. +# Searching For Data In Index -Each search is based on a search service (depending on the element type) and a search model defining the search query. The search models can be created with the [SearchProviderInterface](https://github.com/pimcore/generic-data-index-bundle/blob/2.0/src/Service/Search/SearchService/SearchProviderInterface.php) +The Generic Data Index bundle provides standardized search services for querying +assets, data objects, and documents from the search indices. -The regular way to search for assets, data objects or documents is to use the related search service. +Each search uses a type-specific search service and a search model that defines the query. +Create search models with the +[SearchProviderInterface](https://github.com/pimcore/generic-data-index-bundle/blob/2026.x/src/Service/Search/SearchService/SearchProviderInterface.php). -## Asset Search Service +## Asset Search -### Example usage +Load all assets from the root folder (parent ID 1), ordered by full path: -- Example: This example loads all assets from the root folder (parent ID 1) and orders them by their full path. ```php use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\SearchProviderInterface; use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\Asset\AssetSearchServiceInterface; use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Filter\Tree\ParentIdFilter; use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Sort\Tree\OrderByFullPath; -public function searchAction(SearchProviderInterface $searchProvider, AssetSearchServiceInterface $asserSearchService) -{ +public function searchAction( + SearchProviderInterface $searchProvider, + AssetSearchServiceInterface $assetSearchService +) { $assetSearch = $searchProvider->createAssetSearch() ->addModifier(new ParentIdFilter(1)) ->addModifier(new OrderByFullPath()) ->setPageSize(50) ->setPage(1); - $searchResult = $asserSearchService->search($assetSearch); + $searchResult = $assetSearchService->search($assetSearch); } ``` -## Data Object Search Service +## Data Object Search + +Load all data objects from the root folder for a specific class definition, +ordered by full path: -- Example: This example loads all data objects from the root folder (parent ID 1) with a specific class definition and orders them by their full path. ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Filter\Tree\ParentIdFilter; use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Sort\Tree\OrderByFullPath; use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\DataObject\DataObjectSearchServiceInterface; use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\SearchProviderInterface; +use Pimcore\Model\DataObject\ClassDefinition; + +public function searchAction( + SearchProviderInterface $searchProvider, + DataObjectSearchServiceInterface $dataObjectSearchService +) { + $carClassDefinition = ClassDefinition::getByName('Car'); -public function searchAction(SearchProviderInterface $searchProvider, DataObjectSearchServiceInterface $dataObjectSearchService) -{ $dataObjectSearch = $searchProvider->createDataObjectSearch() ->addModifier(new ParentIdFilter(1)) ->addModifier(new OrderByFullPath()) @@ -51,18 +65,20 @@ public function searchAction(SearchProviderInterface $searchProvider, DataObject } ``` +## Document Search -## Document Search Service +Load all documents from the root folder, ordered by full path: -- Example: This example loads all documents from the root folder (parent ID 1) and orders them by their full path. ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Filter\Tree\ParentIdFilter; use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Sort\Tree\OrderByFullPath; use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\Document\DocumentSearchServiceInterface; use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\SearchProviderInterface; -public function searchAction(SearchProviderInterface $searchProvider, DocumentSearchServiceInterface $documentSearchService) -{ +public function searchAction( + SearchProviderInterface $searchProvider, + DocumentSearchServiceInterface $documentSearchService +) { $documentSearch = $searchProvider->createDocumentSearch() ->addModifier(new ParentIdFilter(1)) ->addModifier(new OrderByFullPath()) @@ -73,13 +89,19 @@ public function searchAction(SearchProviderInterface $searchProvider, DocumentSe } ``` -## Element Search Service +## Element Search (Cross-Type) + +The element search queries assets, data objects, and documents simultaneously. + +:::info -The element search service can be used to search for assets, data objects and documents at the same time. +The element search does not compute `hasChildren` attributes. `hasChildren` always +returns `false` for all elements in element search results. -**Hint:** the element search does not support the calculation of the `hasChildren` attributes. This means that the `hasChildren` attribute will always be `false` for all elements. +::: + +Load all elements required by asset ID 123, ordered by full path: -- Example: This example loads all elements which are required by the asset ID 123 and orders them by their full path. ```php use Pimcore\Bundle\GenericDataIndexBundle\Enum\SearchIndex\ElementType; use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Filter\Dependency\RequiredByFilter; @@ -87,8 +109,10 @@ use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Sort\Tree\OrderB use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\Element\ElementSearchServiceInterface; use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\SearchProviderInterface; -public function searchAction(SearchProviderInterface $searchProvider, ElementSearchServiceInterface $elementSearchService) -{ +public function searchAction( + SearchProviderInterface $searchProvider, + ElementSearchServiceInterface $elementSearchService +) { $elementSearch = $searchProvider->createElementSearch() ->addModifier(new RequiredByFilter(123, ElementType::ASSET)) ->addModifier(new OrderByFullPath()) @@ -101,12 +125,14 @@ public function searchAction(SearchProviderInterface $searchProvider, ElementSea ## Search Modifiers -To influence the data which gets fetched its possible to use so-called search modifiers. -Find out details about search modifiers in the [search modifiers documentation](05_Search_Modifiers/README.md). There you will also find information on how to create your own custom search modifiers. +Search modifiers filter, sort, and aggregate search results. See the +[Search Modifiers documentation](05_Search_Modifiers/README.md) for the full reference +and instructions on creating custom modifiers. -## Retrieve IDs only instead of full objects +## Retrieving IDs Only -If you only want to retrieve your search results as list of IDs for your `AssetSearch`, `DataObjectSearch` or `DocumentSearch` you can use the `SearchResultIdListServiceInterface` to retrieve the IDs only. +Use `SearchResultIdListServiceInterface` to retrieve search results as ID lists +instead of full objects: ```php use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Filter\Tree\ParentIdFilter; @@ -114,36 +140,52 @@ use Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Modifier\Sort\Tree\OrderB use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\SearchProviderInterface; use Pimcore\Bundle\GenericDataIndexBundle\Service\Search\SearchService\SearchResultIdListServiceInterface; -public function searchAction(SearchProviderInterface $searchProvider, SearchResultIdListServiceInterface $searchResultIdListService) -{ +public function searchAction( + SearchProviderInterface $searchProvider, + SearchResultIdListServiceInterface $searchResultIdListService +) { $dataObjectSearch = $searchProvider->createDataObjectSearch() ->addModifier(new ParentIdFilter(1)) ->addModifier(new OrderByFullPath()) ->setPageSize(50) ->setPage(1); - $allIds = $searchResultIdListService->getAllIds($dataObjectSearch); // returns an array of IDs for the full search result without pagination - $idsOnPage = $searchResultIdListService->getIdsForCurrentPage($dataObjectSearch); // returns an array of IDs for the current page + // All IDs for the full search result (ignoring pagination) + $allIds = $searchResultIdListService->getAllIds($dataObjectSearch); + + // IDs for the current page only + $idsOnPage = $searchResultIdListService->getIdsForCurrentPage($dataObjectSearch); } ``` ## Default Search Models -The search services mentioned above offer a flexible and structured way to search for assets, data objects and documents. Nevertheless, if there are requirements which are not covered by the search services it might be needed to develop your own customized search queries. Default search models offer a streamlined way for executing such customized search queries. They are also used by the search services internally to create the executed search queries. -Take a look at the dedicated [Default search models documentation](06_Default_Search_Models/README.md) to find out more. +To build custom OpenSearch/Elasticsearch queries beyond what the search services offer, +use default search models. The search services +use these models internally. + +See the [Default Search Models documentation](06_Default_Search_Models/README.md). ## Permissions -The search service respects the user permissions and user workspaces in connection to his roles. -Details about permissions and workspaces can be found in the [permissions and workspaces documentation](08_Permissions_Workspaces/README.md). +The search services respect user permissions and workspace configurations. + +See [Permissions and Workspaces](08_Permissions_Workspaces/README.md). ## Pimcore Query Language (PQL) -The [Pimcore Query Language (PQL)](./09_Pimcore_Query_Language/README.md) is a query language which can be used to provide the user a flexible way to define search criteria for data objects, assets and documents. +[PQL](./09_Pimcore_Query_Language/README.md) defines a query syntax +for searching data objects, assets, and documents. ## Debug Search Queries -To debug the Search queries which are created by the search service, it is possible to use the following magic parameter in the URL (when debug mode is enabled): -| Get Parameter | Description | -|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `debug-search-queries` | Will change the response of the called URL and outputs all executed search queries.

It is possible to influence the output verbosity with the value of the parameter (1=normal, 2=verbose, 3=very verbose) | +When debug mode is enabled, add the `debug-search-queries` GET parameter to any URL +to inspect the executed search queries: + +| Value | Output verbosity | +|-------|-----------------| +| `1` | Normal | +| `2` | Verbose | +| `3` | Very verbose | + +The response is replaced with the search query output. diff --git a/doc/05_Extending_Data_Index/06_Extend_Search_Index.md b/doc/05_Extending_Data_Index/06_Extend_Search_Index.md index 12b9e130..25d65453 100644 --- a/doc/05_Extending_Data_Index/06_Extend_Search_Index.md +++ b/doc/05_Extending_Data_Index/06_Extend_Search_Index.md @@ -1,78 +1,68 @@ -# Extending Search Index +--- +title: Extending the Search Index +description: Add custom data attributes to the search index using UpdateIndexDataEvent and ExtractMappingEvent. +--- -## Extending Search Index via Events +# Extending the Search Index -The regular index update process stores a defined set of standard data types in the data index which makes it -possible to find, filter, sort and list them.. +## Adding Custom Fields via Events -It is possible to extend the index with custom attributes if needed. For this purpose the following events exist. You -will find code examples below. +The index update process stores system fields and supported data object/asset field types +by default. Extend the index with custom attributes using the following events. ### UpdateIndexDataEvent -This event can be used to store additional fields in the search index. Depending on if you would like to index additional -data for assets or data objects use one of the following two events. +Store additional fields in the search index. Use the event matching your element type: -* `Pimcore\Bundle\GenericDataIndexBundle\Event\Asset\UpdateIndexDataEvent` (assets) -* `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\UpdateIndexDataEvent` (concrete data object classes) -* `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\UpdateFolderIndexDataEvent` (data object folders) -* `Pimcore\Bundle\GenericDataIndexBundle\Event\Document\UpdateIndexDataEvent` (documents) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\Asset\UpdateIndexDataEvent` (assets) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\UpdateIndexDataEvent` (concrete data objects) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\UpdateFolderIndexDataEvent` (data object folders) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\Document\UpdateIndexDataEvent` (documents) -If you take a look at the source of an indexed document within search index you will find a structure like this: +An indexed document in the search index has this structure: ```json { - "system_fields" : { - "id" : 145, - "creationDate" : "2019-05-24T15:42:20+0200", - "modificationDate" : "2019-08-23T15:15:54+0200", - "type" : "image", - "key" : "abandoned-automobile-automotive-1082654.jpg", - ... - }, - "standard_fields" : [ ... ], - "custom_fields" : [ ] + "system_fields": { + "id": 145, + "creationDate": "2019-05-24T15:42:20+0200", + "modificationDate": "2019-08-23T15:15:54+0200", + "type": "image", + "key": "abandoned-automobile-automotive-1082654.jpg" + }, + "standard_fields": [ ... ], + "custom_fields": [ ] } ``` -This is used to separate the data into three sections: +The three sections are: -###### system_fields - -Base system fields which are the same for all assets or data objects (like id, creationDate, fullPath...). - -###### standard_fields - -All data object or asset metadata types which are supported out of the box depending on your data model. - -###### custom_fields - -This is the place where you are able to add data via the `UpdateIndexDataEvent`. As soon as additional fields are added -they are searchable through the full text search (depending on the mapping of the fields). +- **system_fields** - Base fields common to all elements (id, creationDate, fullPath, etc.) +- **standard_fields** - Data object fields or asset metadata supported out of the box +- **custom_fields** - Custom data added via `UpdateIndexDataEvent`. + Added fields are automatically included in full text search (depending on mapping). ### ExtractMappingEvent -With this event it's possible to define the mapping (for elasticsearch see [this mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) and for openSearch please refer to [mapping](https://opensearch.org/docs/latest/field-types/)) -of the additional custom fields. Again there are separate events for assets and data objects. - -* `Pimcore\Bundle\GenericDataIndexBundle\Event\Asset\ExtractMappingEvent` (assets) -* `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\ExtractMappingEvent` (concrete data object classes) -* `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\ExtractFolderMappingEvent` (data object folders) -* `Pimcore\Bundle\GenericDataIndexBundle\Event\Document\ExtractMappingEvent` (documents) +Define the search engine mapping for custom fields (see +[Elasticsearch mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) +or [OpenSearch field types](https://opensearch.org/docs/latest/field-types/)). +Use the event matching your element type: +- `Pimcore\Bundle\GenericDataIndexBundle\Event\Asset\ExtractMappingEvent` (assets) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\ExtractMappingEvent` (concrete data objects) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\DataObject\ExtractFolderMappingEvent` (data object folders) +- `Pimcore\Bundle\GenericDataIndexBundle\Event\Document\ExtractMappingEvent` (documents) -### Example 1: Assets +## Example 1: Asset File Size Category -The following example creates an EventSubscriber which adds another custom field. The logic applies to assets and divides the assets into file size groups: - -* small: < 300KB -* medium: 300KB - 3MB -* big: > 3MB +This event subscriber categorizes assets by file size into `small` (< 300 KB), +`medium` (300 KB - 3 MB), and `big` (> 3 MB): ```php 'onUpdateIndexData', - ExtractMappingEvent::class => 'onExtractMapping', + UpdateIndexDataEvent::class => 'onUpdateIndexData', + ExtractMappingEvent::class => 'onExtractMapping', ]; } - public function onUpdateIndexData(UpdateIndexDataEvent $event) + public function onUpdateIndexData(UpdateIndexDataEvent $event): void { - $asset = $event->getAsset(); + $asset = $event->getElement(); if ($asset instanceof Folder) { return; } - // Ensure that you take the original array and extend it. $customFields = $event->getCustomFields(); - $fileSize = $event->getAsset()->getFileSize(); - $fileSizeSelection = null; - if ($fileSize < 3*1000) { - $fileSizeSelection = 'small'; - } elseif ($fileSize <= 3*1000*1000) { - $fileSizeSelection = 'medium'; - } else { - $fileSizeSelection = 'big'; - } - - $customFields['fileSizeSelection'] = $fileSizeSelection; + $fileSize = $asset->getFileSize(); + $customFields['fileSizeSelection'] = match (true) { + $fileSize < 300_000 => 'small', + $fileSize <= 3_000_000 => 'medium', + default => 'big', + }; $event->setCustomFields($customFields); } - public function onExtractMapping(ExtractMappingEvent $event) + public function onExtractMapping(ExtractMappingEvent $event): void { - // Ensure that you take the original array and extend it. $customFieldsMapping = $event->getCustomFieldsMapping(); - /** - * Take a look at the docs how mapping works. - * A 'keyword' field would be best for regular select and multi select filters. - * For full text search it is possible to define sub-fields with special search index analyzers too. - */ + // 'keyword' works well for select and multi-select filters $customFieldsMapping['fileSizeSelection'] = [ 'type' => 'keyword' ]; @@ -132,92 +110,75 @@ class FileSizeIndexSubscriber implements EventSubscriberInterface $event->setCustomFieldsMapping($customFieldsMapping); } } - - ``` ```yaml -# service definition - +# config/services.yaml services: _defaults: autowire: true - AppBundle\EventListener\FileSizeIndexSubscriber: + App\EventListener\FileSizeIndexSubscriber: tags: - { name: kernel.event_subscriber } ``` +## Example 2: Data Object Variant Count -### Example 2: Data Objects - -In this example a "User Owner" field will be provided for car documents. -"Owner" is defined as Pimcore username of the creator of the car data object. +This event subscriber adds a `numberOfVariants` field to `Car` data objects, +counting direct children (variants) of each car: ```php 'onUpdateIndexData', - ExtractMappingEvent::class => 'onExtractMapping', + UpdateIndexDataEvent::class => 'onUpdateIndexData', + ExtractMappingEvent::class => 'onExtractMapping', ]; } - public function onUpdateIndexData(UpdateIndexDataEvent $event) + public function onUpdateIndexData(UpdateIndexDataEvent $event): void { - $car = $event->getDataObject(); + $car = $event->getElement(); if (!$car instanceof Car) { return; } - // Ensure that you take the original array and extend it. $customFields = $event->getCustomFields(); - $customFields['numberOfVariants'] = count($car->getChildren() ?? []); - $event->setCustomFields($customFields); } - public function onExtractMapping(ExtractMappingEvent $event) + public function onExtractMapping(ExtractMappingEvent $event): void { if ($event->getClassDefinition()->getId() !== 'CAR') { return; } - // Ensure that you take the original array and extend it. $customFieldsMapping = $event->getCustomFieldsMapping(); - - /** - * Take a look at the docs how mapping works. - * A 'keyword' field would be best for regular select and multi select filters. - * For full text search it is possible to define sub-fields with special search index analyzers too. - */ $customFieldsMapping['numberOfVariants'] = [ 'type' => 'integer' ]; - $event->setCustomFieldsMapping($customFieldsMapping); } } - ``` -#### Update index mapping and data +## Rebuild Index After Changes -Call the following console command as soon as the event subscriber is set up in the symfony container configuration. +After registering an event subscriber, rebuild the search index: ```bash -./bin/console generic-data-index:update:index -r -``` \ No newline at end of file +bin/console generic-data-index:update:index -r +``` diff --git a/doc/05_Extending_Data_Index/07_Custom_Field_Definition_Adapters.md b/doc/05_Extending_Data_Index/07_Custom_Field_Definition_Adapters.md new file mode 100644 index 00000000..6ba4f600 --- /dev/null +++ b/doc/05_Extending_Data_Index/07_Custom_Field_Definition_Adapters.md @@ -0,0 +1,81 @@ +--- +title: Custom Field Definition Adapters +description: Register field definition adapters for custom data object field types in the Generic Data Index. +--- + +# Custom Field Definition Adapters + +When adding a custom data object field type (e.g. via a bundle), the Generic Data Index +needs a field definition adapter to know how to index the field's data. + +## Reusing Existing Adapters + +For simple field types that store string data, reuse the built-in `TextKeywordAdapter`. +Register it as a service with the +`pimcore.generic_data_index.data-object.search_index_field_definition` tag: + +```yaml +services: + _defaults: + autowire: true + autoconfigure: true + + my_bundle.gdi.simple_text_adapter: + class: Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\DefaultSearch\DataObject\FieldDefinitionAdapter\TextKeywordAdapter + shared: false + tags: + - name: "pimcore.generic_data_index.data-object.search_index_field_definition" + type: "simpleText" +``` + +The `type` attribute must match the value returned by your field definition's +`getFieldType()` method. Set `shared: false` because adapters are stateful +(each instance gets a field definition assigned). + +### Available Built-in Adapters + +- `TextKeywordAdapter` - string/text fields (`input`, `textarea`, `select`, + `multiselect`, `wysiwyg`, etc.) +- `NumericAdapter` - numeric fields +- `DateAdapter` - date fields +- `BooleanAdapter` - boolean fields + +See all adapters in +`config/services/search/data-object/field-definition-adapters.yml`. + +## Creating Custom Adapters + +For field types that need custom indexing logic, extend `AbstractAdapter`: + +```php +use Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\DefaultSearch\DataObject\FieldDefinitionAdapter\AbstractAdapter; + +final class MyCustomAdapter extends AbstractAdapter +{ + public function getIndexMapping(): array + { + // Return OpenSearch/Elasticsearch mapping for this field + return [ + 'type' => 'keyword' + ]; + } +} +``` + +Register it with the same service tag as shown above. + +## Rebuilding the Index + +After registering a new adapter, rebuild the search index for the affected class: + +```bash +bin/console generic-data-index:update:index -c CLASS_ID -r +``` + +Replace `CLASS_ID` with the class definition ID (e.g. `7` for the Demo class). + +## Reference + +For a working example, see the +[studio-example-bundle](https://github.com/pimcore/studio-example-bundle) +`simpleText` custom datatype. diff --git a/doc/05_Extending_Data_Index/README.md b/doc/05_Extending_Data_Index/README.md index 978cbaf5..7eaf9e04 100644 --- a/doc/05_Extending_Data_Index/README.md +++ b/doc/05_Extending_Data_Index/README.md @@ -1,8 +1,18 @@ -# Extending Data Index +--- +title: Extending the Data Index +description: Extend the Generic Data Index with custom fields, mappings, and field definition adapters. +--- -Generic Data Index bundle provides possibility to use data indices to handle search, listings, and filters. Consequently, it's crucial to store data from Pimcore elements into data indices. -For guidance on indexing data and ensuring its currency, refer to the [Index Management](../02_Configuration/03_Index_Management.md) section. +# Extending the Data Index + +The Generic Data Index stores Pimcore element data in search indices to power +search, listings, and filters. For guidance on index creation and maintenance, see +[Index Management](../02_Configuration/03_Index_Management.md). ## Further Reading -- [Extending Search Index](./06_Extend_Search_Index.md): - Describes how to add additional data attributes to the data index. \ No newline at end of file + +- [Extending Search Index](./06_Extend_Search_Index.md) - + Add custom data attributes to the search index. +- [Custom Field Definition Adapters](./07_Custom_Field_Definition_Adapters.md) - + Register adapters for custom data object field types so the Generic Data Index + can index and filter them.