Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Generic Data Index

# Pimcore Generic Data Index

The Pimcore Generic Data Index Bundle provides a centralized way to index and search elements (assets, data objects and documents) in Pimcore via indices (e.g OpenSearch, Elasticsearch).
The Pimcore Generic Data Index Bundle provides a centralized way to index and search elements (assets, data objects and documents) in Pimcore via indices (e.g. OpenSearch, Elasticsearch).
It is shipped with the OpenSearch and Elasticsearch clients and provides a central configuration for them in order to be used in other bundles.
This bundle can be extended and customized to fit your specific needs, for example if you would like to extend the search indices with custom attributes.

Expand Down
182 changes: 119 additions & 63 deletions doc/01_Installation/02_Upgrade.md
Original file line number Diff line number Diff line change
@@ -1,86 +1,142 @@
# Upgrade Information
---
title: Upgrade Information
description: Version-specific upgrade instructions and breaking changes for the Generic Data Index bundle.
---

Following steps are necessary during updating to newer versions.
# Upgrade Information

## Upgrade to 2026.1.0
- [Searching] Added `forceReload` parameter to element search service interface `byId()`.
- The bundle installer now implements `PostInstallCommandsProviderInterface` from Pimcore's InstallBundle. This means the post-install command `generic-data-index:update:index -r` is automatically executed during `pimcore:install` when using Install Profiles. Manual execution of this command after `pimcore:bundle:install` is still required as before.
- The messenger transport DSN is now configurable via the `%pimcore.messenger.transport_dsn_prefix%` container parameter instead of being hardcoded to `doctrine://default`. This allows the installer to wire the transport DSN from environment variables (e.g. `PIMCORE_MESSENGER_TRANSPORT_DSN_PREFIX`).
- Added support to `PHP` `8.5`.
- Removed support to `PHP` `8.3` and Symfony `v6`.

### PHP and Dependency Requirements

- Added support for `PHP` `8.5`.
- Removed support for `PHP` `8.3` and Symfony `v6`.

### Removed ExtJS / Admin Classic

- `PimcoreGenericDataIndexBundle` no longer implements `PimcoreBundleAdminClassicInterface`.
- Removed `BundleAdminClassicTrait`.

### Interface Changes

- Added optional `bool $forceReload = false` parameter to the `byId()` method on the following interfaces:
- `AssetSearchServiceInterface::byId(int $id, ?User $user = null, bool $forceReload = false)`
- `DataObjectSearchServiceInterface::byId(int $id, ?User $user = null, bool $forceReload = false)`
- `DocumentSearchServiceInterface::byId(int $id, ?User $user = null, bool $forceReload = false)`
- `ElementSearchServiceInterface::byId(ElementType $elementType, int $id, ?User $user = null, bool $forceReload = false)`

### Installer / Messenger Transport Changes

- The bundle installer now implements `PostInstallCommandsProviderInterface`. The post-install command
`generic-data-index:update:index -r` is automatically executed during `pimcore:install` when using
Install Profiles. Manual execution after `pimcore:bundle:install` is still required.
- The messenger transport DSN is now configurable via the `%pimcore.messenger.transport_dsn_prefix%`
container parameter (env: `PIMCORE_MESSENGER_TRANSPORT_DSN_PREFIX`) instead of being hardcoded to
`doctrine://default`.

## Upgrade to 2.2.0
- [Indexing] Added `id` column as new primary key to `generic_data_index_queue`. Please make sure to execute migrations.
- [Searching] Added `trackTotalHits` parameter to `DefaultSearchService` and `SearchExecutionService`. The default value is true, which means that total hits will always be computed accurately, even if they exceed the search engines threshold for accurate hit calculation. Change this parameter to `null`, to use the default threshold, pass an integer value to set a specific one.

- **[Indexing]** Added `id` column as new primary key to `generic_data_index_queue`.
Run migrations after updating.
- **[Searching]** Added `trackTotalHits` parameter to `DefaultSearchService` and
`SearchExecutionService`. Default value: `true` (always compute accurate total hits,
even beyond the search engine's threshold). Set to `null` to use the engine's default
threshold, or pass an integer for a specific limit.

## Upgrade to 2.1.0

- Added support for Symfony 7
- [Indexing] Added sort index for documents
- [Indexing] Improved indexing of field collections to prevent mapping conflicts when properties have the same name but different types
- Execute the following command to reindex all elements to be able to use all new features:
```bin/console generic-data-index:update:index -r```
- **[Indexing]** Added sort index for documents
- **[Indexing]** Improved field collection indexing to prevent mapping conflicts when
properties share a name but differ in type
- Reindex all elements to use the new features:
```bash
bin/console generic-data-index:update:index -r
```

## Upgrade to 2.0.0
- [Indexing] Added inherited fields indicator to data object indexing
- [Indexing] Added functionality to enqueue dependent items
- [Indexing] Added class ID field for data object elements
- [Indexing] Added prefix for index names of data objects, these names changed from e.g. `pimcore_car` to `pimcore_data-object_car`. Old indexes are **not deleted** automatically. You need to delete them manually if necessary.
- [Searching] Added new `ClassIdsFilter` modifier to search for data object elements by class ID or class name
- Added a new method `isElementLocked()` to the `ElementLockService`, which provides functionality to retrieve element locked status based on the index data
- Execute the following command to reindex all elements to be able to use all new features:
```bin/console generic-data-index:update:index -r```

### BC-Breaks
- Removed deprecated alias `generic-data-index.opensearch-client` and replaced it with `generic-data-index.search-client`
- Removed all deprecated classes from OpenSearch namespaces and replaced them with DefaultSearch namespace instead.
- `Pimcore\Bundle\GenericDataIndexBundle\Model\OpenSearch` -> `Pimcore\Bundle\GenericDataIndexBundle\Model\DefaultSearch`
- `Pimcore\Bundle\GenericDataIndexBundle\Enum\SearchIndex\OpenSearch` -> `Pimcore\Bundle\GenericDataIndexBundle\Enum\SearchIndex\DefaultSearch`
- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\Exception\OpenSearch\SearchFailedException` please use `Pimcore\Bundle\GenericDataIndexBundle\Exception\OpenSearch\SearchFailedException` instead
- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\Attribute\OpenSearch\AsSearchModifierHandler` please use `Pimcore\Bundle\GenericDataIndexBundle\Attribute\Search\AsSearchModifierHandler` instead
- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\OpenSearch\Asset\FieldDefinitionAdapter\AbstractAdapter` please use `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\DefaultSearch\Asset\FieldDefinitionAdapter\AbstractAdapter` instead
- Removed deprecated class `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\OpenSearch\DataObject\FieldDefinitionAdapter\AbstractAdapter` please use `Pimcore\Bundle\GenericDataIndexBundle\SearchIndexAdapter\DefaultSearch\DataObject\FieldDefinitionAdapter\AbstractAdapter` instead
- Added default prefix `data-object_` prefix to all data object class definition index names. This change is necessary to avoid conflicts with other index names.
- Add element type to the `getIds` method of `Pimcore\Bundle\GenericDataIndexBundle\Model\Search\Element\SearchResult\ElementSearchResult`
- Added `getSpecialPermissions` method to `Pimcore\Bundle\GenericDataIndexBundle\Service\Permission\ElementPermissionServiceInterface` to get special permissions workspace language permissions for elements
- Removed layout permission from `Pimcore\Bundle\GenericDataIndexBundle\Permission\DataObjectPermissions` as they are not index relevant
- Removed property `isLocked` from Index for elements as it needs to be dynamically calculated
- Changed workspace permissions evaluation in order to align more with the Pimcore Classic bundle permission system

#### Interface changes
- Added `PermissionTypes $permissionType` parameter with default type `PermissionTypes::LIST` to
- `AssetSearchServiceInterface::search` method
- `DocumentSearchServiceInterface::search` method
- `DataObjectSearchServiceInterface::search` method
- `ElementSearchServiceInterface::search` method
- Search services `byId` methods now return elements based on the `PermissionTypes::VIEW` permission
- Added type specific interfaces for searches to avoid mixing up different search types in search services
- `AssetSearch` now implements `AssetSearchInterface`
- `DocumentSearch` now implements `DocumentSearchInterface`
- `ElementSearch` now implements `ElementSearchInterface`
- Search services now require the specific search type for the search
- `AssetSearchServiceInterface::search` now requires a `AssetSearchInterface`
- `DocumentSearchServiceInterface::search` now requires a `DocumentSearchInterface`
- `ElementSearchServiceInterface::search` now requires a `ElementSearchInterface`
- `SearchProviderInterface` now returns type specific search interfaces

- **[Indexing]** Added inherited fields indicator to data object indexing
- **[Indexing]** Added functionality to enqueue dependent items
- **[Indexing]** Added class ID field for data object elements
- **[Indexing]** Index name prefix changed from e.g. `pimcore_car` to
`pimcore_data-object_car`. Old indices are **not deleted** automatically -
delete them manually if necessary.
- **[Searching]** Added `ClassIdsFilter` modifier to filter data objects by class ID
or class name
- Added `isElementLocked()` to `ElementLockService` for retrieving element lock status
from the index
- Reindex all elements to use the new features:
```bash
bin/console generic-data-index:update:index -r
```

### Breaking Changes

- Removed deprecated alias `generic-data-index.opensearch-client`.
Use `generic-data-index.search-client` instead.
- Replaced all deprecated OpenSearch namespace classes with DefaultSearch equivalents:
- `Pimcore\...\Model\OpenSearch` → `Pimcore\...\Model\DefaultSearch`
- `Pimcore\...\Enum\SearchIndex\OpenSearch` → `Pimcore\...\Enum\SearchIndex\DefaultSearch`
- Removed deprecated class `Pimcore\...\Exception\OpenSearch\SearchFailedException`.
Use `Pimcore\...\Exception\DefaultSearch\SearchFailedException` instead.
- Removed deprecated class `AsSearchModifierHandler` from OpenSearch namespace.
Use `Pimcore\...\Attribute\Search\AsSearchModifierHandler` instead.
- Removed deprecated `AbstractAdapter` classes from OpenSearch namespace.
Use `Pimcore\...\SearchIndexAdapter\DefaultSearch\...\AbstractAdapter` instead.
- Added default `data-object_` prefix to all data object class definition index names
to avoid conflicts with other index names.
- Added element type parameter to `getIds` method of `ElementSearchResult`.
- Added `getSpecialPermissions` method to `ElementPermissionServiceInterface`
for workspace language permissions.
- Removed layout permission from `DataObjectPermissions` (not index-relevant).
- Removed `isLocked` property from index elements (now dynamically calculated).
- Changed workspace permissions evaluation to align with Admin Classic permission system.

#### Interface Changes

- Added `PermissionTypes $permissionType` parameter (default: `PermissionTypes::LIST`) to:
- `AssetSearchServiceInterface::search`
- `DocumentSearchServiceInterface::search`
- `DataObjectSearchServiceInterface::search`
- `ElementSearchServiceInterface::search`
- Search service `byId` methods now return elements based on `PermissionTypes::VIEW`
- Added type-specific interfaces:
- `AssetSearch` → `AssetSearchInterface`
- `DocumentSearch` → `DocumentSearchInterface`
- `ElementSearch` → `ElementSearchInterface`
- Search services now require type-specific search objects:
- `AssetSearchServiceInterface::search` requires `AssetSearchInterface`
- `DocumentSearchServiceInterface::search` requires `DocumentSearchInterface`
- `ElementSearchServiceInterface::search` requires `ElementSearchInterface`
- `SearchProviderInterface` now returns type-specific search interfaces.

## Upgrade to 1.3.0
- [Indexing] Added support for Elasticsearch in parallel to Opensearch. Opensearch remains the default search technology. If you are using Elasticsearch, you need to update your symfony configuration as follows:
```yml

- **[Indexing]** Added Elasticsearch support alongside OpenSearch (OpenSearch remains
the default). To use Elasticsearch, update your Symfony configuration:

```yaml
pimcore_generic_data_index:
index_service:
client_params:
client_name: default
client_type: 'elasticsearch'
```
- [Indexing] Introduced new service alias `generic-data-index.search-client`. This will replace deprecated alias `generic-data-index.opensearch-client` which will be removed in the next major version.
The new service alias can be used to inject the search client into your services. This search client is an instance of `Pimcore\SearchClient\SearchClientInterface` which is a common interface for OpenSearch and Elasticsearch clients.
- Classes under OpenSearch namespaces are now deprecated and will be removed in the next major version. Please use the classes under the DefaultSearch namespace instead.
- Execute the following command to reindex all elements to be able to use all new features or when switching between OpenSearch and Elasticsearch:

```bin/console generic-data-index:update:index```
- **[Indexing]** Introduced service alias `generic-data-index.search-client`, replacing
the deprecated `generic-data-index.opensearch-client` (removed in 2.0). The alias
provides a `Pimcore\SearchClient\SearchClientInterface` instance compatible with both
OpenSearch and Elasticsearch.
- OpenSearch namespace classes are deprecated. Use DefaultSearch namespace instead.
- Reindex all elements after upgrading or switching search engines:
```bash
bin/console generic-data-index:update:index
```

## Upgrade to 1.1.0
- Execute the following command to reindex all elements to be able to use all new features:

```bin/console generic-data-index:update:index```
- Reindex all elements to use the new features:
```bash
bin/console generic-data-index:update:index
```
51 changes: 31 additions & 20 deletions doc/01_Installation/README.md
Original file line number Diff line number Diff line change
@@ -1,52 +1,60 @@
# Installation of Generic Data Index
---
title: Installation
description: Install and configure the Generic Data Index bundle with OpenSearch or Elasticsearch.
---

# Installation

:::info

This bundle requires minimum version of OpenSearch 2.7. or Elasticsearch 8.0.0.
This bundle requires OpenSearch >= 2.7 or Elasticsearch >= 8.0.0.

:::

## Bundle Installation

To install the Generic Data Index bundle, follow the steps below:
## Bundle Installation

1) Install the required dependencies:
1. Install the required dependencies:

```bash
composer require pimcore/generic-data-index-bundle
```

2) Make sure the bundle is enabled in the `config/bundles.php` file. The following lines should be added:
2. Enable the bundle in `config/bundles.php`:

```php
use Pimcore\Bundle\GenericDataIndexBundle\PimcoreGenericDataIndexBundle;
// ...
return [
// ...
PimcoreGenericDataIndexBundle::class => ['all' => true],
// ...
];
];
```

3) Install the bundle:
3. Install the bundle:

```bash
bin/console pimcore:bundle:install PimcoreGenericDataIndexBundle
```

4) Setup search client configuration in your Symfony configuration files (e.g. `config.yaml`):

See [OpenSearch Client Setup](../02_Configuration/04_Opensearch.md) or [Elasticsearch Client Setup](../02_Configuration/05_Elasticsearch.md) for more information.
4. Configure the search client in your Symfony configuration (e.g. `config.yaml`).
See [OpenSearch Client Setup](../02_Configuration/04_Opensearch.md)
or [Elasticsearch Client Setup](../02_Configuration/05_Elasticsearch.md).

5) Setup one or multiple Symfony messenger workers for the indexing queue processing. It is recommended to use a tool like Supervisor to manage the workers.
For more information, see the [Symfony Messenger documentation](https://symfony.com/doc/current/messenger.html).
5. Start one or more Symfony Messenger workers for index queue processing.
Use a process manager like Supervisor to keep workers running.
See the [Symfony Messenger documentation](https://symfony.com/doc/current/messenger.html)
for details.

```bash
./bin/console messenger:consume pimcore_generic_data_index_queue scheduler_generic_data_index
bin/console messenger:consume pimcore_generic_data_index_queue scheduler_generic_data_index
```

**Deployment hint:**
:::tip Deployment hint

For deployments of applications with this bundle via deployment pipelines without actual database access, the Symfony cache warming process could fail as doctrine ORM tries to determine the database version on cache warm-up to build its cache. Therefore, it is recommended to configure the database server version in the default DBAL connection like this:
For deployments without database access (e.g. CI pipelines), Doctrine ORM cache warm-up
fails because it tries to detect the database version. Configure the server version
explicitly in the default DBAL connection:

```yaml
doctrine:
Expand All @@ -58,9 +66,12 @@ doctrine:
server_version: mariadb-10.11.0
```

## Commands after Installation
:::

## Post-Installation

After installation, create the indices and queue all elements for indexing:

It is needed to run following command after installation (at least) once to create the indices and add all assets and data objects to the index queue:
```bash
./bin/console generic-data-index:update:index -r
bin/console generic-data-index:update:index -r
```
Loading
Loading