Description
Problem
When the configuration for metadata extraction (= updated xpath) has changed or a metadata field has been added or removed, a re-indexing of all affected documents is required. The issue with the current state is, that the re-indexing of the documents also re-indexes all fulltexts of those documents. While changes to metadata fields certainly require a re-indexing of all metadata, this is not the case for fulltexts as those are being extracted AGAIN in exactly the same way as before. Re-Indexing the fulltexts makes it a timeconsuming task and results in excessive computational overhead, which is bad from an economical and ecological perspective.
Proposed Solution
The proposed solution is to provide a CLI option to re-index/update metadata only for all structures and pages, while preserving fulltexts on the page level. If i understand correctly this should be possible with SOLR's atomic updates, which includes deleting/adding and changing fields. Solarium should be capable to interface this.
Additional Context
Re-Indexing is a common task, because requirements for searchable metadata, facets and sorting can change dynamically on a digitization portal - whether its because of new collections/projects being added with their own specifica, because of updated metadata provided by the catalog (e.g. re-import of catalog data in Kitodo.Production) or because there are errors in the metadata which have been pointed out for example by the DDB-Schematron validation and have been corrected accordingly.
Re-Indexing of "Hamburger Kulturgut Digital" takes about 6-8 Weeks (24/7) mainly because of all the fulltexts being unnecessarily reprocessed. Re-Indexing metadata only would likely reduce this by a magnitude - if not more.
Description
Problem
When the configuration for metadata extraction (= updated
xpath) has changed or a metadata field has been added or removed, a re-indexing of all affected documents is required. The issue with the current state is, that the re-indexing of the documents also re-indexes allfulltextsof those documents. While changes to metadata fields certainly require a re-indexing of all metadata, this is not the case for fulltexts as those are being extracted AGAIN in exactly the same way as before. Re-Indexing the fulltexts makes it a timeconsuming task and results in excessive computational overhead, which is bad from an economical and ecological perspective.Proposed Solution
The proposed solution is to provide a CLI option to re-index/update metadata only for all structures and pages, while preserving fulltexts on the page level. If i understand correctly this should be possible with SOLR's atomic updates, which includes deleting/adding and changing fields. Solarium should be capable to interface this.
Additional Context
Re-Indexing is a common task, because requirements for searchable metadata, facets and sorting can change dynamically on a digitization portal - whether its because of new collections/projects being added with their own specifica, because of updated metadata provided by the catalog (e.g. re-import of catalog data in Kitodo.Production) or because there are errors in the metadata which have been pointed out for example by the DDB-Schematron validation and have been corrected accordingly.
Re-Indexing of "Hamburger Kulturgut Digital" takes about 6-8 Weeks (24/7) mainly because of all the fulltexts being unnecessarily reprocessed. Re-Indexing metadata only would likely reduce this by a magnitude - if not more.