full traversal connector deletes existing items

[rishi-dhar]

We've noticed that the full traversal connector deletes any existing items in the repository and in essence "replaces" them with the new ones on the next run. Is this expected behavior?

How do you deal with adding more items to an existing repository while using the full traversal connector?

[cjohnson78]

The behavior should be that it re-indexes all items using the same Item Name (which results in a replace/overwrite - not a new item) but it does give the items a new Queue Name (Queue A vs. Queue B). Once the traversal is complete, it issues a Delete All Queue Items for the previous Queue Name (which takes care of deleting orphaned items).

[cjohnson78]

The FullTraversalConnector template itself does not issue "Deletes" (only indexingService.deleteQueueItems() as I described above). But, the Repository for your specific implementation can technically return Delete Items when it returns items. The FullTraversalConnector will execute those delete if the Repository returns them. So, can you clarify which specific connector you are using? And can you check if its Repository returns any Delete Items?

[rishi-dhar]

The FullTraversalConnector template itself does not issue "Deletes" (only indexingService.deleteQueueItems() as I described above). But, the Repository for your specific implementation can technically return Delete Items when it returns items. The FullTraversalConnector will execute those delete if the Repository returns them. So, can you clarify which specific connector you are using? And can you check if its Repository returns any Delete Items?

I'll describe our use case. We've built a connector to populate a Datasource with new items every 6 hours but we'd like the older items that we've previously indexed to remain as is and the new items to be added on to the data source. All items have unique names so we aren't overwriting any existing items. The issue is that our repository is fairly large and getting back all items each time the connector runs is not ideal so we are attempting to incrementally update the datasource.

eg. We sync 100 items at 10:00 EST and we see 100 items in Cloud Search
We sync 150 new items at 11:00EST and we see 150 items in Cloud Search (the previous 100 are gone)

[TanmayVartak]

You can set travese.usesQueue to false to disable deletion. https://developers.google.com/cloud-search/docs/reference/sdk/com/google/enterprise/cloudsearch/sdk/indexing/template/FullTraversalConnector#public-static-final-string-traverse_use_queues

…

On Thu, Jul 14, 2022, 11:32 AM Rishi Dhar ***@***.***> wrote: The FullTraversalConnector template itself does not issue "Deletes" (only indexingService.deleteQueueItems() as I described above). But, the Repository for your specific implementation can technically return Delete Items when it returns items. The FullTraversalConnector will execute those delete if the Repository returns them. So, can you clarify which specific connector you are using? And can you check if its Repository returns any Delete Items? I'll describe our use case. We've built a connector to populate a Datasource with new items every 6 hours but we'd like the older items that we've previously indexed to remain as is and the new items to be added on to the data source. All items have unique names so we aren't overwriting any existing items. The issue is that our repository is fairly large and getting back all items each time the connector runs is not ideal so we are attempting to incrementally update the datasource. eg. We sync 100 items at 10:00 EST and we see 100 items in Cloud Search We sync 150 new items at 11:00EST and we see 150 items in Cloud Search (the previous 100 are gone) — Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEV3QSUBKKWFGIWHXO6ZV3VUBMMJANCNFSM53S5ISOQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rishi-dhar]

https://developers.google.com/cloud-search/docs/reference/sdk/com/google/enterprise/cloudsearch/sdk/indexing/template/FullTraversalConnector#public-static-final-string-traverse_use_queues

This actually works well for our use case. Any downsides to this? I don't know if I fully understand what the queues do.

[TanmayVartak]

You are essentially disabling deletes as part of full traversal (That's what you want).

…

On Thu, Jul 14, 2022 at 1:19 PM Rishi Dhar ***@***.***> wrote: https://developers.google.com/cloud-search/docs/reference/sdk/com/google/enterprise/cloudsearch/sdk/indexing/template/FullTraversalConnector#public-static-final-string-traverse_use_queues This actually works well for our use case. Any downsides to this? I don't know if I fully understand what the queues do. — Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEV3QSFIFUBFTCGV7M3S5LVUBY4XANCNFSM53S5ISOQ> . You are receiving this because you commented.Message ID: ***@***.***>