Conversation
|
Since there is no filtering mechanism, other than whatever the prefix does, it seems like the only functional use cases it get a very larger list of all bundles. If might be nice to state that more explicitly if true. |
|
@kislyuk - not sure why this is tagged rfc-community-review if it's not actually in community review? |
xbrianh
force-pushed
the
bhannafi-dss-bundle-enumeration.md
branch
from
68fa2d5 to
aebf3d3
Compare
|
|
||
| ## Detailed Design | ||
|
|
||
| A new bundle enumeration endpoint, `GET /bundles`, will be introduced, taking replica and prefix parameters. A listing |
There was a problem hiding this comment.
maybe call this "UUID prefix"
There was a problem hiding this comment.
I've updated the text to better reflect the usage of prefix as a "uuid prefix".
| all all bundles having UUIDs beginning with `prefix` will be returned directly from object storage. The prefix | ||
| parameter is intended to facilitate parallelized listings. Pagination semantics and all other semantics of this route | ||
| will be in line with the established conventions of the DSS API. | ||
|
|
There was a problem hiding this comment.
Without some kind of filtering, this seems like a very heavyweight endpoint to use. A couple of things come to mind:
- Filter by update date/time. This would allow only obtaining bundles that have changes since the last time the query was run.
- Filter by bundle type. Well, first we have to have bundle types.
There was a problem hiding this comment.
This endpoint is intended for heavyweight use by downstream indexers.
Also, an incremental approach seems preferable: if filtering becomes desirable in the future, it can be added to the endpoint.
There was a problem hiding this comment.
Agreed that this is easy to add downstream. Assuming a full dump is what existing users want, then my speculative use is not a real use case ;-)
There was a problem hiding this comment.
@diekhans you raise a good point, but as @xbrianh pointed out, the use case here is an unfiltered bulk pull of all bundle IDs for external indexing. We did look for a way to use a "lightweight" database to do filtering using our established filter language process (JMESpath), but didn't find any suitable database/indexing engine for such a task.
|
|
||
| * As a downstream service developer, I would like to check if my index contains all the bundles in the DSS. | ||
|
|
||
| ## Detailed Design |
There was a problem hiding this comment.
I was expecting something a bit more swagger-y rather than a narrative description in Detailed Design for a API.
There was a problem hiding this comment.
@brianraymor It's not clear to me how to address your comment. Perhaps you have something in mind similar to the API endpoint descriptions found in the Deletion RFC.
However, those additions will not necessarily improve the clarity and actionability of this document, which defines a simple extension to the DSS API in language that I believe is clear to the developers who will implement it. Do you feel there are technical details missing, or that there is ambiguity that needs to be cleared up?
There was a problem hiding this comment.
The Deletion RFC more closely meets my expectations for a RFC as design document. Another approach is how Azure defines their REST APIs. Any reviewer/developer should be able to read this RFC and understand the DSS API in detail - not just the implementers. This currently reads more like what we called one pagers at Microsoft.
xbrianh
force-pushed
the
bhannafi-dss-bundle-enumeration.md
branch
from
91785bf to
18ddfca
Compare
6 participants
Proposed addition to the DSS API to enumerate bundles independently of the DSS Elasticsearch metadata index.
Last call for community review: Sept 24