Hello,
I notice that some recent work on bin management has led to the removal of the /api/bin_list method:
8c5839f
The new bin-management API methods are all secured, so there no longer seems to be a public API method to get a json bin list for a dataset.
First, I totally acknowledge that the ifcbdb API is not a publicly advertised API, is subject to change at any time, etc. Just looking for advice here.
We have a recently developed project which creates prometheus metrics for an ifcbdb deployment:
https://github.com/axiom-data-science/ifcb-prometheus-exporter
When determining the lag for production of postprocessed datasets (features, blobs, autoclass), the exporter looks back at up to x number of bins (default 20) to try to find the most recent bin for which postprocessed dataset are available (which is not necessarily the most recent bin). The exporter formerly used the /api/bin_list API call for this.
As of ifcbdb v5.1, I can see two fairly inefficient ways we can get this bin data:
- Get the most recent bin id using
/api/most_recent_bin?dataset=<dataset> and then crawl backwards using /api/bin/<bin id> and previous_bin_id (up to 21 requests per dataset using the default settings)
- Export full dataset metadata to CSV using
/api/export_metadata/<dataset> and parse bin data from the CSV (one request per dataset, but a lot of overhead data)
Any thoughts? We could make do with one of the above, or augment /api/export_metadata/ with additional output formats (json), column selection, include x most recent bins filtering, etc, or possibly add back a filterable public bin list API method.
Hello,
I notice that some recent work on bin management has led to the removal of the
/api/bin_listmethod:8c5839f
The new bin-management API methods are all secured, so there no longer seems to be a public API method to get a json bin list for a dataset.
First, I totally acknowledge that the ifcbdb API is not a publicly advertised API, is subject to change at any time, etc. Just looking for advice here.
We have a recently developed project which creates prometheus metrics for an ifcbdb deployment:
https://github.com/axiom-data-science/ifcb-prometheus-exporter
When determining the lag for production of postprocessed datasets (features, blobs, autoclass), the exporter looks back at up to x number of bins (default 20) to try to find the most recent bin for which postprocessed dataset are available (which is not necessarily the most recent bin). The exporter formerly used the
/api/bin_listAPI call for this.As of ifcbdb v5.1, I can see two fairly inefficient ways we can get this bin data:
/api/most_recent_bin?dataset=<dataset>and then crawl backwards using/api/bin/<bin id>andprevious_bin_id(up to 21 requests per dataset using the default settings)/api/export_metadata/<dataset>and parse bin data from the CSV (one request per dataset, but a lot of overhead data)Any thoughts? We could make do with one of the above, or augment
/api/export_metadata/with additional output formats (json), column selection, include x most recent bins filtering, etc, or possibly add back a filterable public bin list API method.