Skip to content

Latest commit

 

History

History
45 lines (37 loc) · 2.88 KB

roadmap.md

File metadata and controls

45 lines (37 loc) · 2.88 KB

Roadmap

This section documents some of the improvements that we plan to make to Sleeper.

The following improvements are actively being worked on:

The following are likely to be worked on in the near future:

  • gchq#5078 Improvements to bulk import.
  • gchq#6132 Create a Spark DataFrame from a Sleeper table.
  • gchq#4215 Optional long running service for queries.
  • gchq#4235 Graceful upgrade of a Sleeper instance.

The following improvements will be worked on in future (these are in no particular order):

  • gchq#6059 Data types for floating point value fields.
  • gchq#6058 Notifications for ingest progress.
  • gchq#6117 Instance health checks.
  • gchq#4213 Batch up partition splitting commits.
  • gchq#1391 Create a library of repeatable, sustained, large-scale performance tests.
  • gchq#1393 Bulk export queries, tracking, restore from export.
  • gchq#4396 Failure handling / backpressure for state store updates.
  • gchq#3693 Improvements to declarative deployment with infrastructure as code.
  • gchq#576 Use Arrow types in the table schema.
  • gchq#4398 Trigger compaction dispatch in transaction log follower.
  • Scaling improvements.
    • gchq#4525 Mitigate limitations on throughput of state store updates.
    • gchq#4218 Batch up updates to job trackers from state store commits.
    • gchq#5674 Table state compression.
    • gchq#4555 Optimise table snapshot creation.
  • Usability improvements.
    • gchq#1328 Unify admin client and related scripts.
    • gchq#1786 REST API.
    • Python API improvements. This is currently basic and needs further work.
  • gchq#1392 Create a predicate language for specifying filters on queries.
  • gchq#1390 Review and extend the integrations with Athena and Trino.
  • gchq#5675 Visibility of long term tracker metrics.
  • Metrics page. Review and extend the metrics produced.
  • Purge data from a table, i.e. delete any items matching a predicate.

We also have an article on potential deployment improvements, examining how the current deployment setup relates to the planned improvements linked above.