Epic: Ray Serve Pack Implementation

[dcmcand]

Goal

Implement the Nebari Ray Serve software pack for integrating Ray Serve into T&E applications.

Scope

• Scope and define the initial implementation requirements
• Build the Helm chart and container images for Ray Serve integration
• Integrate with the Nebari operator and landing page

[kenafoster]

At the end of issue #1 we have a skeleton RayServe deployment.

Next step (and we are taking this as an action item in the customer project that this is supporting) is to refine the rest of the high-level feature "wants" below into specific, prioritized requirements:

• Access to certain models should be controllable via keycloak

  • NOTE: There is no RBAC right now. All nebari users have access to the ray cluster.
  • Need to research whether there is even a concept of granular access control within Ray Serve. The authentication we use right now is "front door" only to the dashboard. Anyone in JHub with the URL to the Ray Serve cluster (i.e. the ...svc.local) is able to hit the full API
  • This opens two questions:
    • Are there API auth controls we can implement (both API and dashboard) in Ray Serve?
    • To fill any gaps, do we need to build API endpoint auth (from within the K8S cluster/JHub) and/or do we approach fine-grained permissions by deploying multiple Ray Serve clusters (see related concern below r.e. Python environments)

• Do we want to be able to have multiple different Python runtime environments in a single Nebari deployment?

  • Assumption 1 - Within one Ray Cluster, you can only specify one docker image for the head and worker nodes. All worker nodes will use this image. This means one Ray Serve cluster will have only one base environment. So there is no concept of granting users access to different base environments within a cluster
  • Assumption 2 - You CAN add dependencies via runtime_env specification which is limited to pip (not ideal for ML libraries of course, and also adds runtime "on the fly" burden). We assume this is not useful for any serious use cases
  • So... it might be that we recommend deploying multiple Ray Serve clusters for use cases where different Python envs are required and/or where granular environment or model permissions are needed
  • Side note - motivating question: Is there a way to specify the image from jupyter hub - Oren doesn't think that's possible. has to be changed via the gitops and the ray cluster must be redeployed to change it. (I.e. why multiple clusters may be needed for multiple environments)

• Per above, if the research validates our assumptions then one of the first development tasks is proving multiple Ray Serve clusters can be deployed in a single K8S cluster/instance of Nebari (seems possible, just give it a different URL in the ray.init)

• Its deploying straight from repo - there is no helm chart yet. Need to publish as helm chart long term.

[oren-openteams]

we can host the helm charts here: https://github.com/nebari-dev/helm-repository