diff --git a/demos/grid-llmd-pool-metrics/README.md b/demos/grid-llmd-pool-metrics/README.md index e66ec57..4dbcf2c 100644 --- a/demos/grid-llmd-pool-metrics/README.md +++ b/demos/grid-llmd-pool-metrics/README.md @@ -96,22 +96,29 @@ mTLS: Grid operator -- HTTPS/mTLS -> nginx :9443 ## Scoring and Routing Policy -This demo explicitly selects Grid's queueDepth scoring strategy with -routingPolicy: scoreFirst. The GridNetwork setting is: +This demo ships two Forge config flavors that share the same +resources/configs assets and differ only in which Grid scoring strategy the +GridNetwork selects, both with routingPolicy: scoreFirst: - scoringPolicy: - strategy: queueDepth +| Flavor | Forge config | GridNetwork setting | +|---|---|---| +| queueDepth (default) | `forge.yaml` | `scoringPolicy: { strategy: queueDepth }` | +| kvCachePressure | `forge-kv-cache.yaml` | `scoringPolicy: { strategy: kvCachePressure }` | + +Run either with `./run.sh`; pass `--kv-cache` to select the second flavor +(see [Quick Start](#quick-start) / [Full Mode](#full-mode) below). Grid normalizes the EPP queue depth using queueCapacity: 4 and computes the -dynamic provider score as: +dynamic provider score for the queueDepth flavor as: score = 1 - normalized_queue_depth -The provider with the lower queue therefore receives the better dynamic score. -KV-cache utilization is collected and shown in the live table as an important -pressure signal, but it is not combined into this demo's Grid score. Grid -selects one provider-level metric strategy at a time so the routing decision -remains explainable. +The provider with the lower queue therefore receives the better dynamic +score. For the kvCachePressure flavor, the provider with the most available +KV-cache capacity receives the better score instead. Both signals are always +collected and shown in the live table regardless of which flavor is running, +but Grid selects one provider-level metric strategy at a time so the routing +decision remains explainable. Available strategies are: @@ -122,7 +129,7 @@ Available strategies are: | kvCachePressure | Prefers the provider with the most available KV-cache capacity. | geographyFirst is the default routing policy and preserves locality ahead of -score. This demo uses scoreFirst so a sufficiently better queue score can move +score. This demo uses scoreFirst so a sufficiently better score can move traffic to the remote pool. ## Controlled Pressure @@ -245,6 +252,16 @@ To validate an mTLS-protected metrics endpoint: Full mode adds the pressure-flip and recovery proofs. +To run the same proofs against the kvCachePressure flavor instead: + +```bash +./run.sh --full --kv-cache --teardown +``` + +`--kv-cache` selects `forge-kv-cache.yaml` in place of `forge.yaml`; every +other flag (`--quick`, `--metrics-mtls`, `--teardown`, `--keep-on-failure`) +works the same with either flavor. + ## Teardown and Keep-on-Failure `--teardown` deletes both Kind clusters after the run, including on failure. @@ -285,8 +302,8 @@ generator requests or with `MOCK_TIME_FACTOR_UNDER_LOAD > 1.0`. - No real GPU inference; vllm-vcr generates random tokens, so response content is meaningless. -- No P99 latency or prefix-cache derivation; those signals are not used by the - selected queueDepth strategy. +- No P99 latency or prefix-cache derivation; those signals are not used by + either the queueDepth or kvCachePressure strategy. - Two-pool topology; each cluster's own provider scores with full locality (1.0) while the remote peer scores at 0.5. - No cost signal; defaults to 0.5. diff --git a/demos/grid-llmd-pool-metrics/forge-kv-cache.yaml b/demos/grid-llmd-pool-metrics/forge-kv-cache.yaml new file mode 100644 index 0000000..72f6ccd --- /dev/null +++ b/demos/grid-llmd-pool-metrics/forge-kv-cache.yaml @@ -0,0 +1,512 @@ +apiVersion: forge.praxis.dev/v1alpha1 +kind: Environment + +metadata: + name: grid-llmd-pool-metrics-kv-cache + +spec: + runtime: + provider: docker + clusterPrefix: grid-llmd-pm + + network: + crossCluster: true + dnsZone: grid-llmd-pm.test + + clusters: + - name: pool-a + stacks: [metallb, pool-a-operator-base, llmd-pool-a, pool-a-site, provider-gateway, consumer-gateway] + properties: + region: pool-a + role: combined + siteName: pool-a + gatewayImage: "ghcr.io/praxis-proxy/grid-ai-rollup:v0.1.3" + operatorImage: "ghcr.io/praxis-proxy/grid-operator:v0.1.3" + eppImage: "ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0" + vcrImage: "ghcr.io/neuralmagic/vllm-vcr:vllm0.23" + imagePullPolicy: IfNotPresent + gatewayImageRepo: "ghcr.io/praxis-proxy/grid-ai-rollup" + gatewayImageTag: "v0.1.3" + operatorImageRepo: "ghcr.io/praxis-proxy/grid-operator" + operatorImageTag: "v0.1.3" + eppImageRepo: "ghcr.io/llm-d/llm-d-inference-scheduler" + eppImageTag: "v0.8.0" + vcrImageRepo: "ghcr.io/neuralmagic/vllm-vcr" + vcrImageTag: "vllm0.23" + poolName: pool-a + + - name: pool-b + stacks: [metallb, pool-b-operator-base, llmd-pool-b, pool-b-site, provider-gateway, consumer-gateway] + properties: + region: pool-b + role: combined + siteName: pool-b + gatewayImage: "ghcr.io/praxis-proxy/grid-ai-rollup:v0.1.3" + operatorImage: "ghcr.io/praxis-proxy/grid-operator:v0.1.3" + eppImage: "ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0" + vcrImage: "ghcr.io/neuralmagic/vllm-vcr:vllm0.23" + imagePullPolicy: IfNotPresent + gatewayImageRepo: "ghcr.io/praxis-proxy/grid-ai-rollup" + gatewayImageTag: "v0.1.3" + operatorImageRepo: "ghcr.io/praxis-proxy/grid-operator" + operatorImageTag: "v0.1.3" + eppImageRepo: "ghcr.io/llm-d/llm-d-inference-scheduler" + eppImageTag: "v0.8.0" + vcrImageRepo: "ghcr.io/neuralmagic/vllm-vcr" + vcrImageTag: "vllm0.23" + poolName: pool-b + + stacks: + metallb: + description: MetalLB load balancer with auto-configured address pool + steps: + - type: url + url: https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml + sha256: 951065e85692aa106f1bb5d5a487d9306154923a794ab1d82122881cbaf588e4 + - type: wait + resource: deployment/controller + namespace: metallb-system + condition: available + timeout: "120s" + - type: metallb-auto-pool + name: forge-pool + + pool-a-operator-base: + description: Pool-A Grid operator with SWIM LoadBalancer + steps: + - type: helm + release: grid-operator + chart: charts/grid-operator + version: "0.1.0" + namespace: grid-system + values: + image: + repository: "{{ cluster.properties.operatorImageRepo }}" + tag: "{{ cluster.properties.operatorImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + swim: + siteName: "pool-a" + seeds: "" + service: + enabled: true + type: "LoadBalancer" + gateway: + serviceName: "provider-gateway" + port: "8443" + - type: wait + resource: deployment/grid-operator + namespace: grid-system + condition: available + timeout: "120s" + - type: capture + resource: svc/grid-operator-swim + namespace: grid-system + jsonpath: "{.status.loadBalancer.ingress[0].ip}" + key: swim-lb-ip + timeout: "60s" + interval: "2s" + + pool-b-operator-base: + description: Pool-B Grid operator with SWIM LoadBalancer + steps: + - type: helm + release: grid-operator + chart: charts/grid-operator + version: "0.1.0" + namespace: grid-system + values: + image: + repository: "{{ cluster.properties.operatorImageRepo }}" + tag: "{{ cluster.properties.operatorImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + swim: + siteName: "pool-b" + seeds: "" + service: + enabled: true + type: "LoadBalancer" + gateway: + serviceName: "provider-gateway" + port: "8443" + - type: wait + resource: deployment/grid-operator + namespace: grid-system + condition: available + timeout: "120s" + - type: capture + resource: svc/grid-operator-swim + namespace: grid-system + jsonpath: "{.status.loadBalancer.ingress[0].ip}" + key: swim-lb-ip + timeout: "60s" + interval: "2s" + + llmd-pool-a: + description: vllm-vcr inference backends and EPP for pool-a + steps: + - type: manifest + path: resources/common/grid-system-namespace.yaml + - type: manifest + path: resources/common/inferencepool-crd.yaml + - type: manifest + path: resources/common/epp-rbac.yaml + - type: manifest + path: resources/pool-a/vcr-deployment.yaml + - type: manifest + path: resources/pool-a/inferencepool.yaml + - type: manifest + path: resources/pool-a/epp-deployment.yaml + - type: manifest + path: resources/pool-a/pressure-generator.yaml + - type: wait + resource: deployment/vcr-1 + namespace: grid-system + condition: available + timeout: "600s" + - type: wait + resource: deployment/vcr-2 + namespace: grid-system + condition: available + timeout: "600s" + - type: wait + resource: deployment/llmd-epp + namespace: grid-system + condition: available + timeout: "120s" + - type: capture + resource: svc/llmd-epp-metrics + namespace: grid-system + jsonpath: "{.spec.clusterIP}" + key: epp-metrics-ip + timeout: "30s" + interval: "2s" + + llmd-pool-b: + description: vllm-vcr inference backends and EPP for pool-b + steps: + - type: manifest + path: resources/common/grid-system-namespace.yaml + - type: manifest + path: resources/common/inferencepool-crd.yaml + - type: manifest + path: resources/common/epp-rbac.yaml + - type: manifest + path: resources/pool-b/vcr-deployment.yaml + - type: manifest + path: resources/pool-b/inferencepool.yaml + - type: manifest + path: resources/pool-b/epp-deployment.yaml + - type: wait + resource: deployment/vcr-1 + namespace: grid-system + condition: available + timeout: "600s" + - type: wait + resource: deployment/vcr-2 + namespace: grid-system + condition: available + timeout: "600s" + - type: wait + resource: deployment/llmd-epp + namespace: grid-system + condition: available + timeout: "120s" + - type: capture + resource: svc/llmd-epp-metrics + namespace: grid-system + jsonpath: "{.spec.clusterIP}" + key: epp-metrics-ip + timeout: "30s" + interval: "2s" + + pool-a-site: + description: Pool-A combined site with Grid CRs + steps: + - type: helm + release: grid-site + chart: charts/grid-site + version: "0.1.0" + namespace: grid-system + values: + commonLabels: + grid.praxis-proxy.io/auto-discover-sites: "true" + gridNetwork: + name: grid-llmd-pool-metrics + gridId: grid-llmd-pool-metrics-v1 + region: pool-a + zone: pool-a-1 + routingPolicy: scoreFirst + scoringPolicy: + strategy: kvCachePressure + swim: + probeInterval: "5s" + suspicionTimeout: "15s" + gossipNodes: 3 + tls: + caSecretRef: + name: consumer-gateway-tls + namespace: grid-system + siteSecretRef: + name: consumer-gateway-tls + namespace: grid-system + gatewayRefs: + - name: consumer-gateway + namespace: grid-system + localSiteName: pool-a + gridSite: + name: pool-a + region: pool-a + zone: pool-a-1 + providerSiteLabel: pool-a + inferenceProviders: + - name: llmd-pool-a-provider + gridNetworkRef: grid-llmd-pool-metrics + providerKind: vllm-vcr + backendKind: local + endpoint: "http://vcr-service.grid-system.svc.cluster.local:8000" + siteSelector: + matchLabels: + grid.praxis-proxy.io/provider-site: pool-a + accessPolicy: + siteSelector: + matchLabels: {} + models: + - name: Qwen/Qwen3-0.6B + capabilities: + - text_generation + contextWindow: 4096 + healthCheck: + path: /health + interval: "30s" + timeout: "5s" + metricsConfig: + path: /metrics + timeout: "2s" + staleMetricsSeconds: 20 + metricsEndpoint: "http://llmd-epp-metrics.grid-system.svc.cluster.local:9090" + poolName: "{{ cluster.properties.poolName }}" + queueCapacity: 4 + signalNames: + kvCacheUtilization: inference_pool_average_kv_cache_utilization + queueDepth: inference_pool_average_queue_size + healthy: inference_pool_ready_pods + + pool-b-site: + description: Pool-B combined site with Grid CRs + steps: + - type: helm + release: grid-site + chart: charts/grid-site + version: "0.1.0" + namespace: grid-system + values: + commonLabels: + grid.praxis-proxy.io/auto-discover-sites: "true" + gridNetwork: + name: grid-llmd-pool-metrics + gridId: grid-llmd-pool-metrics-v1 + region: pool-b + zone: pool-b-1 + routingPolicy: scoreFirst + scoringPolicy: + strategy: kvCachePressure + swim: + probeInterval: "5s" + suspicionTimeout: "15s" + gossipNodes: 3 + tls: + caSecretRef: + name: consumer-gateway-tls + namespace: grid-system + siteSecretRef: + name: consumer-gateway-tls + namespace: grid-system + gatewayRefs: + - name: consumer-gateway + namespace: grid-system + localSiteName: pool-b + gridSite: + name: pool-b + region: pool-b + zone: pool-b-1 + providerSiteLabel: pool-b + inferenceProviders: + - name: llmd-pool-b-provider + gridNetworkRef: grid-llmd-pool-metrics + providerKind: vllm-vcr + backendKind: local + endpoint: "http://vcr-service.grid-system.svc.cluster.local:8000" + siteSelector: + matchLabels: + grid.praxis-proxy.io/provider-site: pool-b + accessPolicy: + siteSelector: + matchLabels: {} + models: + - name: Qwen/Qwen3-0.6B + capabilities: + - text_generation + contextWindow: 4096 + healthCheck: + path: /health + interval: "30s" + timeout: "5s" + metricsConfig: + path: /metrics + timeout: "2s" + staleMetricsSeconds: 20 + metricsEndpoint: "http://llmd-epp-metrics.grid-system.svc.cluster.local:9090" + poolName: "{{ cluster.properties.poolName }}" + queueCapacity: 4 + signalNames: + kvCacheUtilization: inference_pool_average_kv_cache_utilization + queueDepth: inference_pool_average_queue_size + healthy: inference_pool_ready_pods + + provider-gateway: + description: Praxis provider gateway with mTLS and credential mounts + steps: + - type: template-file + source: configs/provider/praxis.yaml + target: .forge/runtime/{{ cluster.name }}/provider/praxis.yaml + - type: exec + command: + - bash + - -c + - >- + kubectl --context kind-grid-llmd-pm-{{ cluster.name }} -n grid-system + create configmap provider-gateway-config + --from-file=praxis.yaml=.forge/runtime/{{ cluster.name }}/provider/praxis.yaml + --dry-run=client -o yaml | + kubectl --context kind-grid-llmd-pm-{{ cluster.name }} apply -f - + - type: helm + release: provider-gateway + chart: charts/praxis-gateway + version: "0.1.0" + namespace: grid-system + values: + fullnameOverride: "provider-gateway" + image: + repository: "{{ cluster.properties.gatewayImageRepo }}" + tag: "{{ cluster.properties.gatewayImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + podSecurityContext: + runAsUser: 100 + runAsGroup: 101 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + config: + existingConfigMap: "provider-gateway-config" + port: + containerPort: 8443 + name: "https-mtls" + service: + type: "LoadBalancer" + port: 8443 + health: + readiness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 3 + periodSeconds: 5 + liveness: + tcpSocket: + port: "https-mtls" + initialDelaySeconds: 5 + periodSeconds: 10 + tls: + enabled: true + existingSecret: "provider-gateway-tls" + credentials: + - name: "vcr-inference-credential" + mountPath: "/etc/praxis/credentials/vcr-inference" + podLabels: + grid.praxis-proxy.io/backend-access: "provider-gateway" + grid.praxis-proxy.io/provider-site: "{{ cluster.name }}" + - type: wait + resource: deployment/provider-gateway + namespace: grid-system + condition: available + timeout: "120s" + - type: capture + resource: svc/provider-gateway + namespace: grid-system + jsonpath: "{.status.loadBalancer.ingress[0].ip}" + key: provider-gateway-ip + timeout: "60s" + interval: "2s" + + consumer-gateway: + description: Praxis consumer gateway with operator-managed overlay + steps: + - type: template-file + source: configs/consumer/praxis.yaml + target: .forge/runtime/{{ cluster.name }}/consumer/praxis.yaml + - type: exec + command: + - bash + - -c + - >- + kubectl --context kind-grid-llmd-pm-{{ cluster.name }} -n grid-system + create configmap consumer-gateway-config + --from-file=praxis.yaml=.forge/runtime/{{ cluster.name }}/consumer/praxis.yaml + --dry-run=client -o yaml | + kubectl --context kind-grid-llmd-pm-{{ cluster.name }} apply -f - + - type: helm + release: consumer-gateway + chart: charts/praxis-gateway + version: "0.1.0" + namespace: grid-system + values: + fullnameOverride: "consumer-gateway" + image: + repository: "{{ cluster.properties.gatewayImageRepo }}" + tag: "{{ cluster.properties.gatewayImageTag }}" + pullPolicy: "{{ cluster.properties.imagePullPolicy }}" + podSecurityContext: + runAsUser: 100 + runAsGroup: 101 + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + config: + existingConfigMap: "consumer-gateway-config" + service: + type: "LoadBalancer" + overlay: + enabled: true + existingConfigMap: "grid-overlay-grid-llmd-pool-metrics-consumer-gateway" + sidecar: + enabled: true + image: + repository: "ghcr.io/praxis-proxy/grid-overlay-sync" + tag: "v0.1.3" + pullPolicy: "IfNotPresent" + expectedNetwork: "grid-llmd-pool-metrics" + expectedLocalSite: "{{ cluster.name }}" + tls: + enabled: true + existingSecret: "consumer-gateway-tls" + podLabels: + grid.praxis-proxy.io/consumer-site: "{{ cluster.name }}" + - type: wait + resource: deployment/consumer-gateway + namespace: grid-system + condition: available + timeout: "120s" + - type: capture + resource: svc/consumer-gateway + namespace: grid-system + jsonpath: "{.status.loadBalancer.ingress[0].ip}" + key: consumer-gateway-ip + timeout: "60s" + interval: "2s" diff --git a/demos/grid-llmd-pool-metrics/run.sh b/demos/grid-llmd-pool-metrics/run.sh index 9c24d7d..6af25cb 100755 --- a/demos/grid-llmd-pool-metrics/run.sh +++ b/demos/grid-llmd-pool-metrics/run.sh @@ -1,4 +1,17 @@ #!/usr/bin/env bash set -euo pipefail DEMO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -exec "$(dirname "${DEMO_DIR}")/../scripts/run-grid-demo.sh" "${DEMO_DIR}" "$@" + +# Intercept --kv-cache to select the kvCachePressure scoring flavor +# (forge-kv-cache.yaml) instead of the default queueDepth flavor +# (forge.yaml). Both share this directory's resources/ and configs/. +args=() +for arg in "$@"; do + if [[ "${arg}" == "--kv-cache" ]]; then + export FORGE_CONFIG_NAME="forge-kv-cache.yaml" + else + args+=("${arg}") + fi +done + +exec "$(dirname "${DEMO_DIR}")/../scripts/run-grid-demo.sh" "${DEMO_DIR}" "${args[@]+"${args[@]}"}" diff --git a/scripts/run-grid-demo.sh b/scripts/run-grid-demo.sh index a419cac..2321fb4 100755 --- a/scripts/run-grid-demo.sh +++ b/scripts/run-grid-demo.sh @@ -9,8 +9,12 @@ usage() { Usage: run-grid-demo.sh [xtask-flags...] Environment: - GRID_REPO Path to a local praxis-proxy/grid checkout. - When unset, Grid is cloned into .grid-checkout/. + GRID_REPO Path to a local praxis-proxy/grid checkout. + When unset, Grid is cloned into .grid-checkout/. + FORGE_CONFIG_NAME Forge config filename within (default: forge.yaml). + Lets a demo ship multiple Forge config flavors side by + side (e.g. forge-kv-cache.yaml) without duplicating its + resources/configs assets. Image overrides (optional): GRID_XTASK_GATEWAY_IMAGE @@ -54,7 +58,14 @@ case "${DEMO_NAME}" in ;; esac -FORGE_CONFIG="${DEMO_DIR}/forge.yaml" +FORGE_CONFIG_NAME="${FORGE_CONFIG_NAME:-forge.yaml}" +# Must be a bare filename within DEMO_DIR, not a path -- rejects any "/" so a +# stray "../" (or an absolute path) can't resolve outside DEMO_DIR. +if [[ "${FORGE_CONFIG_NAME}" == */* ]]; then + echo "error: FORGE_CONFIG_NAME must be a bare filename, not a path: '${FORGE_CONFIG_NAME}'" >&2 + exit 1 +fi +FORGE_CONFIG="${DEMO_DIR}/${FORGE_CONFIG_NAME}" if [[ ! -f "${FORGE_CONFIG}" ]]; then echo "error: forge config not found: ${FORGE_CONFIG}" >&2 exit 1