diff --git a/.github/workflows/build-and-push.yaml b/.github/workflows/build-and-push.yaml index 2f9f1f59..c87e30a7 100644 --- a/.github/workflows/build-and-push.yaml +++ b/.github/workflows/build-and-push.yaml @@ -46,6 +46,7 @@ jobs: - 3d - dem - raster + - pycsw steps: - name: Display matrix and index run: | diff --git a/helm/pycsw/.helmignore b/helm/pycsw/.helmignore new file mode 100644 index 00000000..0cb7b3bf --- /dev/null +++ b/helm/pycsw/.helmignore @@ -0,0 +1,26 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ +*.txt +*local.json +*local.yaml diff --git a/helm/pycsw/Chart.yaml b/helm/pycsw/Chart.yaml new file mode 100644 index 00000000..88bcb4bd --- /dev/null +++ b/helm/pycsw/Chart.yaml @@ -0,0 +1,13 @@ +apiVersion: v2 +name: pycsw +description: Common Helm chart for pycsw service, consumed as a dependency by team deployment charts +type: application +version: 6.6.1 +appVersion: 6.6.1 +dependencies: + - name: nginx + version: 2.2.1 + repository: oci://acrarolibotnonprod.azurecr.io/helm/common + - name: mclabels + version: 1.0.1 + repository: oci://acrarolibotnonprod.azurecr.io/helm/infra diff --git a/helm/pycsw/README.md b/helm/pycsw/README.md new file mode 100644 index 00000000..34360fc0 --- /dev/null +++ b/helm/pycsw/README.md @@ -0,0 +1,107 @@ +# pycsw common chart + +Common Helm chart for the MapColonies pycsw service. It is not deployed directly; +each team consumes it as a dependency from its own deployment repository and injects +its team-specific configuration. + +## Consuming the chart + +In your team deployment repo, create a thin wrapper chart (see `examples/team-wrapper` +in this repository for a complete working example): + +```yaml +# Chart.yaml +dependencies: + - name: pycsw + version: 6.6.1 + repository: oci://acrarolibotnonprod.azurecr.io/helm +``` + +## Team configuration contract + +The chart does **not** ship `pycsw.cfg` or `mappings.py` — these are team-specific. +Your wrapper chart must provide a ConfigMap containing both keys and pass its name +via `existingConfigmap` (the value may be a template): + +```yaml +# wrapper values.yaml +pycsw: + existingConfigmap: '{{ .Release.Name }}-pycsw-team-config' +``` + +```yaml +# wrapper templates/configmap.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Release.Name }}-pycsw-team-config +data: + pycsw.cfg: {{ tpl (.Files.Get "config/pycsw.cfg") . | quote }} + mappings.py: {{ .Files.Get "config/mappings.py" | quote }} +``` + +Rendering fails with a clear error when `existingConfigmap` is not set. + +### Environment variables available to pycsw.cfg + +pycsw expands `${VAR}` placeholders in `pycsw.cfg` at runtime, so the team config +file needs no Helm helpers from this chart. The chart provides: + +| Variable | Content | +|---|---| +| `PYCSW_SERVER_URL` | public server URL derived from the nginx route values | +| `PYCSW_MAPPINGS_FILEPATH` | mount path of the team `mappings.py` | +| `PYCSW_DATABASE_CONNECTION` | full postgres connection string (user/password from the DB secret, host/port/name/SSL params from `db`/`global.db` values) | +| `DB_SCHEMA` | value of `env.db.schema` | + +A minimal `[repository]` section in a team `pycsw.cfg`: + +```ini +[repository] +database=${PYCSW_DATABASE_CONNECTION} +mappings=${PYCSW_MAPPINGS_FILEPATH} +table=${DB_SCHEMA}.records +``` + +### Values teams are expected to set + +| Value | Meaning | +|---|---| +| `existingConfigmap` | name of the team ConfigMap (required) | +| `env.db.schema` | catalog schema, e.g. `RasterCatalogManager` | +| `mclabels.owner`, `mclabels.gisDomain` | team labels | +| `nginx.authorization.domain` | OPA authorization domain | +| `nginx.route.routesMapping` | exposed route paths | +| `replicaCount`, `resources` | scale | + +### Multiple instances in one release + +When consuming this chart more than once in the same release (aliased +dependencies, e.g. a filtered and an unfiltered catalog), each extra instance +must set: + +| Value | Meaning | +|---|---| +| `nameOverride` | distinct pycsw resource names and selector labels | +| `existingConfigmap` | its own team ConfigMap | +| `nginx.nameOverride` | distinct nginx selector labels (nginx >=2.2.1 derives pod selectors from it — identical values make services select each other's pods) | +| `nginx.fullnameOverride` | distinct nginx resource names | +| `nginx.extraVolumes` | point at the instance's own nginx configmap: `{{ .Release.Name }}--nginx-configmap` | + +## Nginx + +Nginx uses the modular extension layout: this chart ships generic +`pycsw-server.conf` / `pycsw-location.conf` snippets mounted into the nginx +subchart's `extensions` directory. Team-specific nginx behavior is driven by +values (`nginx.authorization`, `nginx.route.routesMapping`); additional snippets +can be added through the nginx subchart's own values. + +## Config changes and pod restarts + +The deployment checksums only the chart-owned ConfigMap (env vars, uwsgi.ini). +Changes to the team ConfigMap (`pycsw.cfg`, `mappings.py`) do **not** restart pods +automatically — run a rollout restart after changing team config: + +```bash +kubectl rollout restart deployment -pycsw-deployment +``` diff --git a/helm/pycsw/config/pycsw-location.conf b/helm/pycsw/config/pycsw-location.conf new file mode 100644 index 00000000..dffe1d70 --- /dev/null +++ b/helm/pycsw/config/pycsw-location.conf @@ -0,0 +1,6 @@ + +# pycsw custom uwsgi settings injected into root location +uwsgi_pass {{ include "pycsw.service.fullname" . }}:{{ .Values.servicePort }}; +include uwsgi_params; +uwsgi_hide_header Set-Cookie; +uwsgi_param HTTP_Cookie ""; diff --git a/helm/pycsw/config/pycsw-server.conf b/helm/pycsw/config/pycsw-server.conf new file mode 100644 index 00000000..e372da98 --- /dev/null +++ b/helm/pycsw/config/pycsw-server.conf @@ -0,0 +1,10 @@ +# pycsw server-level settings +keepalive_timeout 500; +proxy_connect_timeout 600; +proxy_send_timeout 600; +send_timeout 600; +client_max_body_size {{ .Values.nginx.clientMaxBodySize }}; +client_header_timeout 600; +client_body_timeout 600; +client_header_buffer_size 12288; # 12K +large_client_header_buffers 4 12288; # 12K diff --git a/helm/pycsw/config/pycswWsgi.ini b/helm/pycsw/config/pycswWsgi.ini new file mode 100644 index 00000000..4c9d53e5 --- /dev/null +++ b/helm/pycsw/config/pycswWsgi.ini @@ -0,0 +1,38 @@ +[uwsgi] +; based on https://github.com/kartoza/docker-mapproxy/blob/master/build_data/uwsgi.ini +chdir = /home/pycsw/pycsw +wsgi-file = cors.py +pidfile = /tmp/pycsw.pid +socket = :{{ .Values.pycswPort }} +processes = {{ .Values.env.uwsgi.processes }} ; Maximum number of workers allowed +cheaper = 2 ; Minimum number of workers allowed +enable-threads = true +threads = {{ .Values.env.uwsgi.threads }} +master = true +disable-logging = true +vacuum = true +die-on-term = true ; Shutdown when receiving SIGTERM (default is respawn) +need-app = true ; This parameter prevents uWSGI from starting if it is unable to find or load your application module +max-requests = 1000 ; Restart workers after this many requests +reload-on-rss = 2048 ; Restart workers after this much resident memory +buffer-size = 14336 ; 14K, Set the internal buffer size for uwsgi packet +worker-reload-mercy = 60 ; How long to wait before forcefully killing workers +wsgi-disable-file-wrapper = true +harakiri = 60 +py-callos-afterfork = true ; allow workers to trap signals +cheaper-algo = busyness +cheaper-initial = 2 ; Workers created at startup +cheaper-overload = 1 ; Length of a cycle in seconds +cheaper-step = 2 ; How many workers to spawn at a time +cheaper-busyness-multiplier = 30 ; How many cycles to wait before killing workers +cheaper-busyness-min = 20 ; Below this threshold, kill workers (if stable for multiplier cycles) +cheaper-busyness-max = 70 ; Above this threshold, spawn new workers +cheaper-busyness-backlog-alert = 2 ; Spawn emergency workers if more than this many requests are waiting in the queue +cheaper-busyness-backlog-step = 2 ; How many emergency workers to create if there are too many requests in the queue +chmod-socket = 664 +uid = 1000 +gid = 0 +http-socket = :8080 +stats = :{{ .Values.env.uwsgi.statsServer.stats }} +stats-http = {{ .Values.env.uwsgi.statsServer.enabled }} +stats-min = {{ .Values.env.uwsgi.statsServer.statsMinify }} diff --git a/helm/pycsw/templates/_helpers.tpl b/helm/pycsw/templates/_helpers.tpl new file mode 100644 index 00000000..fd126954 --- /dev/null +++ b/helm/pycsw/templates/_helpers.tpl @@ -0,0 +1,183 @@ +{{/* +Expand the name of the chart. nameOverride allows deploying multiple instances +of this chart in the same release (e.g. as aliased dependencies) without +resource name collisions. +*/}} +{{- define "pycsw.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "pycsw.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "pycsw.labels" -}} +helm.sh/chart: {{ include "pycsw.chart" . }} +{{ include "pycsw.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{ include "mclabels.labels" . }} +{{- end }} + +{{/* +Returns the tag of the chart. +*/}} +{{- define "pycsw.tag" -}} +{{- default (printf "v%s" .Chart.AppVersion) .Values.image.tag }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "pycsw.selectorLabels" -}} +app.kubernetes.io/name: {{ include "pycsw.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{ include "mclabels.selectorLabels" . }} +{{- end }} + +{{/* +Returns the environment from global if exists or from the chart's values, defaults to development +*/}} +{{- define "pycsw.environment" -}} +{{- if .Values.global.environment }} + {{- .Values.global.environment -}} +{{- else -}} + {{- .Values.environment | default "development" -}} +{{- end -}} +{{- end -}} + +{{/* +Returns the pycsw server url based on the provided values, default localhost. +Host and path are taken from nginx.route.host/path when set, otherwise from +the first nginx.route.routesMapping entry. +*/}} +{{- define "pycsw.serverURL" -}} + {{- $route := .Values.nginx.route -}} + {{- $first := (first ($route.routesMapping | default list)) | default dict -}} + {{- $host := $route.host | default $first.host -}} + {{- $path := $route.path | default $first.path -}} + {{- if not $host }} + {{- printf "http://localhost:8000" -}} + {{- else -}} + {{- $protocol := ternary "https" "http" $route.tls.enabled -}} + {{- printf "%s://%s" $protocol $host -}} + {{- if $path -}} + {{- printf "%s" $path -}} + {{- end -}} + {{- end -}} +{{- end -}} + +{{/* +Returns the cloud provider name from global if exists or from the chart's values, defaults to minikube +*/}} +{{- define "pycsw.cloudProviderFlavor" -}} +{{- if .Values.global.cloudProvider.flavor }} + {{- .Values.global.cloudProvider.flavor -}} +{{- else if .Values.cloudProvider -}} + {{- .Values.cloudProvider.flavor | default "minikube" -}} +{{- else -}} + {{ "minikube" }} +{{- end -}} +{{- end -}} + +{{/* +Returns the cloud provider docker registry url from global if exists or from the chart's values +*/}} +{{- define "pycsw.cloudProviderDockerRegistryUrl" -}} +{{- if .Values.global.cloudProvider.dockerRegistryUrl }} + {{- printf "%s/" .Values.global.cloudProvider.dockerRegistryUrl -}} +{{- else if .Values.cloudProvider.dockerRegistryUrl -}} + {{- printf "%s/" .Values.cloudProvider.dockerRegistryUrl -}} +{{- else -}} +{{- end -}} +{{- end -}} + +{{/* +Returns the cloud provider image pull secret name from global if exists or from the chart's values +*/}} +{{- define "pycsw.cloudProviderImagePullSecretName" -}} +{{- if .Values.global.cloudProvider.imagePullSecretName }} + {{- .Values.global.cloudProvider.imagePullSecretName -}} +{{- else if .Values.cloudProvider.imagePullSecretName -}} + {{- .Values.cloudProvider.imagePullSecretName -}} +{{- end -}} +{{- end -}} + +{{/* +Returns the tracing url from global if exists or from the chart's values +*/}} +{{- define "pycsw.tracingUrl" -}} +{{- if .Values.global.tracing.url }} + {{- .Values.global.tracing.url -}} +{{- else if .Values.cloudProvider -}} + {{- .Values.env.tracing.url -}} +{{- end -}} +{{- end -}} + +{{/* +Returns the tracing url from global if exists or from the chart's values +*/}} +{{- define "pycsw.metricsUrl" -}} +{{- if .Values.global.metrics.url }} + {{- .Values.global.metrics.url -}} +{{- else -}} + {{- .Values.env.metrics.url -}} +{{- end -}} +{{- end -}} + +{{/* +Returns the name of the ConfigMap provided by the consuming chart, holding +the pycsw.cfg and mappings.py keys. The value may itself be a template. +Fails rendering when not provided. +*/}} +{{- define "pycsw.teamConfigmapName" -}} +{{- $name := include "common.tplvalues.render" (dict "value" .Values.existingConfigmap "context" .) -}} +{{- required "existingConfigmap is required: set it to the name of a ConfigMap containing the keys pycsw.cfg and mappings.py (provided by the consuming chart)" $name -}} +{{- end -}} + +{{/* +Returns the postgres connection string as an environment variable value. +Uses Kubernetes dependent-variable syntax $(VAR) so it is resolved at container +start from DB_USER/DB_PASSWORD (secret) and DB_HOST/DB_PORT/DB_NAME (configmap). +Consumed by the team-provided pycsw.cfg via database=${PYCSW_DATABASE_CONNECTION}. +*/}} +{{- define "pycsw.connectionStringEnv" -}} +{{- $db := (include "common.db.merged" .) | fromYaml }} +{{- "postgresql://$(DB_USER)" -}} +{{- if .Values.env.db.requirePassword -}} +{{- ":$(DB_PASSWORD)" -}} +{{- end -}} +{{- "@$(DB_HOST):$(DB_PORT)/$(DB_NAME)" -}} +{{- if $db.sslEnabled -}} +{{- "?sslmode=require" -}} +{{- if $db.secrets.caFileKey -}} +{{- "&sslrootcert=/.postgresql/ca.pem" -}} +{{- end -}} +{{- if $db.secrets.certFileKey -}} +{{- "&sslcert=/.postgresql/cert.pem" -}} +{{- end -}} +{{- if $db.secrets.keyFileKey -}} +{{- "&sslkey=/.postgresql/key.pem" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{- define "pycsw.cors.allowedHeaders" -}} +{{- $authentication := (include "common.authentication.merged" .) | fromYaml }} +{{- $headerList := list -}} +{{- if ne .Values.env.cors.allowedHeaders "" -}} +{{- range $k, $v := (split "," .Values.env.cors.allowedHeaders) -}} +{{- $headerList = append $headerList $v -}} +{{- end -}} +{{- $headerList = uniq $headerList -}} +{{- quote (join "," $headerList) -}} +{{- end -}} +{{- end -}} diff --git a/helm/pycsw/templates/_resources.tpl b/helm/pycsw/templates/_resources.tpl new file mode 100644 index 00000000..b0931a31 --- /dev/null +++ b/helm/pycsw/templates/_resources.tpl @@ -0,0 +1,42 @@ +{{/* +Create service name as used by the service name label. +*/}} +{{- define "pycsw.service.fullname" -}} +{{- printf "%s-%s-%s" .Release.Name (include "pycsw.name" .) "service" }} +{{- end }} + +{{/* +Create configmap name as used by the service name label. +*/}} +{{- define "pycsw.configmap.fullname" -}} +{{- printf "%s-%s-%s" .Release.Name (include "pycsw.name" .) "configmap" | indent 1 }} +{{- end }} + +{{/* +Create pycsw nginx configmap name as used by the service name label. +*/}} +{{- define "pycsw.nginx-configmap.fullname" -}} +{{- printf "%s-%s-%s" .Release.Name (include "pycsw.name" .) "nginx-configmap" | indent 1 }} +{{- end }} + +{{/* +Create deployment name as used by the service name label. +*/}} +{{- define "pycsw.deployment.fullname" -}} +{{- printf "%s-%s-%s" .Release.Name (include "pycsw.name" .) "deployment" | indent 1 }} +{{- end }} + + +{{/* +Create route name as used by the service name label. +*/}} +{{- define "pycsw.route.fullname" -}} +{{- printf "%s-%s-%s" .Release.Name (include "pycsw.name" .) "route" | indent 1 }} +{{- end }} + +{{/* +Create ingress name as used by the service name label. +*/}} +{{- define "pycsw.ingress.fullname" -}} +{{- printf "%s-%s-%s" .Release.Name (include "pycsw.name" .) "ingress" | indent 1 }} +{{- end }} diff --git a/helm/pycsw/templates/_tplValues.tpl b/helm/pycsw/templates/_tplValues.tpl new file mode 100644 index 00000000..36a81f4a --- /dev/null +++ b/helm/pycsw/templates/_tplValues.tpl @@ -0,0 +1,54 @@ +{{/* +Copyright VMware, Inc. +SPDX-License-Identifier: APACHE-2.0 +*/}} + +{{/* vim: set filetype=mustache: */}} +{{/* +Renders a value that contains template perhaps with scope if the scope is present. +Usage: +{{ include "common.tplvalues.render" ( dict "value" .Values.path.to.the.Value "context" $ ) }} +{{ include "common.tplvalues.render" ( dict "value" .Values.path.to.the.Value "context" $ "scope" $app ) }} +*/}} +{{- define "common.tplvalues.render" -}} +{{- $value := typeIs "string" .value | ternary .value (.value | toYaml) }} +{{- if contains "{{" (toJson .value) }} + {{- if .scope }} + {{- tpl (cat "{{- with $.RelativeScope -}}" $value "{{- end }}") (merge (dict "RelativeScope" .scope) .context) }} + {{- else }} + {{- tpl $value .context }} + {{- end }} +{{- else }} + {{- $value }} +{{- end }} +{{- end -}} + +{{/* +Merge a list of values that contains template after rendering them. +Merge precedence is consistent with http://masterminds.github.io/sprig/dicts.html#merge-mustmerge +Usage: +{{ include "common.tplvalues.merge" ( dict "values" (list .Values.path.to.the.Value1 .Values.path.to.the.Value2) "context" $ ) }} +*/}} +{{- define "common.tplvalues.merge" -}} +{{- $dst := dict -}} +{{- range .values -}} +{{- $dst = include "common.tplvalues.render" (dict "value" . "context" $.context "scope" $.scope) | fromYaml | merge $dst -}} +{{- end -}} +{{ $dst | toYaml }} +{{- end -}} + +{{/* +End of usage example +*/}} + +{{/* +Custom definitions +*/}} + +{{- define "common.db.merged" -}} +{{- include "common.tplvalues.merge" ( dict "values" ( list .Values.db .Values.global.db ) "context" . ) }} +{{- end -}} + +{{- define "common.authentication.merged" -}} +{{- include "common.tplvalues.merge" ( dict "values" ( list .Values.authentication .Values.global.authentication ) "context" . ) }} +{{- end -}} diff --git a/helm/pycsw/templates/configmap.yaml b/helm/pycsw/templates/configmap.yaml new file mode 100644 index 00000000..5eac948a --- /dev/null +++ b/helm/pycsw/templates/configmap.yaml @@ -0,0 +1,33 @@ +{{- $chartName := include "pycsw.name" . -}} +{{- $configmapName := include "pycsw.configmap.fullname" . }} +{{- $tracingUrl := include "pycsw.tracingUrl" . -}} +{{- $metricsUrl := include "pycsw.metricsUrl" . -}} +{{- $db := (include "common.db.merged" .) | fromYaml }} +{{- if .Values.enabled -}} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ $configmapName }} +data: + {{ if .Values.env.tracing.enabled }} + TELEMETRY_TRACING_ENABLED: 'true' + TELEMETRY_TRACING_URL: {{ $tracingUrl }} + {{ end }} + {{ if .Values.env.metrics.enabled }} + TELEMETRY_METRICS_ENABLED: 'true' + TELEMETRY_METRICS_URL: {{ $metricsUrl }} + {{ end }} + uwsgi.ini: {{ tpl (.Files.Get "config/pycswWsgi.ini") . | quote }} + CORS_ENABLED: {{ .Values.env.cors.enabled | quote }} + CORS_ALLOWED_HEADERS: {{ include "pycsw.cors.allowedHeaders" . | default (quote "") }} + CORS_ALLOWED_ORIGIN: {{ .Values.env.cors.allowedOrigin | quote }} + LOG_FORMAT: {{ .Values.env.logFormat | quote }} + DB_HOST: {{ $db.host }} + DB_PORT: {{ $db.port | quote }} + DB_NAME: {{ $db.name }} + POSTGRES_ENABLE_SSL_AUTH: {{ $db.sslEnabled | quote }} + DB_SCHEMA: {{ quote .Values.env.db.schema }} + PGAPPNAME: {{ $chartName }} + COORDINATE_PRECISION: {{ quote .Values.env.bboxPrecisionDigits }} + AUTH_ENABLED: {{ .Values.nginx.authorization.enabled | quote }} #maybe deprecated or used in other networks +{{- end }} diff --git a/helm/pycsw/templates/deployment.yaml b/helm/pycsw/templates/deployment.yaml new file mode 100644 index 00000000..13534678 --- /dev/null +++ b/helm/pycsw/templates/deployment.yaml @@ -0,0 +1,219 @@ +{{- $releaseName := .Release.Name -}} +{{- $chartName := include "pycsw.name" . -}} +{{- $configmapName := include "pycsw.configmap.fullname" . }} +{{- $deploymentName := include "pycsw.deployment.fullname" . }} +{{- $cloudProviderFlavor := include "pycsw.cloudProviderFlavor" . -}} +{{- $cloudProviderDockerRegistryUrl := include "pycsw.cloudProviderDockerRegistryUrl" . -}} +{{- $cloudProviderImagePullSecretName := include "pycsw.cloudProviderImagePullSecretName" . -}} +{{- $db := (include "common.db.merged" .) | fromYaml }} +{{ $sslSecretName := ternary .Values.authentication.ssl.externalSecretName (printf "%s%s" .Release.Name "-open-ssl") .Values.authentication.ssl.useExternal }} +{{ $postgresSecretName := ternary $db.secrets.externalSecretName (printf "%s%s" .Release.Name "-postgres-secret") $db.secrets.useExternal }} +{{- $imageTag := include "pycsw.tag" . -}} +{{- $serverURL := include "pycsw.serverURL" . -}} + +{{- if .Values.enabled -}} + +{{- $teamConfigmapName := include "pycsw.teamConfigmapName" . }} + +{{- if .Values.authentication.ssl.enabled }} +{{- if and .Values.authentication.ssl.useExternal (not .Values.authentication.ssl.externalSecretName) }} +{{- fail "When using external ssl secret the value for .Values.authentication.ssl.externalSecretName should be set" }} +{{- end }} +{{- end }} + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ $deploymentName }} + labels: + app: {{ $chartName }} + component: {{ $chartName }} + environment: {{ include "pycsw.environment" . }} + release: {{ $releaseName }} + {{- include "pycsw.labels" . | nindent 4 }} + annotations: + collectord.io/index: {{ quote .Values.splunkIndex }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: {{ $chartName }} + release: {{ $releaseName }} + run: {{ $releaseName }}-{{ $chartName }} + {{- include "pycsw.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + app: {{ $chartName }} + release: {{ $releaseName }} + run: {{ $releaseName }}-{{ $chartName }} + {{- include "pycsw.selectorLabels" . | nindent 8 }} + annotations: + {{- if .Values.resetOnConfigChange }} + checksum/configmap: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + {{- end }} + {{- if .Values.podAnnotations }} + {{- toYaml .Values.podAnnotations | nindent 8 }} + {{- end }} + {{ include "mclabels.annotations" . | nindent 8 }} + + spec: + {{- if $cloudProviderImagePullSecretName }} + imagePullSecrets: + - name: {{ $cloudProviderImagePullSecretName | quote }} + {{- end }} + containers: + - name: {{ $releaseName }} + {{- with .Values.image }} + image: {{ $cloudProviderDockerRegistryUrl }}{{ .repository }}:{{ $imageTag }} + {{- end }} + imagePullPolicy: {{ .Values.imagePullPolicy }} + volumeMounts: + {{- if .Values.global.ca.secretName }} + - mountPath: {{ printf "%s/%s" .Values.global.ca.path .Values.global.ca.key | quote }} + name: root-ca + subPath: {{ quote .Values.global.ca.key }} + {{- end }} + {{- if .Values.global.ca.secretName }} + - name: root-ca + mountPath: "/usr/local/share/ca-certificates" + {{- end }} + {{- if .Values.authentication.ssl.enabled }} + - name: open-ssl + mountPath: "/etc/ssl/openssl.cnf" + subPath: "openssl.cnf" + {{- end }} + - name: pyscw-config + mountPath: /etc/pycsw/pycsw.cfg + subPath: pycsw.cfg + - name: pyscw-mappings + mountPath: /etc/pycsw/mappings.py + subPath: mappings.py + - mountPath: "/pycsw/uwsgi.ini" + name: uwsgi-config + subPath: uwsgi.ini + {{- if $db.sslEnabled }} + {{- if $db.secrets.caFileKey }} + - name: ca-file + mountPath: /certs/ca.crt + subPath: ca.pem + {{- end }} + {{- if $db.secrets.keyFileKey }} + - name: key-file + mountPath: /certs/key.pem + subPath: key.pem + {{- end }} + {{- if $db.secrets.certFileKey }} + - name: cert-file + mountPath: /certs/cert.pem + subPath: cert.pem + {{- end }} + {{- end }} + env: + {{- if .Values.global.ca.secretName }} + - name: REQUESTS_CA_BUNDLE + value: {{ printf "%s/%s" .Values.global.ca.path .Values.global.ca.key | quote }} + - name: NODE_EXTRA_CA_CERTS + value: {{ printf "%s/%s" .Values.global.ca.path .Values.global.ca.key | quote }} + {{- end }} + - name: PYCSW_SERVER_URL + value: {{ $serverURL }} + - name: PYCSW_MAPPINGS_FILEPATH + value: /etc/pycsw/mappings.py + - name: DB_USER + valueFrom: + secretKeyRef: + name: {{ $postgresSecretName }} + key: username + {{- if .Values.env.db.requirePassword }} + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: {{ $postgresSecretName }} + key: password + {{- end }} + - name: PYCSW_DATABASE_CONNECTION + value: {{ include "pycsw.connectionStringEnv" . | quote }} + - name: POSTGRES_CERTS_MOUNT_PATH + value: /certs + - name: POSTGRES_CERT_FILE_NAME + value: cert.pem + - name: POSTGRES_KEY_FILE_NAME + value: key.pem + envFrom: + - configMapRef: + name: {{ $configmapName }} + ports: + - name: http + containerPort: {{ .Values.pycswPort }} + protocol: TCP + readinessProbe: + httpGet: + path: /csw?service=CSW&request=GetCapabilities&version=3.0.0 + port: 8080 + initialDelaySeconds: 30 + timeoutSeconds: 5 + {{- if .Values.resources.enabled }} + resources: + {{- toYaml .Values.resources.value | nindent 12 }} + {{- end }} + ################################################## uwsgi exporter deployment############################################## + {{- if .Values.env.metrics.enabled }} + {{- template "pycsw-chart.uwsgi-exporter-container" (merge (dict "releaseName" .releaseName "chartName" .chartName "cloudProviderDockerRegistryUrl" $cloudProviderDockerRegistryUrl "resources" .Values.uwsgiExporter.resources) .) }} + {{- end }} + volumes: + {{- if .Values.global.ca.secretName }} + - name: root-ca + secret: + secretName: {{ .Values.global.ca.secretName }} + {{- end }} + {{- if .Values.authentication.ssl.enabled }} + - name: open-ssl + secret: + secretName: {{ $sslSecretName }} + {{- end }} + - name: pyscw-config + configMap: + name: {{ $teamConfigmapName }} + items: + - key: "pycsw.cfg" + path: "pycsw.cfg" + - name: pyscw-mappings + configMap: + name: {{ $teamConfigmapName }} + items: + - key: "mappings.py" + path: "mappings.py" + - name: uwsgi-config + configMap: + name: {{ $configmapName }} + items: + - key: "uwsgi.ini" + path: "uwsgi.ini" + {{- if $db.sslEnabled }} + {{- if $db.secrets.caFileKey }} + - name: ca-file + secret: + secretName: {{ $postgresSecretName }} + items: + - key: {{ $db.secrets.caFileKey }} + path: ca.pem + {{- end }} + {{- if $db.secrets.keyFileKey }} + - name: key-file + secret: + secretName: {{ $postgresSecretName }} + items: + - key: {{ $db.secrets.keyFileKey }} + path: key.pem + {{- end }} + {{- if $db.secrets.certFileKey }} + - name: cert-file + secret: + secretName: {{ $postgresSecretName }} + items: + - key: {{ $db.secrets.certFileKey }} + path: cert.pem + {{- end }} + {{- end }} +{{- end -}} diff --git a/helm/pycsw/templates/nginx/nginx-configmap.yaml b/helm/pycsw/templates/nginx/nginx-configmap.yaml new file mode 100644 index 00000000..9be63978 --- /dev/null +++ b/helm/pycsw/templates/nginx/nginx-configmap.yaml @@ -0,0 +1,17 @@ +{{- if .Values.nginx.enabled -}} +{{- $chartName := include "pycsw.name" . -}} +{{- $releaseName := .Release.Name -}} +{{- $nginxConfigmapName := include "pycsw.nginx-configmap.fullname" . }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ $nginxConfigmapName }} + labels: + app: {{ $releaseName }}-{{ $chartName }} + component: nginx-configmap + environment: {{ .Values.environment }} + release: {{ $releaseName }} +data: + pycsw-location.conf: {{ tpl (.Files.Get "config/pycsw-location.conf") . | quote }} + pycsw-server.conf: {{ tpl (.Files.Get "config/pycsw-server.conf") . | quote }} +{{- end }} diff --git a/helm/pycsw/templates/service.yaml b/helm/pycsw/templates/service.yaml new file mode 100644 index 00000000..6e0e522b --- /dev/null +++ b/helm/pycsw/templates/service.yaml @@ -0,0 +1,32 @@ +{{- $releaseName := .Release.Name -}} +{{- $chartName := include "pycsw.name" . -}} +{{- $cloudProviderFlavor := include "pycsw.cloudProviderFlavor" . -}} +{{- $serviceName := include "pycsw.service.fullname" . -}} +{{- if .Values.enabled -}} +apiVersion: v1 +kind: Service +metadata: + name: {{ $serviceName }} + labels: + app: {{ $chartName }} + component: {{ $chartName }} + environment: {{ include "pycsw.environment" . }} + release: {{ $releaseName }} + {{- include "pycsw.labels" . | nindent 4 }} +spec: + {{- if eq $cloudProviderFlavor "minikube" }} + type: NodePort + {{- end }} + ports: + - port: {{ .Values.servicePort }} + targetPort: {{ .Values.pycswPort }} + protocol: TCP + {{- if eq $cloudProviderFlavor "minikube" }} + nodePort: {{ .Values.nodePort }} + {{- end }} + selector: + app: {{ $chartName }} + release: {{ $releaseName }} + run: {{ $releaseName }}-{{ $chartName }} + {{- include "pycsw.selectorLabels" . | nindent 4 }} +{{- end }} diff --git a/helm/pycsw/templates/uwsgi-exporter/uwsgi-exporter-container.yaml b/helm/pycsw/templates/uwsgi-exporter/uwsgi-exporter-container.yaml new file mode 100644 index 00000000..300d463a --- /dev/null +++ b/helm/pycsw/templates/uwsgi-exporter/uwsgi-exporter-container.yaml @@ -0,0 +1,33 @@ +{{- define "pycsw-chart.uwsgi-exporter-container" }} + - name: uwsgi-exporter + image: {{ .cloudProviderDockerRegistryUrl }}{{ .Values.uwsgiExporter.image.repository }}:{{ .Values.uwsgiExporter.image.tag }} + imagePullPolicy: {{ .Values.imagePullPolicy }} + args: + - "--stats.uri=http://localhost:{{ .Values.env.uwsgi.statsServer.stats }}" + - "--log.level={{ .Values.uwsgiExporter.env.logLevel }}" + {{- if .resources.enabled }} + resources: + {{- toYaml .resources.value | nindent 12 }} + {{- end }} + livenessProbe: + httpGet: + path: "/-/healthy" + port: "uwsgimetrics" + initialDelaySeconds: 2 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + successThreshold: 1 + readinessProbe: + httpGet: + path: "/metrics" + port: "uwsgimetrics" + initialDelaySeconds: 2 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + successThreshold: 1 + ports: + - name: "uwsgimetrics" + containerPort: {{ .Values.uwsgiExporter.port }} +{{- end }} diff --git a/helm/pycsw/values.yaml b/helm/pycsw/values.yaml new file mode 100644 index 00000000..cf1cc5f0 --- /dev/null +++ b/helm/pycsw/values.yaml @@ -0,0 +1,262 @@ +global: + cloudProvider: + dockerRegistryUrl: "" + imagePullSecretName: "" + flavor: openshift + tracing: {} + metrics: {} + environment: "" + opalaEnabled: false + ca: + secretName: "" + path: "/usr/local/share/ca-certificates" + key: "ca.crt" + db: + host: "" + name: "" + port: 5432 + sslEnabled: false + secrets: + useExternal: true + externalSecretName: "" + caFileKey: "" + certFileKey: "" + keyFileKey: "" + +mclabels: + #environment: development + component: backend + partOf: serving + # owner and gisDomain must be set by the consuming chart + owner: "" + gisDomain: "" + prometheus: + enabled: true + port: 9117 + +# override the chart name used in resource names/labels; required when deploying +# multiple instances of this chart in the same release (aliased dependencies) +nameOverride: "" + +# REQUIRED: name of a ConfigMap provided by the consuming chart, containing the keys: +# pycsw.cfg - the pycsw configuration file +# mappings.py - the pycsw field mappings +# The value may be a template, e.g. '{{ .Release.Name }}-pycsw-config' +existingConfigmap: "" + +enabled: true +environment: development +replicaCount: 1 +resetOnConfigChange: true +splunkIndex: "" +imagePullPolicy: Always + +cloudProvider: + dockerRegistryUrl: "" + imagePullSecretName: "" + flavor: openshift + +authentication: + ssl: + enabled: false + useExternal: false + externalSecretName: "" +db: + host: "" + name: "" + port: 5432 + sslEnabled: false + secrets: + useExternal: false + externalSecretName: "" + caFileKey: "" + certFileKey: "" + keyFileKey: "" + +image: + repository: pycsw + # defaults to the chart appVersion (prefixed with v) when empty + tag: "" + +pycswPort: 8000 +nodePort: 30018 +servicePort: 8080 + +# add pod annotations +# example: +# podAnnotations: +# annotation1: annotation-value-1 +# annotation2: annotation-value-2 +podAnnotations: {} + +env: + logFormat: >- + %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" %({x-forwarded-for}i)s %(L)s + bboxPrecisionDigits: 10 + uwsgi: + processes: 6 + threads: 10 + statsServer: + enabled: true + stats: 1717 + statsMinify: true + cors: + enabled: true + allowedOrigin: "*" + allowedHeaders: "x-api-key" + db: + # database schema holding the records table, e.g. RasterCatalogManager + schema: "" + requirePassword: true + tracing: + enabled: false + url: http://localhost:55681/v1/trace + metrics: + enabled: true + url: http://localhost:55681/v1/metrics + +resources: + enabled: true + value: + limits: + cpu: 2 + memory: 1200Mi + requests: + cpu: 2 + memory: 1200Mi + +nginx: + enabled: true + # distinct nameOverride per instance is REQUIRED when deploying multiple + # pycsw instances in one release: nginx (>=2.2.1) derives its pod selector + # labels from nameOverride, so identical values make services select each + # other's pods + nameOverride: "" + fullnameOverride: "" + replicaCount: 2 + image: + repository: common/nginx + # defaults to the nginx chart appVersion when empty + tag: "" + mclabels: + #environment: development + partOf: serving + # owner and gisDomain must be set by the consuming chart + owner: "" + gisDomain: "" + prometheus: + enabled: true + port: 9117 + nginx: + extensions: + server: + enabled: true + fileName: pycsw-server.conf + location: + enabled: true + fileName: pycsw-location.conf + port: 8080 + targetPort: 8080 + nodePort: 30003 + # rendered into config/pycsw-server.conf (client_max_body_size) + clientMaxBodySize: 5000 + + opentelemetry: + serviceName: pycsw-nginx + exporterHost: infra-otel.infra-services + exporterPort: 4317 + samplerMethod: "AlwaysOff" + ratio: 10 + + backend: + enabled: false + + prometheusExporter: + enabled: true + image: + repository: common/nginx-prometheus-exporter + tag: 1.5.1 + pullPolicy: IfNotPresent + resources: + enabled: true + value: + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 100m + memory: 128Mi + + authorization: + enabled: false + # authorization domain of the consuming team, e.g. raster / dem / 3d + domain: "" + url: http://opa-service/v1/data/http/authz/decision + + # when using nameOverride, override the configmap name accordingly: + # "{{ .Release.Name }}--nginx-configmap" + extraVolumes: + - name: nginx-config + configMap: + name: "{{ .Release.Name }}-pycsw-nginx-configmap" + + extraVolumeMounts: + - name: nginx-config + mountPath: "/etc/nginx/conf.d/extensions/pycsw-location.conf" + subPath: pycsw-location.conf + - name: nginx-config + mountPath: "/etc/nginx/conf.d/extensions/pycsw-server.conf" + subPath: pycsw-server.conf + + resources: + enabled: true + value: + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 100m + memory: 128Mi + + route: + enabled: true + # routesMapping must be set by the consuming chart, e.g. + # routesMapping: + # - path: /api/raster/v1 + # host: + routesMapping: [] + timeout: + enabled: false # defaults to 30s by openshift + duration: 60s # supported units (us, ms, s, m, h, d) + tls: + enabled: true + useCerts: false + certificate: "" + key: "" + caCertificate: "" + + ingress: + enabled: false + ingressClassName: "" + # ingressMapping must be set by the consuming chart when ingress is enabled + ingressMapping: [] + tls: + enabled: true + useExternal: "" + +uwsgiExporter: + image: + repository: "timonwong/uwsgi-exporter" + tag: "latest" + env: + logLevel: info # one of [debug, info, warn, error] + port: 9117 + resources: + enabled: true + value: + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 100m + memory: 128Mi diff --git a/release-please-config.json b/release-please-config.json index 4aa7d069..207662f6 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -34,6 +34,16 @@ "type": "yaml", "path": "helm/raster/Chart.yaml", "jsonpath": "$.appVersion" + }, + { + "type": "yaml", + "path": "helm/pycsw/Chart.yaml", + "jsonpath": "$.version" + }, + { + "type": "yaml", + "path": "helm/pycsw/Chart.yaml", + "jsonpath": "$.appVersion" } ] }