diff --git a/docs/index.rst b/docs/index.rst index c2ee0db6..300a7dc8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -14,10 +14,6 @@ Welcome to CogStack-Nifi's documentation! news.md nifi/main.md security/main.md - security/certificates.md - security/elasticsearch_opensearch.md - security/nifi.md - security/services.yml deploy/main.md deploy/services.md deploy/workflows.md diff --git a/docs/security/certificates.md b/docs/security/certificates.md index c10267ce..6b6dc134 100644 --- a/docs/security/certificates.md +++ b/docs/security/certificates.md @@ -1,4 +1,4 @@ -# Certificates and Root CA +## πŸ›οΈ Certificates and Root CA This section describes the full structure of the `security/certificates/` directory and explains how certificates are generated, organized, and used across CogStack-NiFi services. @@ -8,7 +8,7 @@ This Root CA signs all service certificates (NiFi, OpenSearch, Kibana, JupyterHu --- -## πŸ“‚ Directory structure +### πŸ“‚ Certificate directory structure ```text security/ @@ -55,7 +55,7 @@ security/ --- -## βš™οΈ Environment configuration +### βš™οΈ Environment configuration All certificate-generation scripts source variables from `.env` files under `security/env/`: @@ -66,7 +66,7 @@ All certificate-generation scripts source variables from `.env` files under `sec | `certificates_nifi.env` | NiFi keystore/truststore names and passwords. | | `users_*.env` | Default credentials used by generation scripts. | -## πŸ“œ openssl-x509.conf +### πŸ“œ openssl-x509.conf Set up a reusable certificate config to define SANs and subject. This is used globally for all services except ES native. Feel free to add custom DNS @@ -155,7 +155,7 @@ CN = cogstack --- -## πŸ› οΈ Generation workflow +### πŸ› οΈ Generation workflow 1. **Generate Root CA** @@ -197,7 +197,7 @@ CN = cogstack --- -## 🧠 Best practices +### 🧠 Best practices - **Do not commit** private keys (`*.key`, `*.p12`, `*.jks`) to version control. - **Back up** the Root CA files securely β€” they’re your trust anchor. diff --git a/docs/security/elasticsearch_opensearch.md b/docs/security/elasticsearch_opensearch.md index 10c4d64b..0a5451e7 100644 --- a/docs/security/elasticsearch_opensearch.md +++ b/docs/security/elasticsearch_opensearch.md @@ -1,4 +1,4 @@ -# Elasticsearch / OpenSearch Security +## 🌐 Elasticsearch / OpenSearch Security This section describes how to secure both **Elasticsearch (native)** and **OpenSearch** clusters used in the CogStack-NiFi stack, including certificate setup, user management, and role configuration. @@ -6,7 +6,7 @@ All related certificates are stored in `security/certificates/elastic/`, and are --- -## πŸ”’ Overview +### πŸ”’ Overview Both **Elasticsearch** and **OpenSearch** deployments require: @@ -18,7 +18,7 @@ Certificates and credentials are generated using the scripts provided in `securi --- -## πŸ“„ Environment files used +### πŸ“„ Environment files used All scripts reference the following environment configuration files: @@ -38,7 +38,7 @@ cd ../security --- -## 🧩 Common certificate layout +### 🧩 Common certificate layout Certificate naming and folder structure are consistent across both ES and OpenSearch: @@ -67,9 +67,9 @@ Each version has its own generation scripts, but they all depend on the same `.e --- -## πŸ—οΈ Generating certificates +### πŸ—οΈ Generating certificates -### Elasticsearch (native) +#### Elasticsearch (native) To generate certificates for Elasticsearch: @@ -94,7 +94,7 @@ Make sure the environment variables are set correctly before running the script. --- -### OpenSearch +#### OpenSearch For OpenSearch nodes: @@ -149,9 +149,9 @@ All certificate references in `services/kibana/config/kibana_opensearch.yml` or --- -## πŸ” Users and roles +### πŸ” Users and roles -### OpenSearch +#### OpenSearch 1. Edit `security/es_roles/opensearch/internal_users.yml` to define users. 2. Optionally generate password hashes: @@ -173,7 +173,7 @@ OpenSearch includes default roles (`admin`, `kibanaserver`, `readall`, `snapshot --- -### Elasticsearch (native) +#### Elasticsearch (native) Run after containers start: @@ -197,7 +197,7 @@ You can modify credentials in `security/env/elasticsearch_users.env`. --- -## ⚠️ Notes +#### ⚠️ Notes - The `security/certificates/` folder is also **mounted inside NiFi** so NiFi processors can access ES/OS securely without restarting. - For OpenSearch role details, see the [OpenSearch Security Plugin documentation](https://opensearch.org/docs/latest/security-plugin/index/). @@ -205,7 +205,7 @@ You can modify credentials in `security/env/elasticsearch_users.env`. --- -## βœ… Verification +#### βœ… Verification To verify HTTPS access and trust: @@ -216,5 +216,5 @@ curl -vk --cacert ./root-ca.pem https://elasticsearch-1:9200 To check inter-node encryption (inside a container): ```bash -openssl s_client -connect elasticsearch-2:9300 -CAfile ./root-ca.pem +openssl s_client -connect elasticsearch-1:9300 -CAfile ./root-ca.pem ``` diff --git a/docs/security/main.md b/docs/security/main.md index 6d62b543..0e9453e7 100644 --- a/docs/security/main.md +++ b/docs/security/main.md @@ -1,4 +1,6 @@ -# Security Overview +# πŸ›‘οΈ Security + +## πŸ—ΊοΈ Overview All core CogStack-NiFi services β€” including **NiFi**, **Elasticsearch/OpenSearch**, **Kibana/OpenSearch Dashboards**, **JupyterHub**, **NGINX** and **Gitea** β€” are now deployed with **HTTPS enabled by default**. Each component is provisioned with its own X.509 certificates issued by the shared root CA generated via the `create_root_ca_cert.sh` script. @@ -15,7 +17,7 @@ Security is achieved through: > ⚠️ **Important:** Always generate unique certificates and credentials for each deployment. > The repository provides sample certificates for demonstration only. -## Components secured with HTTPS +## 🧩 Components secured with HTTPS | Service | HTTPS/TLS Enabled | Certificate Location | Script(s) Used | |----------|------------------|----------------------|----------------| @@ -29,7 +31,7 @@ Security is achieved through: --- -## Folder structure +## πŸ“‚ Folder structure The `security/` directory centralizes all certificate, credential, and role management for CogStack-NiFi. Below is the high-level structure with explanations for each sub-folder. @@ -69,14 +71,22 @@ security/ --- -## Next steps +```{include} ./certificates.md + +--- + +```{include} ./services.md + +--- + +```{include} ./main.md + +--- + +```{include} ./nifi.md -Refer to the detailed pages for each topic: +--- -```{toctree} -:maxdepth: 2 -:caption: Security Topics +```{include} ./elasticsearch_opensearch.md -certificates -services -nifi +--- diff --git a/docs/security/nifi.md b/docs/security/nifi.md index 9f609a51..b5406bd7 100644 --- a/docs/security/nifi.md +++ b/docs/security/nifi.md @@ -1,4 +1,4 @@ -# πŸ” NiFi/NiFi Registry TLS & Admin Access Setup (with NGINX) +## πŸ” NiFi/NiFi Registry TLS & Admin Access Setup (with NGINX) This guide documents how to configure TLS and certificate-based admin access for Apache NiFi Registry (v1.26+) using OpenSSL-generated certificates and NGINX as a reverse proxy. For background on how these certificates are generated, see [Certificates and Root CA](certificates.md). @@ -6,7 +6,7 @@ This section focuses on applying those certificates to secure **NiFi** and **NiF --- -## πŸ“ Folder Structure +### πŸ“ Folder Structure ```text security/certificates @@ -59,9 +59,9 @@ Example (`nifi/conf/nifi.properties`): Default username :
-``` -username: admin -password: cogstackNiFi +```text + username: admin + password: cogstackNiFi ``` - the `login-identity-providers.xml` file in `/nifi/conf/` stores the password for the user account, to generate a password one must use the following command within the container : `/opt/nifi/nifi-current/bin/nifi.sh set-single-user-credentials USERNAME PASSWORD`, once done, you would need to copy the file from `/opt/nifi/nifi-current/conf/login-identity-providers.xml` locally with docker cp and replace the one in the `nifi/conf` folder and rebuild the container. @@ -76,7 +76,7 @@ Troubleshooting Security : if you encounter errors related to sensitive key prop If for some reason you do not wish to authenticate every time you connect to NiFi, you can enable the client certificates in the [nginx.conf](../services/nginx/config/nginx.conf) line 86-87 and delete the commented lines. -## `nifi-nginx` +### `nifi-nginx` Alternatively, one can secure the access to selected services by using NGINX reverse proxy. This may be essential in case some of the web services that need to be exposed to end-users do not offer SSL encryption. @@ -88,7 +88,7 @@ In order to be able to properly access the nifi instance securely, you also need --- -## πŸ” authorizers.xml – Multiple Admins +### πŸ” authorizers.xml – Multiple Admins ```xml @@ -112,7 +112,7 @@ In order to be able to properly access the nifi instance securely, you also need --- -## 🌐 `nifi-registry.properties` +### 🌐 `nifi-registry.properties` ```properties nifi.registry.web.context.path=/nifi-registry @@ -123,7 +123,7 @@ nifi.registry.security.identity.mapping.pattern.dn=^.*?CN=(.*?)(,|$) --- -## 🌍 NGINX Reverse Proxy Example +### 🌍 NGINX Reverse Proxy Example ```nginx server { @@ -165,7 +165,7 @@ server { --- -## βœ… Final Checklist +### βœ… Final Checklist - [x] Certificate CN matches identity in `authorizers.xml` - [x] NGINX uses correct client cert + root CA @@ -175,7 +175,7 @@ server { --- -## πŸ§ͺ Test Identity +### πŸ§ͺ Test Identity ```bash curl -vk --cert ./nifi.pem --key ./nifi.key https://localhost:18443/nifi-registry-api/tenants/me @@ -183,7 +183,7 @@ curl -vk --cert ./nifi.pem --key ./nifi.key https://localhost:18443/nifi-r --- -## πŸ›  If Settings Icon Missing +### πŸ›  If Settings Icon Missing - Double-check that the user shown by `/tenants/me` is **exactly** the same as `Initial Admin Identity` - Recreate `authorizations.xml` if needed by clearing it @@ -199,7 +199,7 @@ To ensure a seamless integration between **NiFi** and **NiFi Registry**, especia If your admin certificate resolves to identity `CN=cogstack`, then: -#### βœ… You must define it in both `authorizers.xml` files: +#### βœ… You must define it in both `authorizers.xml` files **For NiFi** (`conf/authorizers.xml`): @@ -213,4 +213,4 @@ If your admin certificate resolves to identity `CN=cogstack`, then: C=UK, ST=London, L=UK, O=cogstack, OU=cogstack, CN=cogstack ``` -> πŸ“ If you’re using `nifi.security.identity.mapping.pattern.dn` to reduce the full DN to something simpler like `cogstack`, ensure that the mapped identity is **also** consistent and declared in both systems. \ No newline at end of file +> πŸ“ If you’re using `nifi.security.identity.mapping.pattern.dn` to reduce the full DN to something simpler like `cogstack`, ensure that the mapped identity is **also** consistent and declared in both systems. diff --git a/docs/security/services.md b/docs/security/services.md index dde2e082..c1d71b0e 100644 --- a/docs/security/services.md +++ b/docs/security/services.md @@ -1,10 +1,10 @@ -# Service Security +## Security for CogStack provided other services -## General services TLS configuration +### General services TLS configuration All services in cogstack that are not listed in `deploy/services.yml`and use TLS will be described on this page. For services that are still part of the stack but in `dervices/` (again, service not also present in `deploy/services.yml`) the TLS setup is handled differently, and the setup is described in each service's README.md. Generally, most should just use the root-ca certs from `security/certificates/root/`. -## Gitea TLS Configuration +### Gitea TLS Configuration This section describes how **Gitea** is secured using the shared **Root Certificate Authority (CA)** generated by `create_root_ca_cert.sh`. @@ -27,13 +27,12 @@ security/certificates/root/ | `root-ca.key` | Root CA private key (used only when generating new certificates) | | `root-ca.p12` | Optional PKCS#12 keystore (not required by Gitea) | - ### 🧠 Notes - The **Root CA** (`root-ca.pem`) is shared across all CogStack services for internal TLS trust. - You do **not** need to create a new `gitea.crt` or `gitea.key`; the Root CA cert/key pair is sufficient. - Ensure `root-ca.key` remains private and is not committed to version control. -- The same CA also secures NiFi, Elasticsearch, OpenSearch, Kibana, and JupyterHub. +- The same CA also secures NiFi, ElasticSearch, OpenSearch, Kibana, and JupyterHub. --- @@ -46,5 +45,3 @@ curl -vk --cacert ./security/certificates/root/root-ca.pem https://gitea.local:2 ``` You should see a valid TLS handshake and an HTTP 200 response. - ----