ODP-5751: Enable Knox automation#14
Draft
JeffreySmith wants to merge 4 commits into
Draft
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a new knox-enablement/ automation toolkit to configure and enable Apache Knox in an Ambari-managed Hadoop cluster, including proxy-user setup, topology/whitelist updates, SSO+LDAP configuration, certificate export/import, and service restarts.
Changes:
- Adds a Knox topology template plus steps to render/update topology URLs and dispatch whitelist settings in Ambari.
- Adds orchestration steps for Knox start/Demo LDAP, Ambari SSO + LDAP setup, Ranger trust/cert handling, and ordered service restarts.
- Adds supporting modules (Ambari API/config wrapper, SSH wrapper, state caching) plus CLI entrypoint, setup script, and documentation.
Reviewed changes
Copilot reviewed 26 out of 28 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| knox-enablement/templates/knox-topology.j2 | New topology template used as the base for automated topology configuration. |
| knox-enablement/steps/update_whitelist.py | Automates gateway.dispatch.whitelist generation and update based on cluster domain. |
| knox-enablement/steps/update_topology.py | Renders topology by replacing URL placeholders with cluster hostnames and optionally injects HA provider XML. |
| knox-enablement/steps/start_knox.py | Verifies Knox is installed, starts Knox if needed, and starts Demo LDAP. |
| knox-enablement/steps/set_proxy_users.py | Sets Knox proxy-user configuration in core-site. |
| knox-enablement/steps/restart_services.py | Restarts services with stale configs and always restarts Knox. |
| knox-enablement/steps/restart_ambari.py | Restarts Ambari server after SSO/LDAP setup. |
| knox-enablement/steps/import_knox_cert.py | Imports Knox cert into Java cacerts on Ranger hosts and configures Ranger truststore. |
| knox-enablement/steps/export_knox_cert.py | Exports Knox gateway cert via knoxcli.sh and stores it in shared state. |
| knox-enablement/steps/base.py | Provides shared helpers to construct AmbariConfigs/SSH client instances. |
| knox-enablement/steps/ambari_sso_setup.py | Runs ambari-server setup-sso non-interactively to configure Knox SSO for Ambari. |
| knox-enablement/steps/ambari_ldap_setup.py | Runs ambari-server setup-ldap and sync-ldap for Knox Demo LDAP. |
| knox-enablement/steps/add_ranger_policy.py | Adds a Ranger policy to permit Ambari access via Knox for the public group. |
| knox-enablement/steps/init.py | Exposes step modules for imports/registration. |
| knox-enablement/setup.sh | Creates venv and installs dependencies for running the automation. |
| knox-enablement/requirements.txt | Declares Python dependencies for the automation scripts. |
| knox-enablement/README.md | Documents structure, setup, steps, and flows for the tool. |
| knox-enablement/modules/ssh_client.py | Paramiko-based SSH wrapper used by steps for remote CLI execution. |
| knox-enablement/modules/service_utils.py | Ambari service-operation helper (start/stop/restart, stale config checks, demo LDAP). |
| knox-enablement/modules/knox_state.py | Shared singleton state for Knox host/cert data across steps. |
| knox-enablement/modules/configs.py | Ambari config helper (includes a wrapper class used by steps). |
| knox-enablement/modules/ambari_state.py | Cached Ambari cluster/service/host discovery helper. |
| knox-enablement/modules/init.py | Exposes module APIs for step imports. |
| knox-enablement/main.py | CLI entrypoint defining steps and named flows to run the automation. |
| knox-enablement/env.example | Example environment configuration for SSH/Ambari/Ranger. |
| knox-enablement/config/settings.py | Loads .env and defines configuration objects used by modules/steps. |
| knox-enablement/config/init.py | Marks config as a package. |
| .gitignore | Adds ignores for venv and Python bytecode. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+23
to
+26
| class RangerConfig: | ||
| USERNAME = os.getenv("RANGER_USERNAME", "admin") | ||
| PASSWORD = os.getenv("RANGER_PASSWORD", "Acceldata@01") | ||
| PORT = int(os.getenv("RANGER_PORT", 6080)) |
Comment on lines
+28
to
+33
| # ------------------------------------------- | ||
| # Ranger Configuration | ||
| # ------------------------------------------- | ||
| RANGER_USERNAME=admin | ||
| RANGER_PASSWORD=Acceldata@01 | ||
| RANGER_PORT=6080 |
Comment on lines
+147
to
+149
| print(f"\n --- FULL CERTIFICATE ---") | ||
| print(pem_content) | ||
| print("=" * 60) |
Comment on lines
+200
to
+214
| def get_ambari_server_host(self) -> str: | ||
| """Get the Ambari server hostname from API.""" | ||
| endpoint = "/api/v1/services/AMBARI/components/AMBARI_SERVER" | ||
| result = self._api_request(endpoint) | ||
|
|
||
| if result and result.get("hostComponents"): | ||
| return result["hostComponents"][0]["RootServiceHostComponents"]["host_name"] | ||
|
|
||
| # Fallback: get first host | ||
| endpoint = "/api/v1/hosts" | ||
| result = self._api_request(endpoint) | ||
| if result and result.get("items"): | ||
| return result["items"][0]["Hosts"]["host_name"] | ||
|
|
||
| return AmbariConfig.HOST |
Comment on lines
+53
to
+59
| if knox_state == "INSTALLED": | ||
| print(" Knox is installed but not started. Starting...") | ||
| success = service_utils.start_service("KNOX") | ||
| if not success: | ||
| raise RuntimeError("Failed to start Knox service") | ||
| knox_state = service_utils.get_service_state("KNOX") | ||
| print(f" Knox state after start: {knox_state}") |
Comment on lines
+60
to
+65
| resp = requests.get( | ||
| f"{base_url}/service/{knox_service_name}/policy", | ||
| auth=auth, | ||
| headers=headers, | ||
| params={"policyName": policy_name} | ||
| ) |
Comment on lines
+110
to
+115
| resp = requests.post( | ||
| f"{base_url}/policy", | ||
| auth=auth, | ||
| headers=headers, | ||
| json=policy_data | ||
| ) |
Comment on lines
+24
to
+28
| def connect(self) -> None: | ||
| """Establish SSH connection.""" | ||
| self.client = paramiko.SSHClient() | ||
| self.client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) | ||
|
|
Comment on lines
+28
to
+32
| echo "" | ||
| echo "=== Setup Complete ===" | ||
| echo "Virtual environment is active. Run: python main.py step0" | ||
| echo "" | ||
| echo "To reactivate later: source venv/bin/activate" |
Comment on lines
+1
to
+11
| from steps import set_proxy_users | ||
| from steps import start_knox | ||
| from steps import update_topology | ||
| from steps import update_whitelist | ||
| from steps import restart_services | ||
| from steps import export_knox_cert | ||
| from steps import ambari_sso_setup | ||
| from steps import ambari_ldap_setup | ||
| from steps import import_knox_cert | ||
| from steps import restart_ambari | ||
| from steps import add_ranger_policy |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the work that was done by Harshith before he left. As I recall, he stated it was good to go, and that the PR essentially just needed to be raised.