Skip to content

Fix SSL cert verification failures behind enterprise inspection proxies#168

Open
edurdevic wants to merge 1 commit into
databricks:mainfrom
edurdevic:fix/ssl-cert-enterprise-proxy
Open

Fix SSL cert verification failures behind enterprise inspection proxies#168
edurdevic wants to merge 1 commit into
databricks:mainfrom
edurdevic:fix/ssl-cert-enterprise-proxy

Conversation

@edurdevic

Copy link
Copy Markdown

Problem

ucode init fails with CERTIFICATE_VERIFY_FAILED: self-signed certificate in certificate chain on networks with a corporate TLS inspection proxy, even though curl to the same AI Gateway endpoint succeeds.

Root cause: Python's urllib.request.urlopen does not load the OS trust store on all platforms. The proxy injects a self-signed CA that curl finds via the system keychain, but Python's default SSL context misses it. The error surfaces as:

Databricks Unity AI Gateway probe failed on this workspace
(network error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
self-signed certificate in certificate chain (ssl.c:1001)).

Fix

Add _make_ssl_context() in databricks.py that creates a default SSL context and additionally loads any CA bundle pointed to by REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, or SSL_CERT_FILE — the same env vars honored by curl and the requests library. All three urlopen call sites (_http_get_json, _http_post_json, discover_sql_warehouse_http_path) now use this context.

Workaround for affected customers (before this ships)

export REQUESTS_CA_BUNDLE=/path/to/enterprise-ca.pem
ucode init

Test plan

  • All existing unit tests pass (uv run pytest)
  • Lint clean (uv run ruff check . + uv run ruff format --check)
  • Manual test on a workspace behind an SSL inspection proxy

This pull request and its description were written by Isaac.

Python's urllib does not load the OS trust store on all platforms, so
workspaces behind a corporate TLS inspection proxy (which injects a
self-signed CA) fail with CERTIFICATE_VERIFY_FAILED even though curl
succeeds. Honor REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, and SSL_CERT_FILE
— the same env vars respected by curl and the requests library — so
customers can point ucode at their enterprise CA bundle.

Co-authored-by: Isaac
Comment thread src/ucode/databricks.py
ctx.load_verify_locations(cafile=ca_bundle)
except ssl.SSLError:
pass
break

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug — break fires even when load_verify_locations fails (_make_ssl_context, line ~214)

          try:
              ctx.load_verify_locations(cafile=ca_bundle)
          except ssl.SSLError:
              pass
          break   # ← outside try/except, always executes

Comment thread src/ucode/databricks.py
_debug(f"databrickscfg ({cfg_path})", f"read error: {exc}")


def _make_ssl_context() -> ssl.SSLContext:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor — _make_ssl_context() called on every request, should be cached

It hits the env vars + filesystem on every urlopen call. The rest of the file already uses @functools.cache for this pattern. Adding @functools.cache makes it a one-shot per process.

Comment thread src/ucode/databricks.py
try:
ctx.load_verify_locations(cafile=ca_bundle)
except ssl.SSLError:
pass

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OSError (e.g. permission denied) is not caught here — Path.is_file() returns True for files the current user cannot read, but load_verify_locations will then raise OSError: Permission denied, which propagates uncaught and crashes the process.

Suggest broadening the except catch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants