Fix datalake DNS move to 5353 - #1440
Merged
Merged
Conversation
The playground still hit intermittent
FirecrackerError('PUT /snapshot/load -> 400: Resource busy
(os error 16). Invalid TUN/TAP Backend
provided by fc-tap-<slot>')
after the earlier _shutdown / _teardown fixes. Root cause: `ip tuntap
del` is soft. If any fd holder still exists (a dying-but-not-yet-
exited firecracker process from the previous restore), the kernel
only clears the persistent flag; the interface lingers in `ip link
show` until the last fd closes. Concurrent ensure_tap then sees the
still-present interface, skips the re-add, and the next fc that
tries to open it hits EBUSY.
Fix: poll `ip link show dev <tap>` for up to 2 s after `ip tuntap
del`. If the interface disappears (normal path), return. If it's
still there after 2 s, force-delete with `ip link delete`, which
breaks any lingering fd — by then that's what we want, since the
holder didn't respond to SIGKILL in time.
The playground competes with up to 40 firecracker guests eating whole vCPUs. Under a heavy Run-all burst the async event loop starves and /api/query takes seconds to enqueue a restore — the UI feels stuck. Give the API server head-of-line over the guests: - Nice=-10: guest fc processes stay at 0, so the API always wins the scheduler when both are runnable. systemd applies this before dropping to User=ubuntu; RestrictRealtime=yes below still allows negative nice (nice is not real-time scheduling). - IOSchedulingClass=best-effort + IOSchedulingPriority=0: highest best-effort I/O priority. Modern NVMe with mq-deadline/none mostly ignores ionice, but there's no downside on schedulers that do. Already deployed on the current host; committing so the repo is consistent.
Datalake systems (clickhouse-datalake{,-partitioned},
duckdb-datalake{,-partitioned}) started failing every query with
IO Error: Could not resolve hostname
Not found address of host: clickhouse-public-datasets.s3.eu-central-1.amazonaws.com
Root cause: dnsmasq was configured with `bind-interfaces`, which
snapshots the interface address list at startup and never rebinds
for TAPs added later. The fc-tap-<slot> interfaces come and go across
the playground's lifetime (each restore/kick cycle recreates them),
so most fresh TAPs had no listener at 10.200.<slot>.1:53. The
iptables REDIRECT of the VM's UDP/53 landed on nothing.
Fix path: use `bind-dynamic` (tracks interface changes) with
`interface=fc-tap-*`. But dnsmasq's bind-dynamic still opens a
wildcard fallback socket that collides with systemd-resolved on
127.0.0.53:53. Move dnsmasq off :53 entirely — port 5353 — and
have iptables REDIRECT rewrite UDP/53 → UDP/5353 on the host.
Guests still send DNS to :53 as normal.
Also swaps port 53 → 5353 in setup_host_firewall's INPUT allowlist
so the new port is source-restricted to internal CIDR + loopback.
Verified: all six clickhouse/duckdb parquet + datalake variants
return the correct 99,997,497 row count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.