The per-user Nebi launched by jupyter-server-proxy in the singleuser pod does not stay up under jupyter-server 2.20. Opening the Nebi tab leaves it erroring, and the server log shows nebi serve being relaunched roughly every 200ms, never reaching a listening state.
I traced this to the jupyter-server version, not Nebi or jupyter-server-proxy. Isolation test on an EKS cluster, only the lab image differs:
|
JL 4.6.0 image |
JL 4.2.5 image |
| nebi binary |
quay.io/nebari/nebi:sha-be3187c |
same |
| jupyter-server-proxy |
4.4.0 |
4.4.0 |
| jupyter-server |
2.20.0 |
2.15.0 |
| Nebi under proxy |
relaunch loop, never binds |
single process, Server listening, stays up |
| Nebi tab |
errors |
works |
Same nebi binary, same jupyter-server-proxy. On 2.20 the log repeats Starting Nebi server -> Job queue initialized -> dies, never reaching server.go:196 Server listening. On 2.15 it logs Server listening once and stays up. Run directly in the pod (nebi serve --port N), nebi works on both, so the binary is fine. The break is specific to how it gets launched and managed by jupyter-server-proxy on the newer jupyter-server.
Two things probably combine here:
- jupyter-server tightened a lot of request handling between 2.16 and 2.20 (CORS origin validation, the
LoginFormHandler._redirect_safe / ?next= open-redirect fix in 2.18, a sandbox CSP in 2.20). The proxied path returns a 302, and my hunch is the proxy readiness/redirect handling no longer settles. I have not bisected the exact minor yet, 2.18 is my suspect since that is the 4-CVE release.
nebi serve runs DB migrations on every launch (~200ms cold start), so it loses the readiness race and gets relaunched before it binds.
Why it matters now: this blocks moving the image to JupyterLab 4.6, since 4.6 pulls jupyter-server 2.20. The gallery work that prompted the 4.6 bump turned out to be a config fix and runs fine on 4.2.5, so that is not the forcing function, but a future legitimate lab bump will re-break Nebi unless this is sorted.
Open question: should the fix live in the images/nebi/jupyter_server_config.py server-proxy config (readiness/timeout handling, avoid relaunch churn), or in Nebi's local serve (faster start, skip re-migrating when the DB is current)? I can bisect the jupyter-server minor if that helps narrow it.
Related: #50.
The per-user Nebi launched by jupyter-server-proxy in the singleuser pod does not stay up under jupyter-server 2.20. Opening the Nebi tab leaves it erroring, and the server log shows
nebi servebeing relaunched roughly every 200ms, never reaching a listening state.I traced this to the jupyter-server version, not Nebi or jupyter-server-proxy. Isolation test on an EKS cluster, only the lab image differs:
quay.io/nebari/nebi:sha-be3187cServer listening, stays upSame nebi binary, same jupyter-server-proxy. On 2.20 the log repeats
Starting Nebi server->Job queue initialized-> dies, never reachingserver.go:196 Server listening. On 2.15 it logsServer listeningonce and stays up. Run directly in the pod (nebi serve --port N), nebi works on both, so the binary is fine. The break is specific to how it gets launched and managed by jupyter-server-proxy on the newer jupyter-server.Two things probably combine here:
LoginFormHandler._redirect_safe/?next=open-redirect fix in 2.18, asandboxCSP in 2.20). The proxied path returns a 302, and my hunch is the proxy readiness/redirect handling no longer settles. I have not bisected the exact minor yet, 2.18 is my suspect since that is the 4-CVE release.nebi serveruns DB migrations on every launch (~200ms cold start), so it loses the readiness race and gets relaunched before it binds.Why it matters now: this blocks moving the image to JupyterLab 4.6, since 4.6 pulls jupyter-server 2.20. The gallery work that prompted the 4.6 bump turned out to be a config fix and runs fine on 4.2.5, so that is not the forcing function, but a future legitimate lab bump will re-break Nebi unless this is sorted.
Open question: should the fix live in the
images/nebi/jupyter_server_config.pyserver-proxy config (readiness/timeout handling, avoid relaunch churn), or in Nebi's localserve(faster start, skip re-migrating when the DB is current)? I can bisect the jupyter-server minor if that helps narrow it.Related: #50.