atServer fails siliently if it does not have enough memory to start up (could also be a docker issue ?)

[cconstab]

Is your feature request related to a problem? Please describe.

After migrating @ ssh_1 to canary although the atServer looked up in docker it was not answering to TLS connections.

AFter adding more memory to the atServer everything sprung into action.

What was interesting was the atServer did not crash and docker did not kill the atServer and nothing interesting in the logs

Describe the solution you'd like

We should check memory settings as we migrate.....

we should make sure that on startup and memory issues the atServer code whines in the logs or crashes out in docker

Describe alternatives you've considered

No response

Additional context

Here are some logs that the atServer showed when in this situation

cconstab@canary-01:~$ openssl s_client 0:5703
4057535CE9780000:error:8000006F:system library:BIO_connect:Connection refused:../crypto/bio/bio_sock2.c:114:calling connect()
4057535CE9780000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:116:
connect:errno=111
cconstab@canary-01:~$ docker service logs -f 92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.7ee3ivo3ypii@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:23:54.268620|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.7ee3ivo3ypii@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:23:54.288183|AtSecondaryServer|currentAtSign : @ssh_1
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.nlce4s2q9fug@canary-02.us-central1-b.c.secondaries.internal    | INFO|2026-01-26 16:23:30.073118|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.nlce4s2q9fug@canary-02.us-central1-b.c.secondaries.internal    | INFO|2026-01-26 16:23:30.118436|AtSecondaryServer|currentAtSign : @ssh_1
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.2zx2941nuuxf@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:22:30.927992|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.2zx2941nuuxf@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:22:30.969971|AtSecondaryServer|currentAtSign : @ssh_1
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.gyvmdnze29xr@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:23:19.624040|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.gyvmdnze29xr@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:23:19.644114|AtSecondaryServer|currentAtSign : @ssh_1
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.mdzye9x2u8xq@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:22:55.238974|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.mdzye9x2u8xq@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:22:55.265997|AtSecondaryServer|currentAtSign : @ssh_1
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.rtwee97t3edn@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:24:11.926892|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.rtwee97t3edn@canary-03.us-central1-c.c.secondaries.internal    | INFO|2026-01-26 16:24:11.965087|AtSecondaryServer|currentAtSign : @ssh_1
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.o2pk2s5r83u4@canary-02.us-central1-b.c.secondaries.internal    | INFO|2026-01-26 16:24:36.967337|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.o2pk2s5r83u4@canary-02.us-central1-b.c.secondaries.internal    | INFO|2026-01-26 16:24:36.975618|AtSecondaryServer|currentAtSign : @ssh_1
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.gwld6bsna3jl@canary-02.us-central1-b.c.secondaries.internal    | INFO|2026-01-26 16:24:54.791315|DefaultOutboundConnectionFactory|Using MTLS cert when making outbound client connections
92a7c219-e2eb-5f50-ac23-d7acba106cf5_secondary.1.gwld6bsna3jl@canary-02.us-central1-b.c.secondaries.internal    | INFO|2026-01-26 16:24:54.839642|AtSecondaryServer|currentAtSign : @ssh_1

[cpswan]

This would happen if the memory on the service has been bumped, but the yaml has been left untouched.

If we follow the process recently documented by @gkc this won't happen. But that's not how we've always done it.

[cconstab]

I think we can also have an issue without the bump.
If a atServer needs memory and gets killed and restarts with no bump.
It can start and never open the socket/startbup.