Consider running on one other provider in addition to Fly.io

[gerhard]

This is why: https://status.changelog.com/incident/513790

[pmbanugo]

What alternatives are you considering? I have Google Cloud Run and Cloudflare Containers (not yet released) on my list. I'm not sure about Cloud Run's pricing yet.

[gerhard]

I was thinking straight up K8s. I am already running production K8s in London, Paris & Amsterdam, and this prompted me to look into expanding to US East.

I am also in the process of rolling out my own distributed HTTP / TCP monitoring & status page, so this just fits right in. Oh, and I have also dabbled with Spin runtime workloads, so all roads seem to be pointing to my own distributed compute platform. This has been 12 years in the making...

I am curious about Cloudflare Container Platform too, but I am still waiting for access.

Namespace.so is on my shortlist too, but some key building blocks are still TBD.

2025 will be a really interesting year 🧑‍💻

[pmbanugo]

Spin looks interesting but I've got no clue about Web Assembly and probably won't explore that area anytime soon. Namespace is interesting but their services are for builds/CI. I didn't notice anything about compute

[nabsul]

In my opinion, a good secondary provider for the incident scenario would be a service that costs almost nothing in the regular case but can quickly fail over and scale up when it's needed. It doesn't matter if it's expensive on a per-request level because it's meant to be a stop gap during typically short (but painful) outages.

I'm thinking something that's a complete clone/backup of your system that you can fail over to, but it doesn't have to be as efficiently architected or as performant.

Maybe something like AWS's elastic container service?

Could your home lab handle serving the changelog for a brief period of time?

What about a static snapshot of the Changelog such that you can browse all the public content, but login and dynamic features are unavailable?

[gerhard]

I'm keen to switch across 2 (to begin with) providers in a continuously concurrent way (one is active, the other one is a hot standby). The reasons for this are:

• all deploys go to all providers, all the time - it sucks when you need something to just work, and you realise that the deployment is out of date, something is misconfigured, DNS does not work, certs are out of date, etc.
• compare providers on a continuous basis (maybe B is better than A, but we don't have the metrics to tell us that)
• by automatically alternating which one is active, we exercise both systems, so that when one fails, we simply make the switch-over explicit (the same switch-over that happens automatically every week? fortnight? month?)

The reason why we don't want to run across multiple providers in parallel is because the service will be exposed to each provider's hiccups.

Since I wrote this, these are the providers that I'm currently considering:

• Railway
• Northflank Service
• Koyeb Serverless Compute
• Bunny Containers
• Cloudflare Containers