UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS 💀

Nice! We’re really enjoying WarpBuild; I’ve also had good results with Amazon CodeBuild but never used it as a GHA replacement.

Got the perfect soundtrack youtu.be/6sOpbRL8R4g?...

The difficult part is that all the smart placement decisions happen within a zone, but you first need something at the regional API layer to decide upfront where to try send the request. Getting the balance wrong might mean much worse availability at the regional level during a single zone outage.

A primary driving factor is availability: the EC2 control plane wants to get out of the shared regional path and home onto a specific AZ as quickly as possible. This way you minimise shared points of failure and allow unaffected AZs to continue working in the presence of an outage.

I forget exactly what the classic networking API used to look like but making the AZ (or in the VPC world, subnet id) parameter optional. Seems easy enough but there were hard reasons why that was difficult to support!

Soaked up a lot of internal and external effort dealing with it. One of the worst consequences was that for large customers with multiple accounts, you wanted to know that AZ names are aligned within your organisation. But if you had launched any EC2 instances, even indirectly, we couldn’t change.

Fun fact, I was on the team that owned the service behind EC2 AZ randomisation a few (ok, quite a few…) years back. It was a very early, heavy handed decision to try better distribute placement - in hindsight should have been solved with better UI and APIs.

Important correction!

Is it really that weird? I thought it’s fairly common pattern in the Unix tools world. npm works the same way to separate its own arguments from those passed to scripts. Test binary is a bit of a leap, but I’d expect someone who understands how Rust programs get compiled to have an intuition for it.