Doug Madory
banner
eldomador.bsky.social
Doug Madory
@eldomador.bsky.social
Internet Analysis (BGP/NetFlow) at @kentik.bsky.social (formerly of Dyn Research and Renesys).
https://www.linkedin.com/in/dougmadory/
Let's Encrypt impacted by Cloudflare outage.
November 18, 2025 at 2:25 PM
Twitter/X traffic by source CDN:
November 18, 2025 at 2:02 PM
Reposted by Doug Madory
According to @kentik.bsky.social aggregate traffic data, @cloudflare.social 's disruptions began at 11:32 UTC.
November 18, 2025 at 1:40 PM
According to @kentik.bsky.social aggregate traffic data, @cloudflare.social 's disruptions began at 11:32 UTC.
November 18, 2025 at 1:40 PM
Be sure to check out the interactive visualization at the end!
November 14, 2025 at 8:32 PM
Tanzania Internet is offline again after a ~2 hour recovery.

Follow connectivity in Tanzania in near realtime:
ioda.inetintel.cc.gatech.edu/country/TZ?f...
October 30, 2025 at 5:51 PM
Lesson: at cloud scale, configuration management is reliability engineering. A single unchecked update can topple a continent-wide edge.

#Azure #Cloud #Outage #IncidentResponse #SRE
October 30, 2025 at 2:22 PM
Root cause: configuration automation allowed an invalid state to propagate globally. Fixes will include stricter change validation, better rollback logic, and improved monitoring.
October 30, 2025 at 2:22 PM
Microsoft paused all new configs at 17:30 UTC, then rolled back to a “last known good” state and slowly rebalanced traffic. Service stabilized around 00:05 UTC on Oct 30.
October 30, 2025 at 2:22 PM
AFD sits in front of Azure App Service, SQL Database, Entra ID, Defender, Sentinel, Copilot, and the Azure Portal—so the blast radius was huge.
October 30, 2025 at 2:22 PM
At 15:45 UTC, nodes began failing to load a tenant config that had passed automation without proper validation. As unhealthy nodes dropped, traffic piled onto the rest, causing timeouts and high latency.
October 30, 2025 at 2:22 PM
Reposted by Doug Madory
Internet traffic began returning to normal in Tanzania beginning at 11:39 UTC today following an internet shutdown lasting nearly 26 hours.
October 30, 2025 at 1:46 PM
Internet traffic began returning to normal in Tanzania beginning at 11:39 UTC today following an internet shutdown lasting nearly 26 hours.
October 30, 2025 at 1:46 PM
Reposted by Doug Madory
Our thoughts are with the affected provider during this difficult time; we can’t imagine what that must be like.

- AWS, probably
October 29, 2025 at 9:00 PM
The image was a joke about the old 'last known good configuration' reboot option on Windows computers.
October 29, 2025 at 9:29 PM