Doug Madory
@eldomador.bsky.social
Internet Analysis (BGP/NetFlow) at @kentik.bsky.social (formerly of Dyn Research and Renesys).
https://www.linkedin.com/in/dougmadory/
https://www.linkedin.com/in/dougmadory/
Let's Encrypt impacted by Cloudflare outage.
November 18, 2025 at 2:25 PM
Let's Encrypt impacted by Cloudflare outage.
Twitter/X traffic by source CDN:
November 18, 2025 at 2:02 PM
Twitter/X traffic by source CDN:
Reposted by Doug Madory
According to @kentik.bsky.social aggregate traffic data, @cloudflare.social 's disruptions began at 11:32 UTC.
November 18, 2025 at 1:40 PM
According to @kentik.bsky.social aggregate traffic data, @cloudflare.social 's disruptions began at 11:32 UTC.
According to @kentik.bsky.social aggregate traffic data, @cloudflare.social 's disruptions began at 11:32 UTC.
November 18, 2025 at 1:40 PM
According to @kentik.bsky.social aggregate traffic data, @cloudflare.social 's disruptions began at 11:32 UTC.
Be sure to check out the interactive visualization at the end!
November 14, 2025 at 8:32 PM
Be sure to check out the interactive visualization at the end!
Tanzania Internet is offline again after a ~2 hour recovery.
Follow connectivity in Tanzania in near realtime:
ioda.inetintel.cc.gatech.edu/country/TZ?f...
Follow connectivity in Tanzania in near realtime:
ioda.inetintel.cc.gatech.edu/country/TZ?f...
October 30, 2025 at 5:51 PM
Lesson: at cloud scale, configuration management is reliability engineering. A single unchecked update can topple a continent-wide edge.
#Azure #Cloud #Outage #IncidentResponse #SRE
#Azure #Cloud #Outage #IncidentResponse #SRE
October 30, 2025 at 2:22 PM
Lesson: at cloud scale, configuration management is reliability engineering. A single unchecked update can topple a continent-wide edge.
#Azure #Cloud #Outage #IncidentResponse #SRE
#Azure #Cloud #Outage #IncidentResponse #SRE
Root cause: configuration automation allowed an invalid state to propagate globally. Fixes will include stricter change validation, better rollback logic, and improved monitoring.
October 30, 2025 at 2:22 PM
Root cause: configuration automation allowed an invalid state to propagate globally. Fixes will include stricter change validation, better rollback logic, and improved monitoring.
Microsoft paused all new configs at 17:30 UTC, then rolled back to a “last known good” state and slowly rebalanced traffic. Service stabilized around 00:05 UTC on Oct 30.
October 30, 2025 at 2:22 PM
Microsoft paused all new configs at 17:30 UTC, then rolled back to a “last known good” state and slowly rebalanced traffic. Service stabilized around 00:05 UTC on Oct 30.
AFD sits in front of Azure App Service, SQL Database, Entra ID, Defender, Sentinel, Copilot, and the Azure Portal—so the blast radius was huge.
October 30, 2025 at 2:22 PM
AFD sits in front of Azure App Service, SQL Database, Entra ID, Defender, Sentinel, Copilot, and the Azure Portal—so the blast radius was huge.
At 15:45 UTC, nodes began failing to load a tenant config that had passed automation without proper validation. As unhealthy nodes dropped, traffic piled onto the rest, causing timeouts and high latency.
October 30, 2025 at 2:22 PM
At 15:45 UTC, nodes began failing to load a tenant config that had passed automation without proper validation. As unhealthy nodes dropped, traffic piled onto the rest, causing timeouts and high latency.
Reposted by Doug Madory
Internet traffic began returning to normal in Tanzania beginning at 11:39 UTC today following an internet shutdown lasting nearly 26 hours.
October 30, 2025 at 1:46 PM
Internet traffic began returning to normal in Tanzania beginning at 11:39 UTC today following an internet shutdown lasting nearly 26 hours.
Internet traffic began returning to normal in Tanzania beginning at 11:39 UTC today following an internet shutdown lasting nearly 26 hours.
October 30, 2025 at 1:46 PM
Internet traffic began returning to normal in Tanzania beginning at 11:39 UTC today following an internet shutdown lasting nearly 26 hours.
Reposted by Doug Madory
Our thoughts are with the affected provider during this difficult time; we can’t imagine what that must be like.
- AWS, probably
- AWS, probably
October 29, 2025 at 9:00 PM
Our thoughts are with the affected provider during this difficult time; we can’t imagine what that must be like.
- AWS, probably
- AWS, probably
The image was a joke about the old 'last known good configuration' reboot option on Windows computers.
October 29, 2025 at 9:29 PM
The image was a joke about the old 'last known good configuration' reboot option on Windows computers.