@jpaulreed
@jpaulreed.com
Incident response & management+human factors in software development & operations.
“How many incidents does it take to get to the center of a sociotechnical system?"
(Yes, that @jpaulreed on The Old Bird Site/Mastodon)
“How many incidents does it take to get to the center of a sociotechnical system?"
(Yes, that @jpaulreed on The Old Bird Site/Mastodon)
OH: "Cool, once we have that, then we can run around like heads with our chickens cut off"
November 13, 2025 at 12:22 AM
OH: "Cool, once we have that, then we can run around like heads with our chickens cut off"
Reposted by @jpaulreed
TV conspiracies: we’ve hacked their computers but they’ve perfectly cleaned up the evidence!
Real conspiracies:
To: conspirator
From: famous person
Attachment: evidence.png
Real conspiracies:
To: conspirator
From: famous person
Attachment: evidence.png
November 12, 2025 at 8:19 PM
TV conspiracies: we’ve hacked their computers but they’ve perfectly cleaned up the evidence!
Real conspiracies:
To: conspirator
From: famous person
Attachment: evidence.png
Real conspiracies:
To: conspirator
From: famous person
Attachment: evidence.png
Is this the aurora borealis everyone is talking about?
November 12, 2025 at 8:36 AM
Is this the aurora borealis everyone is talking about?
Reposted by @jpaulreed
So it turns out... the US air travel system was incredibly, deeply dependent on federal funding to just run day-to-day all this time, to the benefit of private airline shareholders, when everyone thinks that state-run trains are leeching off the government. Weird!
November 9, 2025 at 12:10 AM
So it turns out... the US air travel system was incredibly, deeply dependent on federal funding to just run day-to-day all this time, to the benefit of private airline shareholders, when everyone thinks that state-run trains are leeching off the government. Weird!
“Resilience” does NOT mean maintaining service levels in the face of upsets or changing operational dynamics (immaterial of why those dynamics may have changed).
It’s actually exactly the opposite.
It’s actually exactly the opposite.
November 9, 2025 at 3:01 AM
“Resilience” does NOT mean maintaining service levels in the face of upsets or changing operational dynamics (immaterial of why those dynamics may have changed).
It’s actually exactly the opposite.
It’s actually exactly the opposite.
Reposted by @jpaulreed
Tunneling involves moving planes below normal altitude to relieve pressure on the upper altitude controllers and traffic volume. A bunch of flights from NYC to the Carolinas and Virginia are flying at 6-8000' instead of 30000+ to increase flow, though less efficient fuel burn. (2/2)
Make sense?
Make sense?
November 9, 2025 at 1:39 AM
Tunneling involves moving planes below normal altitude to relieve pressure on the upper altitude controllers and traffic volume. A bunch of flights from NYC to the Carolinas and Virginia are flying at 6-8000' instead of 30000+ to increase flow, though less efficient fuel burn. (2/2)
Make sense?
Make sense?
It me!
November 7, 2025 at 1:21 AM
It me!
We're all connecting the (multiple) nationwide listeria outbreaks to the reduced resilience in the food distribution network due the government shutdown and more generally gutting food inspectors, right?
RIGHT??
RIGHT??
November 5, 2025 at 9:33 PM
We're all connecting the (multiple) nationwide listeria outbreaks to the reduced resilience in the food distribution network due the government shutdown and more generally gutting food inspectors, right?
RIGHT??
RIGHT??
Is there such a thing as "corpsplaining?"
Asking for a friend.
Asking for a friend.
November 3, 2025 at 10:53 PM
Is there such a thing as "corpsplaining?"
Asking for a friend.
Asking for a friend.
He gets it.
160 people and it was still DNS.
October 31, 2025 at 11:22 PM
He gets it.
I just facilitated a 160-person incident review AMA!
October 31, 2025 at 6:37 PM
I just facilitated a 160-person incident review AMA!
Reposted by @jpaulreed
I will fight anyone who maintains that spreadsheets are not programming systems. Excel is the most popular functional programming system in the world. Heck, the most popular of any programming system.
October 30, 2025 at 4:24 AM
I will fight anyone who maintains that spreadsheets are not programming systems. Excel is the most popular functional programming system in the world. Heck, the most popular of any programming system.
Reposted by @jpaulreed
I wish engineering culture valued saying "I don't know," reaching out to experts, researching, trusting other teams and distributing execution
80% of the rework I've had to do in my career is people who just assumed they alone could solve anything.
80% of the rework I've had to do in my career is people who just assumed they alone could solve anything.
October 30, 2025 at 3:27 AM
I wish engineering culture valued saying "I don't know," reaching out to experts, researching, trusting other teams and distributing execution
80% of the rework I've had to do in my career is people who just assumed they alone could solve anything.
80% of the rework I've had to do in my career is people who just assumed they alone could solve anything.
First time in small claims court
October 29, 2025 at 5:19 PM
First time in small claims court
What can I say... Internet-scale outages are the "gift" that keep on "giving."
This is becoming way too much of an evergreen skeet
October 28, 2025 at 9:45 PM
What can I say... Internet-scale outages are the "gift" that keep on "giving."
OH: "Why are all my LFI / Safety II Nerd friends STILL utterly destroyed this week?"
October 28, 2025 at 9:38 PM
OH: "Why are all my LFI / Safety II Nerd friends STILL utterly destroyed this week?"
Reposted by @jpaulreed
Tesla's "Full Self-Driving" hallucinates pedestrians and people on TikTok think it's seeing ghosts... it's not an all-too common failure mode in a safety-critical system, it's seasonal marketing!
insideevs.com/news/776769/...
insideevs.com/news/776769/...
‘I Was NOT Expecting This To Work’: Man Drives To Graveyard. Then His Tesla Radar Starts Detecting Spirits
There are several instances of Tesla drivers purportedly seeing ghosts on their environmental visualization screens, particularly when in or near cemeteries.
insideevs.com
October 24, 2025 at 6:34 PM
Tesla's "Full Self-Driving" hallucinates pedestrians and people on TikTok think it's seeing ghosts... it's not an all-too common failure mode in a safety-critical system, it's seasonal marketing!
insideevs.com/news/776769/...
insideevs.com/news/776769/...
OH: "Why are all my industry incident analysis friends utterly destroyed this week?"
October 24, 2025 at 1:43 AM
OH: "Why are all my industry incident analysis friends utterly destroyed this week?"
Can we start sarcastically saying “Thanks Benioff” now?
October 23, 2025 at 3:00 AM
Can we start sarcastically saying “Thanks Benioff” now?
Reposted by @jpaulreed
The number of people who didn’t know what us-east-1 was on Sunday and are suddenly experts in running multi-region distributed systems at scale today is truly remarkable.
October 22, 2025 at 5:59 AM
The number of people who didn’t know what us-east-1 was on Sunday and are suddenly experts in running multi-region distributed systems at scale today is truly remarkable.
Reposted by @jpaulreed
Reposted by @jpaulreed
the AWS datacenter outage coupled with the understanding that Amazon let go of 40% of its DevOps team in favor of an AI solution has me imagining a scenario where the incident response to an event like this is a couple people furiously entering AI prompts to identify and resolve this crash... 1/2
October 20, 2025 at 10:58 PM
the AWS datacenter outage coupled with the understanding that Amazon let go of 40% of its DevOps team in favor of an AI solution has me imagining a scenario where the incident response to an event like this is a couple people furiously entering AI prompts to identify and resolve this crash... 1/2
OH: "Why are all my industry incident-friends utterly destroyed today?"
October 20, 2025 at 11:29 PM
OH: "Why are all my industry incident-friends utterly destroyed today?"
Reposted by @jpaulreed
From a dev at Big Tech: “3 devs across 3 teams are stuck for a month on a build problem that’s caused by connections resetting between two gateways and a messaging queue.
I cannot solve it because I’m blocked access to the other 2 teams’ systems.
I cannot see AI solving this.”
I cannot solve it because I’m blocked access to the other 2 teams’ systems.
I cannot see AI solving this.”
October 17, 2025 at 4:57 PM
From a dev at Big Tech: “3 devs across 3 teams are stuck for a month on a build problem that’s caused by connections resetting between two gateways and a messaging queue.
I cannot solve it because I’m blocked access to the other 2 teams’ systems.
I cannot see AI solving this.”
I cannot solve it because I’m blocked access to the other 2 teams’ systems.
I cannot see AI solving this.”