Fred Hebert
@ferd.ca
Staff SRE @ honeycomb.io, Tech Book Author, Resilience in Software Foundation board member, Erlang Ecosystem Foundation co-founder, Resilience Engineering fan. SRE-not-sorry.
blog: https://ferd.ca
notes: https://ferd.ca/notes/
blog: https://ferd.ca
notes: https://ferd.ca/notes/
Pinned
Fred Hebert
@ferd.ca
· Mar 17
People tend to have a mental model where a system is stable until disturbed, far more often than they have one where the system is balanced because it is constantly intervened with.
The latter is a more useful approach to thinking about complex systems.
The latter is a more useful approach to thinking about complex systems.
As someone who tends to know things (or how to access this knowledge) about systems I'm continuously involved with, I had a few people literally tell me the cool thing about AI would be not having to ask me questions anymore.
And I get what they mean but I also hear what they say.
And I get what they mean but I also hear what they say.
November 4, 2025 at 5:11 PM
As someone who tends to know things (or how to access this knowledge) about systems I'm continuously involved with, I had a few people literally tell me the cool thing about AI would be not having to ask me questions anymore.
And I get what they mean but I also hear what they say.
And I get what they mean but I also hear what they say.
I had been struggling with the riff of a song on bass for a short while and that I felt should have been way easier than it was, and after taking a good break and coming back at it with a slightly different approach it’s absolutely straightforward.
This happens outside music too, it’s great.
This happens outside music too, it’s great.
November 3, 2025 at 12:31 AM
I had been struggling with the riff of a song on bass for a short while and that I felt should have been way easier than it was, and after taking a good break and coming back at it with a slightly different approach it’s absolutely straightforward.
This happens outside music too, it’s great.
This happens outside music too, it’s great.
post-incident item: make all changes and effects advertent
October 29, 2025 at 6:21 PM
post-incident item: make all changes and effects advertent
I’ll do some pedant stuff and question the frame: move from software-as-object into software-as-boundary.
What is good software? What’s a good vehicle? What’s a good meal? What’s a good home? What’s a good school? What’s good land?
The answer depends on who/what interacts with it and how.
What is good software? What’s a good vehicle? What’s a good meal? What’s a good home? What’s a good school? What’s good land?
The answer depends on who/what interacts with it and how.
for context: the second edition of "Observability Engineering" starts off swinging with an opinionated chapter called "The Fundamentals of Building Good Software".
which begs the question... what IS ✨ good software ✨?
i'll drop my answers 👇 but curious to hear others.
which begs the question... what IS ✨ good software ✨?
i'll drop my answers 👇 but curious to hear others.
@charity.wtf asked a great question on LinkedIn today. Her question was: "What is ✨ good software ✨?"
I'd love to hear your answers, but I'd also like to share mine. So if you'll allow me a moment of indulgence... 🧵
I'd love to hear your answers, but I'd also like to share mine. So if you'll allow me a moment of indulgence... 🧵
October 27, 2025 at 2:41 PM
I’ll do some pedant stuff and question the frame: move from software-as-object into software-as-boundary.
What is good software? What’s a good vehicle? What’s a good meal? What’s a good home? What’s a good school? What’s good land?
The answer depends on who/what interacts with it and how.
What is good software? What’s a good vehicle? What’s a good meal? What’s a good home? What’s a good school? What’s good land?
The answer depends on who/what interacts with it and how.
Reposted by Fred Hebert
Hey can y'all do me a favor? If you run across any post-incident write-ups from companies affected by the AWS outage, could you send those to me?Feel free to leave a link in the comments, or you can drop them here:
www.thevoid.community/submit-incid...
Thanks!
www.thevoid.community/submit-incid...
Thanks!
Submit an Incident Report
The VOID is a community-contributed collection of software-related incident reports.
www.thevoid.community
October 24, 2025 at 10:50 PM
Hey can y'all do me a favor? If you run across any post-incident write-ups from companies affected by the AWS outage, could you send those to me?Feel free to leave a link in the comments, or you can drop them here:
www.thevoid.community/submit-incid...
Thanks!
www.thevoid.community/submit-incid...
Thanks!
Reposted by Fred Hebert
5. For example, research produced and funded by tech companies often either frames problems as user-driven, or explores solutions as the obligation of users (e.g. community notes). Seldom does it explore consequences of design, UX, or algorithmic implementation, let alone the business model.
October 24, 2025 at 1:01 AM
5. For example, research produced and funded by tech companies often either frames problems as user-driven, or explores solutions as the obligation of users (e.g. community notes). Seldom does it explore consequences of design, UX, or algorithmic implementation, let alone the business model.
Yesterday’s dealing with the AWS outage felt kinda fun in a high-adrenaline sense, and demanded a lot of creativity and improvisation. Today I was just god damn drained though.
Usually fatigue starts on the same day, and one can shitpost through it, but the drive to sarcasm wasn’t even there today.
Usually fatigue starts on the same day, and one can shitpost through it, but the drive to sarcasm wasn’t even there today.
October 22, 2025 at 2:35 AM
Yesterday’s dealing with the AWS outage felt kinda fun in a high-adrenaline sense, and demanded a lot of creativity and improvisation. Today I was just god damn drained though.
Usually fatigue starts on the same day, and one can shitpost through it, but the drive to sarcasm wasn’t even there today.
Usually fatigue starts on the same day, and one can shitpost through it, but the drive to sarcasm wasn’t even there today.
Reposted by Fred Hebert
Learned that an anonymous outside expert on submersibles did an interview with the OceanGate Titan investigation, and they released a transcript, with all the names redacted. The first line of his first answer? "I'm sure you're familiar with my film Titanic."
October 16, 2025 at 9:28 PM
Learned that an anonymous outside expert on submersibles did an interview with the OceanGate Titan investigation, and they released a transcript, with all the names redacted. The first line of his first answer? "I'm sure you're familiar with my film Titanic."
Last year I had grown some decently sized carrots, which felt pretty cool. This year’s harvest is just full on monstrously large eldritch taproots.
They taste great, they’re just comically impractical to store.
They taste great, they’re just comically impractical to store.
October 13, 2025 at 3:54 AM
Last year I had grown some decently sized carrots, which felt pretty cool. This year’s harvest is just full on monstrously large eldritch taproots.
They taste great, they’re just comically impractical to store.
They taste great, they’re just comically impractical to store.
Washing all my synthetic fabric clothes, putting them in the dryer, and eating a spoonful of whatever is in the lint filter.
You gotta do what you gotta do to maintain brain plasticity and adapt to all of them products rolling out workflow changes and dark patterns.
Helps deal with the news too.
You gotta do what you gotta do to maintain brain plasticity and adapt to all of them products rolling out workflow changes and dark patterns.
Helps deal with the news too.
October 7, 2025 at 11:07 PM
Washing all my synthetic fabric clothes, putting them in the dryer, and eating a spoonful of whatever is in the lint filter.
You gotta do what you gotta do to maintain brain plasticity and adapt to all of them products rolling out workflow changes and dark patterns.
Helps deal with the news too.
You gotta do what you gotta do to maintain brain plasticity and adapt to all of them products rolling out workflow changes and dark patterns.
Helps deal with the news too.
Radiology is a specialty that was long promised to be automated via ML, but that still persists today. There are fascinating studies on AI/human joint performance around it.
I appreciate this article, and the distinction between benchmarks settings and the more situated nature of real work.
I appreciate this article, and the distinction between benchmarks settings and the more situated nature of real work.
AI radiology today is powerful, but it consists of many narrow islands of automation that have failed to replace radiologists' time.
This isn't the full picture. Read more in the @worksinprogress.bsky.social piece: worksinprogress.co/issue/the-a...
This isn't the full picture. Read more in the @worksinprogress.bsky.social piece: worksinprogress.co/issue/the-a...
The algorithm will see you now - Works in Progress Magazine
Radiology combines digital images, clear benchmarks, and repeatable tasks. But replacing humans with AI is harder than it seems.
worksinprogress.co
September 27, 2025 at 3:57 AM
Radiology is a specialty that was long promised to be automated via ML, but that still persists today. There are fascinating studies on AI/human joint performance around it.
I appreciate this article, and the distinction between benchmarks settings and the more situated nature of real work.
I appreciate this article, and the distinction between benchmarks settings and the more situated nature of real work.
The rebar3 -> rebar4 kickstarter ends in a few hours!
Late pledges stay open, but rewards (T-shirt, mug, stickers) disappear with the campaign.
We're seeking a more sustainable future for #Erlang build tools, and this campaign helps support this goal: www.kickstarter.com/projects/pee...
Late pledges stay open, but rewards (T-shirt, mug, stickers) disappear with the campaign.
We're seeking a more sustainable future for #Erlang build tools, and this campaign helps support this goal: www.kickstarter.com/projects/pee...
From Rebar3 to Rebar4: Integrating with Erlang/OTP
Building on top of Rebar3 to Fully Integrate with Erlang/OTP for All BEAM Languages, creating Rebar4 the next generation build tool.
www.kickstarter.com
September 24, 2025 at 1:59 PM
The rebar3 -> rebar4 kickstarter ends in a few hours!
Late pledges stay open, but rewards (T-shirt, mug, stickers) disappear with the campaign.
We're seeking a more sustainable future for #Erlang build tools, and this campaign helps support this goal: www.kickstarter.com/projects/pee...
Late pledges stay open, but rewards (T-shirt, mug, stickers) disappear with the campaign.
We're seeking a more sustainable future for #Erlang build tools, and this campaign helps support this goal: www.kickstarter.com/projects/pee...
Reposted by Fred Hebert
I was doing some software history research and stumbled on this absolutely FASCINATING letter from 1964: dl.acm.org/doi/10.1145/...
Some random defense contractor writes in to say "You should deliver a minimal prototype as fast as possible to get feedback and involve users at every stage of labor"
Some random defense contractor writes in to say "You should deliver a minimal prototype as fast as possible to get feedback and involve users at every stage of labor"
Some observations concerning large programming efforts | Proceedings of the April 21-23, 1964, spring joint computer conference
dl.acm.org
September 22, 2025 at 8:40 PM
I was doing some software history research and stumbled on this absolutely FASCINATING letter from 1964: dl.acm.org/doi/10.1145/...
Some random defense contractor writes in to say "You should deliver a minimal prototype as fast as possible to get feedback and involve users at every stage of labor"
Some random defense contractor writes in to say "You should deliver a minimal prototype as fast as possible to get feedback and involve users at every stage of labor"
This analogy was in my head for months and I had to write about it. Treating incidents as "fix it and move on" is a losing proposition. It's more useful to see them as emergent outcomes of all the tradeoffs we make, and to use them as "landmarks" to orient ourselves in a solution space.
Ongoing Tradeoffs, and Incidents as Landmarks
Think of incidents as landmarks when finding your way. The tradeoffs you make can inform the type of incidents you get, and they in turn let you evaluate how you balance priorities and goal conflicts.
ferd.ca
September 20, 2025 at 2:18 PM
This analogy was in my head for months and I had to write about it. Treating incidents as "fix it and move on" is a losing proposition. It's more useful to see them as emergent outcomes of all the tradeoffs we make, and to use them as "landmarks" to orient ourselves in a solution space.
Reposted by Fred Hebert
Software incidents are painful, and we're trying to help change that. If you deal with incidents in your work, please help us help you!
Take the survey here: www.thevoid.community/survey
Take the survey here: www.thevoid.community/survey
VOID Survey
The VOID is a community-contributed collection of software-related incident reports.
www.thevoid.community
September 10, 2025 at 3:38 PM
Software incidents are painful, and we're trying to help change that. If you deal with incidents in your work, please help us help you!
Take the survey here: www.thevoid.community/survey
Take the survey here: www.thevoid.community/survey
Reposted by Fred Hebert
Proud to back the Rebar4 Kickstarter — moving the BEAM ecosystem forward with the community. 🙌
Thanks to the @theerlef.bsky.social (EEF) for a €1,750 contribution to our Rebar4 Kickstarter. It moves us closer to funding work to prepare for OTP integration and cut external deps.
Back: www.kickstarter.com/projects/pee...
Back: www.kickstarter.com/projects/pee...
September 8, 2025 at 7:04 PM
Proud to back the Rebar4 Kickstarter — moving the BEAM ecosystem forward with the community. 🙌
‘helm’ has a levenshtein distance of 1 from both ‘help’ and ‘hell’ and I think that’s on purpose
August 28, 2025 at 11:08 PM
‘helm’ has a levenshtein distance of 1 from both ‘help’ and ‘hell’ and I think that’s on purpose
Fred's rule about posting:
You can either write long, contextually rich texts that are unambiguous but also won't be read much because they're too long, or shorter but engaging texts that are going to be interpreted in ways you disagree with, and there's no middle ground.
You can either write long, contextually rich texts that are unambiguous but also won't be read much because they're too long, or shorter but engaging texts that are going to be interpreted in ways you disagree with, and there's no middle ground.
August 26, 2025 at 12:56 PM
Fred's rule about posting:
You can either write long, contextually rich texts that are unambiguous but also won't be read much because they're too long, or shorter but engaging texts that are going to be interpreted in ways you disagree with, and there's no middle ground.
You can either write long, contextually rich texts that are unambiguous but also won't be read much because they're too long, or shorter but engaging texts that are going to be interpreted in ways you disagree with, and there's no middle ground.
"Behavioural science is unlikely to change the world without a heterogeneity revolution" is this week's paper, arguing that if we stopped chasing main effects and considered diverse context-dependent dynamics as the norm, policy research could get better models.
Notes at: ferd.ca/notes/paper-...
Notes at: ferd.ca/notes/paper-...
(PDF) Behavioural science is unlikely to change the world without a heterogeneity revolution
PDF | In the past decade, behavioural science has gained influence in policymaking but suffered a crisis of confidence in the replicability of its... | Find, read and cite all the research you need on...
www.researchgate.net
August 26, 2025 at 12:25 PM
"Behavioural science is unlikely to change the world without a heterogeneity revolution" is this week's paper, arguing that if we stopped chasing main effects and considered diverse context-dependent dynamics as the norm, policy research could get better models.
Notes at: ferd.ca/notes/paper-...
Notes at: ferd.ca/notes/paper-...
The Leftover principle is a consequence of “automate what you can and what is left over is done by humans”, implying people are left with trivial tasks not worth automating and tasks too rare or complex to automate.
Assuming this can be done without negative impact to leftover tasks is often wrong.
Assuming this can be done without negative impact to leftover tasks is often wrong.
For leaders with expectations about how more productive, efficient, etc. the software engineers in their org will be with AI:
1. How are you handling the "Left-Over Principle" challenges?
2. Also: customer comms about the incidents involving code produced AI?
(Seems clear #2 is depends on #1)
1. How are you handling the "Left-Over Principle" challenges?
2. Also: customer comms about the incidents involving code produced AI?
(Seems clear #2 is depends on #1)
August 21, 2025 at 12:45 PM
The Leftover principle is a consequence of “automate what you can and what is left over is done by humans”, implying people are left with trivial tasks not worth automating and tasks too rare or complex to automate.
Assuming this can be done without negative impact to leftover tasks is often wrong.
Assuming this can be done without negative impact to leftover tasks is often wrong.
Reposted by Fred Hebert
In my second post of key questions for developers, we dig through the insights from multiple @ferd.ca posts to determine how products evolve and the environments that facilitate that process.
programmersstone.blog/posts/what-a...
#ProductDevelopment #PsychologicalSafety
programmersstone.blog/posts/what-a...
#ProductDevelopment #PsychologicalSafety
What are we afraid of?
This article is part of the series JEG2's Questions.
One of the lessons I learned during my time in management is that it was far more important for me to worry about creating environments where good ...
programmersstone.blog
August 12, 2025 at 4:16 PM
In my second post of key questions for developers, we dig through the insights from multiple @ferd.ca posts to determine how products evolve and the environments that facilitate that process.
programmersstone.blog/posts/what-a...
#ProductDevelopment #PsychologicalSafety
programmersstone.blog/posts/what-a...
#ProductDevelopment #PsychologicalSafety
designing truthful bumper stickers
August 5, 2025 at 2:13 PM
designing truthful bumper stickers
I’ve sowed white sweetclover for the last two years in the strip of soil next to my house and they’ve never even managed to sprout there.
Yet these fuckers are growing wild right outside my fence, taunting me.
Yet these fuckers are growing wild right outside my fence, taunting me.
July 30, 2025 at 3:09 AM
I’ve sowed white sweetclover for the last two years in the strip of soil next to my house and they’ve never even managed to sprout there.
Yet these fuckers are growing wild right outside my fence, taunting me.
Yet these fuckers are growing wild right outside my fence, taunting me.
You know videogame bad guys are software folks because it starts with a tiny issue (your hero) & they keep delegating to straightforward minions but as the problem grows they have to churn ever fancier solutions until final stage where they must fight you themselves but by then you’re way too strong
July 12, 2025 at 1:46 PM
You know videogame bad guys are software folks because it starts with a tiny issue (your hero) & they keep delegating to straightforward minions but as the problem grows they have to churn ever fancier solutions until final stage where they must fight you themselves but by then you’re way too strong