Jason McC. Smith
@jasonmccsmith.bsky.social
Software archaeologist, author, general technology troublemaker and troubleshooter. Standards wonk, research focus on semantic analysis of large systems.
And I can't help but think that sounds remarkably like many of the conversations I have had regarding LLM-backed 'AI', even in professional settings.
Just... no. Stop treating them like magic wish boxes.
Just... no. Stop treating them like magic wish boxes.
October 1, 2025 at 7:07 PM
And I can't help but think that sounds remarkably like many of the conversations I have had regarding LLM-backed 'AI', even in professional settings.
Just... no. Stop treating them like magic wish boxes.
Just... no. Stop treating them like magic wish boxes.
OTOH, this could be another TouchBar, but they've been chasing this ultraminaturization for literally decades.
This smells like they're nearing an internal engineering goal.
Call it a hunch.
This smells like they're nearing an internal engineering goal.
Call it a hunch.
September 10, 2025 at 1:19 AM
OTOH, this could be another TouchBar, but they've been chasing this ultraminaturization for literally decades.
This smells like they're nearing an internal engineering goal.
Call it a hunch.
This smells like they're nearing an internal engineering goal.
Call it a hunch.
Glasses? It's *almost* small enough to fit in a thick earpiece. Battery on opposite side?
Lapel pin? Humane gave a run at this, but it wasn't there.
When you see Apple produce something that just seems... too far, it's often a good bet they're about to make a hard left, after proving the tech.
Lapel pin? Humane gave a run at this, but it wasn't there.
When you see Apple produce something that just seems... too far, it's often a good bet they're about to make a hard left, after proving the tech.
September 10, 2025 at 1:19 AM
Glasses? It's *almost* small enough to fit in a thick earpiece. Battery on opposite side?
Lapel pin? Humane gave a run at this, but it wasn't there.
When you see Apple produce something that just seems... too far, it's often a good bet they're about to make a hard left, after proving the tech.
Lapel pin? Humane gave a run at this, but it wasn't there.
When you see Apple produce something that just seems... too far, it's often a good bet they're about to make a hard left, after proving the tech.
Below that is screen, battery, USB port/speaker/mic assembly, and wireless charger.
And that's it.
They shrunk the guts to that bump.
Now, I'm willing to be wrong, but that bump is a pretty damned powerful little computing device. Give it an alternate display, and not-conjoined battery, and...
And that's it.
They shrunk the guts to that bump.
Now, I'm willing to be wrong, but that bump is a pretty damned powerful little computing device. Give it an alternate display, and not-conjoined battery, and...
September 10, 2025 at 1:19 AM
Below that is screen, battery, USB port/speaker/mic assembly, and wireless charger.
And that's it.
They shrunk the guts to that bump.
Now, I'm willing to be wrong, but that bump is a pretty damned powerful little computing device. Give it an alternate display, and not-conjoined battery, and...
And that's it.
They shrunk the guts to that bump.
Now, I'm willing to be wrong, but that bump is a pretty damned powerful little computing device. Give it an alternate display, and not-conjoined battery, and...
I do so almost every time.
September 3, 2025 at 5:30 PM
I do so almost every time.
Unfortunately, unless we are diligent about reviewing the outputs and making sure we stay within our own abilities and knowledge, this includes making mistakes faster.
Use the tools carefully. Review the outputs. Know their limitations *and* your own.
Otherwise it's all just hallucinations.
Use the tools carefully. Review the outputs. Know their limitations *and* your own.
Otherwise it's all just hallucinations.
July 16, 2025 at 9:12 PM
Unfortunately, unless we are diligent about reviewing the outputs and making sure we stay within our own abilities and knowledge, this includes making mistakes faster.
Use the tools carefully. Review the outputs. Know their limitations *and* your own.
Otherwise it's all just hallucinations.
Use the tools carefully. Review the outputs. Know their limitations *and* your own.
Otherwise it's all just hallucinations.
None of the outputs are real, factual, or correct until we say they are, as measured against our own experiences, expertise, and biases.
At the moment, the best we can hope for is to speed up our own abilities by performing some of the grunt work, to get to results faster.
At the moment, the best we can hope for is to speed up our own abilities by performing some of the grunt work, to get to results faster.
July 16, 2025 at 9:12 PM
None of the outputs are real, factual, or correct until we say they are, as measured against our own experiences, expertise, and biases.
At the moment, the best we can hope for is to speed up our own abilities by performing some of the grunt work, to get to results faster.
At the moment, the best we can hope for is to speed up our own abilities by performing some of the grunt work, to get to results faster.
This is not an unknown problem. I am not saying anything that isn't already being pursued as improvements, but the number of people I run into in professional circles who choose to treat these systems as actual sources of truth 'with occasional mistakes' is rather horrifying.
July 16, 2025 at 9:12 PM
This is not an unknown problem. I am not saying anything that isn't already being pursued as improvements, but the number of people I run into in professional circles who choose to treat these systems as actual sources of truth 'with occasional mistakes' is rather horrifying.
Until there is a fundamental shift in how these systems work, admire the hell out of the advances in the delivery, but recognize that the content is highly suspect.
You are the only source of truth for the output you query for
Your colleague is the only source of truth for their output.
You are the only source of truth for the output you query for
Your colleague is the only source of truth for their output.
July 16, 2025 at 9:12 PM
Until there is a fundamental shift in how these systems work, admire the hell out of the advances in the delivery, but recognize that the content is highly suspect.
You are the only source of truth for the output you query for
Your colleague is the only source of truth for their output.
You are the only source of truth for the output you query for
Your colleague is the only source of truth for their output.
So of course it is tested, right?
When you've produced code that you're not sure how it works, or what the requirements really should have been, exactly how are you going to test it?
Unsurprisingly, "let the 'AI' do it" is a frequent response.
And still, no sufficient review against reality.
When you've produced code that you're not sure how it works, or what the requirements really should have been, exactly how are you going to test it?
Unsurprisingly, "let the 'AI' do it" is a frequent response.
And still, no sufficient review against reality.
July 16, 2025 at 9:12 PM
So of course it is tested, right?
When you've produced code that you're not sure how it works, or what the requirements really should have been, exactly how are you going to test it?
Unsurprisingly, "let the 'AI' do it" is a frequent response.
And still, no sufficient review against reality.
When you've produced code that you're not sure how it works, or what the requirements really should have been, exactly how are you going to test it?
Unsurprisingly, "let the 'AI' do it" is a frequent response.
And still, no sufficient review against reality.
If you're working on a word processor, maybe this is okay. Maybe no one will notice if that span of bold is actually two spans butted up against each other.
But a mission critical system? Flight control? A power plant? A *medical device*?
Blindly incorporating this output is negligent at best.
But a mission critical system? Flight control? A power plant? A *medical device*?
Blindly incorporating this output is negligent at best.
July 16, 2025 at 9:12 PM
If you're working on a word processor, maybe this is okay. Maybe no one will notice if that span of bold is actually two spans butted up against each other.
But a mission critical system? Flight control? A power plant? A *medical device*?
Blindly incorporating this output is negligent at best.
But a mission critical system? Flight control? A power plant? A *medical device*?
Blindly incorporating this output is negligent at best.
And the hallucinations creep into production. The code that at first glance appears valid ("Hey, it *compiles*, what do you want?") ends up being buggy in edge cases that were not considered, or worse, that were considered, stated, but not handled except through 'trust' of the output.
July 16, 2025 at 9:12 PM
And the hallucinations creep into production. The code that at first glance appears valid ("Hey, it *compiles*, what do you want?") ends up being buggy in edge cases that were not considered, or worse, that were considered, stated, but not handled except through 'trust' of the output.
So folks reach past their grasp. They ask the 'AI' to produce code, or models, or documentation, for things that they don't have the expertise to properly review.
They begin to make the mistake of trusting the output when they no longer have the expertise to determine its validity.
They begin to make the mistake of trusting the output when they no longer have the expertise to determine its validity.
July 16, 2025 at 9:12 PM
So folks reach past their grasp. They ask the 'AI' to produce code, or models, or documentation, for things that they don't have the expertise to properly review.
They begin to make the mistake of trusting the output when they no longer have the expertise to determine its validity.
They begin to make the mistake of trusting the output when they no longer have the expertise to determine its validity.
However, boilerplate is boring. Boilerplate is pedestrian. Boilerplate isn't hip, it isn't now, it isn't the *vibe*, man.
It's not disruptive, or innovative, or any of the things that attract a lot of people to coding and app development.
It won't get you promoted, it's not cutting edge.
It's not disruptive, or innovative, or any of the things that attract a lot of people to coding and app development.
It won't get you promoted, it's not cutting edge.
July 16, 2025 at 9:12 PM
However, boilerplate is boring. Boilerplate is pedestrian. Boilerplate isn't hip, it isn't now, it isn't the *vibe*, man.
It's not disruptive, or innovative, or any of the things that attract a lot of people to coding and app development.
It won't get you promoted, it's not cutting edge.
It's not disruptive, or innovative, or any of the things that attract a lot of people to coding and app development.
It won't get you promoted, it's not cutting edge.
For producing boilerplate, they can be useful. "I need to set up a Docker container inside a zero trust environment." "I want a Java function that performs the same work as this Python code snippet." "Write an SQL statement for me to query this schema but if the columns were renamed to..."
July 16, 2025 at 9:12 PM
For producing boilerplate, they can be useful. "I need to set up a Docker container inside a zero trust environment." "I want a Java function that performs the same work as this Python code snippet." "Write an SQL statement for me to query this schema but if the columns were renamed to..."
The more precision you need in the *content* of the answer, however, the more they drift from accurate and correct, and veer off into WTH. The only bulwark against error is the human already knowing enough to review the output for validity and veracity.
Which brings me to coding with these tools.
Which brings me to coding with these tools.
July 16, 2025 at 9:12 PM
The more precision you need in the *content* of the answer, however, the more they drift from accurate and correct, and veer off into WTH. The only bulwark against error is the human already knowing enough to review the output for validity and veracity.
Which brings me to coding with these tools.
Which brings me to coding with these tools.
Instead, it's all too easy for it to become just another form of echo chamber, reinforcing the biases of the user, but now with the 'authority' of 'AI' behind it.
Don't get me wrong, as an UX interface they're fantastic, and if I want them to generate a fictional story, they're fun.
Don't get me wrong, as an UX interface they're fantastic, and if I want them to generate a fictional story, they're fun.
July 16, 2025 at 9:12 PM
Instead, it's all too easy for it to become just another form of echo chamber, reinforcing the biases of the user, but now with the 'authority' of 'AI' behind it.
Don't get me wrong, as an UX interface they're fantastic, and if I want them to generate a fictional story, they're fun.
Don't get me wrong, as an UX interface they're fantastic, and if I want them to generate a fictional story, they're fun.
The human in the loop is the only arbiter of whether the output is a valid or not... which means that the only connection the output has with reality is the pre-existing understanding and biases of the user.
"Yup, that seems right" is not the high bar for 'AI' that is being touted.
"Yup, that seems right" is not the high bar for 'AI' that is being touted.
July 16, 2025 at 9:12 PM
The human in the loop is the only arbiter of whether the output is a valid or not... which means that the only connection the output has with reality is the pre-existing understanding and biases of the user.
"Yup, that seems right" is not the high bar for 'AI' that is being touted.
"Yup, that seems right" is not the high bar for 'AI' that is being touted.
Often they are very good guesses, and the alignment with reality is good, but in every case *it is accidental*.
There is no fact checking, there is no reasoning, there is no behind the scenes "hey does this really make sense". There is just a statistically generated series of words.
There is no fact checking, there is no reasoning, there is no behind the scenes "hey does this really make sense". There is just a statistically generated series of words.
July 16, 2025 at 9:12 PM
Often they are very good guesses, and the alignment with reality is good, but in every case *it is accidental*.
There is no fact checking, there is no reasoning, there is no behind the scenes "hey does this really make sense". There is just a statistically generated series of words.
There is no fact checking, there is no reasoning, there is no behind the scenes "hey does this really make sense". There is just a statistically generated series of words.
The results are *just as probabilistic* as the word choices. There is no distinction made.
The *only* arbiter of whether an output corresponds to reality is the human reader consuming the output.
Think about that for a moment.
Every output is a guess, only determined to be true by the reader.
The *only* arbiter of whether an output corresponds to reality is the human reader consuming the output.
Think about that for a moment.
Every output is a guess, only determined to be true by the reader.
July 16, 2025 at 9:12 PM
The results are *just as probabilistic* as the word choices. There is no distinction made.
The *only* arbiter of whether an output corresponds to reality is the human reader consuming the output.
Think about that for a moment.
Every output is a guess, only determined to be true by the reader.
The *only* arbiter of whether an output corresponds to reality is the human reader consuming the output.
Think about that for a moment.
Every output is a guess, only determined to be true by the reader.
They are *very good* at predicting what the next word in a sequence should be, and as a UX interface are a stunning jump forward for how results are delivered.
The results, however, are not, at the moment in most engines, created from a reliable system of rules, facts, or logic.
The results, however, are not, at the moment in most engines, created from a reliable system of rules, facts, or logic.
July 16, 2025 at 9:12 PM
They are *very good* at predicting what the next word in a sequence should be, and as a UX interface are a stunning jump forward for how results are delivered.
The results, however, are not, at the moment in most engines, created from a reliable system of rules, facts, or logic.
The results, however, are not, at the moment in most engines, created from a reliable system of rules, facts, or logic.