It's also very domain-dependent and sensitive to individual threat models, that we are barely beginning to understand as LLM code escapes application layers and moves upstream to dependencies.

I don't trust obviously vibe-coded dependencies no matter the domain, for now.

Like all tests, writing good ones that balance pragmatism and low maintenance burden with rigor requires SME and domain mastery, no silver bullet.

You're still arguing from trust in correctness improving, not about the fragile OSS ecosystem of accountability being able to shoulder mastery-erosion in ICs-a problem actively exacerbated by more competent LLMs

These companies are pushing the narrative towards trust because they are betting they can deliver more accurate results over time. They are also betting on not being held accountable for their models' output, from training data theft to harm caused. That's a rotten head

I would love to be surprised, but this still only delivers trust. Where does accountability come from? I don't think negative reinforcement during training counts

indeed

it's your domain mastery and codebase intimacy that allows you to vet those safeguards, both skills that agentic coding undermines

your passion for fuzzing and snapshot testing is the only real mitigation we have against LLM-poisoned upstream dependencies that have abdicated accountability, all other software quality control relies on skilled human operators along the chain, and even then

there are no consequences to the model for getting anything wrong, so there is no accountability. there are onlt consequences to the human for shipping software that gets something wrong, and consequences to the humans who receive that software

I think it's easy to attribute FUD as complacency with status quo because of having a foothold in the means of software production, but my attachment to it is rooted in not taking for granted the human dynamics of a distributed ecosystem of trust and accountability that binds us and enables us

blocking irresponsible code authors from having "their" code appear as a resource is not a mechanism available in any software today, irrespective of LLMs

that's why reputation is king, and vibe coders launder the mistrust of LLM models as accountable authors

your point holds merit applied to the strawman that we already build systems we don't understand but does not fairly represent nor address the concerns skeptics actually have

my core critique here is that you are conflating transparency concerns (we don't understand AI as whole systems and AI worries people) with mastery-erosion concerns (we deliver value by shipping system subcomponents we understand and AI worries people) to build a strawman

invested in success, invested in correctness, invested in DX, invested in appreciating edge cases and trade-offs

financial and growth incentives aside, humans are amazing at becoming invested in the problems they solve. AI has no model of self, no stakes, and no real temporal awareness, so can't

the accountability is wherever a human is in the loop. see this 🧵 for one failure mode in OSS: bsky.app/profile/full...

accountability is very different from trust

I think we can only interact with external components rarely *because of* the contract of accountability. To take the trust that enables OSS for granted is a folly.

We'll all be back writing our own auth cryptography if we don't trust that the community standard isn't vibe-coded soon enough.

we abdicate it at component boundaries to maintain the focus to be better stewards of the domains we are accountable for.

if nobody's the master of the component on the other side of a boundary of my project, trust and accountability dissolves and the cellular automata that keeps this working dies

put otherwise, not all invalid states should be illegal