Yes, that is the nuclear option, and I Imagine will be the try/catch handler at the largest scope, but its rather expensive and you still need to know where to push the eject button

Of course, GPT-5 could come out and render all of this irrelevant, but even so, for any black box ai that has an upper context barrier, a processing graph should allow you to handle much more complexity than just putting that ai in a loop with access to tools. I think.

/end

I think what’s needed is a combination of old style “expert systems” with LLMs at the edges of those decision graphs, solving the last mile problems with ad-hoc context accrued through the workflow. It’s the only way I can see to get reliable results.

All of this tooling and context explosion makes me very skeptical of “general purpose agent” approaches in the near term… you’d spend half your context defining the tools before you even gave them instructions. And IME overloading GPT with options reliably goes off the rails

Getting an LLM system to tackle this problem would mean tooling for building, running, inspecting, debugging, fetching documentation, making a short term decision followed by a cleanup action, github PR, monitoring when the fix is published to remove the monkey patch…

Even spotting it took a few runs, and quite a bit of combined context: how does this browser API work? How does the other code expect plugins to be organized? How does the library I’m using differ from the others that work properly?

How would an llm agent/pipeline be able to handle this sort of problem?

There was no error thrown. It just “didn’t work”

To find it I had to inspect event listener attached to one of my elements and step through someone else’s handler until spotting it.

Fixing this bug either means rearchitecting my code or monkey patching the DOM or submitting a PR to the other library.

All of which are within reach for me (monkey patch to unblock me during dev, sending a PR once i clean it up)

But this is the hard/exciting part…

I ran into an edge case that stumped me for a while. Without boring you with details, it broke functionality due to an implementation detail in one of the libraries that I expect to consume my code. I had to trace control flow outside of my codebase to find it.

As I’ve been working I’ve been keeping track of what I’m doing, and happy to find that for most of the debugging, I at least have a hazy idea of how i could get an LLM pipeline to replicate my process.

Until the other day

Anyways, this week has seen a return to more manual programming as I try to port a library to another framework. I was able to use GPT to do the bulk of it, and it could’ve done more if i had an easier way to declare my pipeline, but a pipeline tool is what I’m building (chicken/egg)