Using git as a database is a seductive idea. You get version history for free. Pull requests give you a review workflow. It’s distributed by design. GitHub will host it for free. Everyone already knows how to use it. Package managers keep falling for this. And it keeps not working out. ## Cargo The crates.io index started as a git repository. Every Cargo client cloned it. This worked fine when the registry was small, but the index kept growing. Users would see progress bars like “Resolving deltas: 74.01%, (64415/95919)” hanging for ages, the visible symptom of Cargo’s libgit2 library grinding through delta resolution on a repository with thousands of historic commits. The problem was worst in CI. Stateless environments would download the full index, use a tiny fraction of it, and throw it away. Every build, every time. RFC 2789 introduced a sparse HTTP protocol. Instead of cloning the whole index, Cargo now fetches files directly over HTTPS, downloading only the metadata for dependencies your project actually uses. (This is the “full index replication vs on-demand queries” tradeoff in action.) By April 2025, 99% of crates.io requests came from Cargo versions where sparse is the default. The git index still exists, still growing by thousands of commits per day, but most users never touch it. ## Homebrew GitHub explicitly asked Homebrew to stop using shallow clones. Updating them was “an extremely expensive operation” due to the tree layout and traffic of homebrew-core and homebrew-cask. Users were downloading 331MB just to unshallow homebrew-core. The .git folder approached 1GB on some machines. Every `brew update` meant waiting for git to grind through delta resolution. Homebrew 4.0.0 in February 2023 switched to JSON downloads for tap updates. The reasoning was blunt: “they are expensive to git fetch and git clone and GitHub would rather we didn’t do that… they are slow to git fetch and git clone and this provides a bad experience to end users.” Auto-updates now run every 24 hours instead of every 5 minutes, and they’re much faster because there’s no git fetch involved. ## CocoaPods CocoaPods is the package manager for iOS and macOS development. It hit the limits hard. The Specs repo grew to hundreds of thousands of podspecs across a deeply nested directory structure. Cloning took minutes. Updating took minutes. CI time vanished into git operations. GitHub imposed CPU rate limits. The culprit was shallow clones, which force GitHub’s servers to compute which objects the client already has. The team tried various band-aids: stopping auto-fetch on `pod install`, converting shallow clones to full clones, sharding the repository. The CocoaPods blog captured it well: “Git was invented at a time when ‘slow network’ and ‘no backups’ were legitimate design concerns. Running endless builds as part of continuous integration wasn’t commonplace.” CocoaPods 1.8 gave up on git entirely for most users. A CDN became the default, serving podspec files directly over HTTP. The migration saved users about a gigabyte of disk space and made `pod install` nearly instant for new setups. ## Go modules Grab’s engineering team went from 18 minutes for `go get` to 12 seconds after deploying a module proxy. That’s not a typo. Eighteen minutes down to twelve seconds. The problem was that `go get` needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Cloning entire repositories to get a single file. Go had security concerns too. The original design wanted to remove version control tools entirely because “these fragment the ecosystem: packages developed using Bazaar or Fossil, for example, are effectively unavailable to users who cannot or choose not to install these tools.” Beyond fragmentation, the Go team worried about security bugs in version control systems becoming security bugs in `go get`. You’re not just importing code; you’re importing the attack surface of every VCS tool on the developer’s machine. GOPROXY became the default in Go 1.13. The proxy serves source archives and go.mod files independently over HTTP. Go also introduced a checksum database (sumdb) that records cryptographic hashes of module contents. This protects against force pushes silently changing tagged releases, and ensures modules remain available even if the original repository is deleted. ## Beyond package managers The same pattern shows up wherever developers try to use git as a database. Git-based wikis like Gollum (used by GitHub and GitLab) become “somewhat too slow to be usable” at scale. Browsing directory structure takes seconds per click. Loading pages takes longer. GitLab plans to move away from Gollum entirely. Git-based CMS platforms like Decap hit GitHub’s API rate limits. A Decap project on GitHub scales to about 10,000 entries if you have a lot of collection relations. A new user with an empty cache makes a request per entry to populate it, burning through the 5,000 request limit quickly. If your site has lots of content or updates frequently, use a database instead. Even GitOps tools that embrace git as a source of truth have to work around its limitations. ArgoCD’s repo server can run out of disk space cloning repositories. A single commit invalidates the cache for all applications in that repo. Large monorepos need special scaling considerations. ## The pattern The hosting problems are symptoms. The underlying issue is that git inherits filesystem limitations, and filesystems make terrible databases. **Directory limits.** Directories with too many files become slow. CocoaPods had 16,000 pod directories in a single Specs folder, requiring huge tree objects and expensive computation. Their fix was hash-based sharding: split directories by the first few characters of a hashed name, so no single directory has too many entries. Git itself does this internally with its objects folder, splitting into 256 subdirectories. You’re reinventing B-trees, badly. **Case sensitivity.** Git is case-sensitive, but macOS and Windows filesystems typically aren’t. Check out a repo containing both `File.txt` and `file.txt` on Windows, and the second overwrites the first. Azure DevOps had to add server-side enforcement to block pushes with case-conflicting paths. **Path length limits.** Windows restricts paths to 260 characters, a constraint dating back to DOS. Git supports longer paths, but Git for Windows inherits the OS limitation. This is painful with deeply nested node_modules directories, where `git status` fails with “Filename too long” errors. **Missing database features.** Databases have CHECK constraints and UNIQUE constraints; git has nothing, so every package manager builds its own validation layer. Databases have locking; git doesn’t. Databases have indexes for queries like “all packages depending on X”; with git you either traverse every file or build your own index. Databases have migrations for schema changes; git has “rewrite history and force everyone to re-clone.” The progression is predictable. Start with a flat directory of files. Hit filesystem limits. Implement sharding. Hit cross-platform issues. Build server-side enforcement. Build custom indexes. Eventually give up and use HTTP or an actual database. You’ve built a worse version of what databases already provide, spread across git hooks, CI pipelines, and bespoke tooling. None of this means git is bad. Git excels at what it was designed for: distributed collaboration on source code, with branching, merging, and offline work. The problem is using it for something else entirely. Package registries need fast point queries for metadata. Git gives you a full-document sync protocol when you need a key-value lookup. If you’re building a package manager and git-as-index seems appealing, look at Cargo, Homebrew, CocoaPods, Go. They all had to build workarounds as they grew, causing pain for users and maintainers. The pull request workflow is nice. The version history is nice. You will hit the same walls they did.

Comments

After putting together ecosyste-ms/package-manager-resolvers, I started wondering what dependency resolution algorithm GitHub Actions uses. When you write `uses: actions/checkout@v4` in a workflow file, you’re declaring a dependency. GitHub resolves it, downloads it, and executes it. That’s package management. So I went spelunking into the runner codebase to see how it works. What I found was concerning. Package managers are a critical part of software supply chain security. The industry has spent years hardening them after incidents like left-pad, event-stream, and countless others. Lockfiles, integrity hashes, and dependency visibility aren’t optional extras. They’re the baseline. GitHub Actions ignores all of it. Compared to mature package ecosystems: Feature | npm | Cargo | NuGet | Bundler | Go | Actions ---|---|---|---|---|---|--- Lockfile | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ Transitive pinning | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ Integrity hashes | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ Dependency tree visibility | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ Resolution specification | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ The core problem is the lack of a lockfile. Every other package manager figured this out decades ago: you declare loose constraints in a manifest, the resolver picks specific versions, and the lockfile records exactly what was chosen. GitHub Actions has no equivalent. Every run re-resolves from your workflow file, and the results can change without any modification to your code. Research from USENIX Security 2022 analyzed over 200,000 repositories and found that 99.7% execute externally developed Actions, 97% use Actions from unverified creators, and 18% run Actions with missing security updates. The researchers identified four fundamental security properties that CI/CD systems need: admittance control, execution control, code control, and access to secrets. GitHub Actions fails to provide adequate tooling for any of them. A follow-up study using static taint analysis found code injection vulnerabilities in over 4,300 workflows across 2.7 million analyzed. Nearly every GitHub Actions user is running third-party code with no verification, no lockfile, and no visibility into what that code depends on. **Mutable versions.** When you pin to `actions/checkout@v4`, that tag can move. The maintainer can push a new commit and retag. Your workflow changes silently. A lockfile would record the SHA that `@v4` resolved to, giving you reproducibility while keeping version tags readable. Instead, you have to choose: readable tags with no stability, or unreadable SHAs with no automated update path. GitHub has added mitigations. Immutable releases lock a release’s git tag after publication. Organizations can enforce SHA pinning as a policy. You can limit workflows to actions from verified creators. These help, but they only address the top-level dependency. They do nothing for transitive dependencies, which is the primary attack vector. **Invisible transitive dependencies.** SHA pinning doesn’t solve this. Composite actions resolve their own dependencies, but you can’t see or control what they pull in. When you pin an action to a SHA, you only lock the outer file. If it internally pulls `some-helper@v1` with a mutable tag, your workflow is still vulnerable. You have zero visibility into this. A lockfile would record the entire resolved tree, making transitive dependencies visible and pinnable. Research on JavaScript Actions found that 54% contain at least one security weakness, with most vulnerabilities coming from indirect dependencies. The tj-actions/changed-files incident showed how this plays out in practice: a compromised action updated its transitive dependencies to exfiltrate secrets. With a lockfile, the unexpected transitive change would have been visible in a diff. **No integrity verification.** npm records `integrity` hashes in the lockfile. Cargo records checksums in `Cargo.lock`. When you install, the package manager verifies the download matches what was recorded. Actions has nothing. You trust GitHub to give you the right code for a SHA. A lockfile with integrity hashes would let you verify that what you’re running matches what you resolved. **Re-runs aren’t reproducible.** GitHub staff have confirmed this explicitly: “if the workflow uses some actions at a version, if that version was force pushed/updated, we will be fetching the latest version there.” A failed job re-run can silently get different code than the original run. Cache interaction makes it worse: caches only save on successful jobs, so a re-run after a force-push gets different code _and_ has to rebuild the cache. Two sources of non-determinism compounding. A lockfile would make re-runs deterministic: same lockfile, same code, every time. **No dependency tree visibility.** npm has `npm ls`. Cargo has `cargo tree`. You can inspect your full dependency graph, find duplicates, trace how a transitive dependency got pulled in. Actions gives you nothing. You can’t see what your workflow actually depends on without manually reading every composite action’s source. A lockfile would be a complete manifest of your dependency tree. **Undocumented resolution semantics.** Every package manager documents how dependency resolution works. npm has a spec. Cargo has a spec. Actions resolution is undocumented. The runner source is public, and the entire “resolution algorithm” is in ActionManager.cs. Here’s a simplified version of what it does: // Simplified from actions/runner ActionManager.cs async Task PrepareActionsAsync(steps) { // Start fresh every time - no caching DeleteDirectory("_work/_actions"); await PrepareActionsRecursiveAsync(steps, depth: 0); } async Task PrepareActionsRecursiveAsync(actions, depth) { if (depth > 10) throw new Exception("Composite action depth exceeded max depth 10"); foreach (var action in actions) { // Resolution happens on GitHub's server - opaque to us var downloadInfo = await GetDownloadInfoFromGitHub(action.Reference); // Download and extract - no integrity verification var tarball = await Download(downloadInfo.TarballUrl); Extract(tarball, $"_actions/{action.Owner}/{action.Repo}/{downloadInfo.Sha}"); // If composite, recurse into its dependencies var actionYml = Parse($"_actions/{action.Owner}/{action.Repo}/{downloadInfo.Sha}/action.yml"); if (actionYml.Type == "composite") { // These nested actions may use mutable tags - we have no control await PrepareActionsRecursiveAsync(actionYml.Steps, depth + 1); } } } That’s it. No version constraints, no deduplication (the same action referenced twice gets downloaded twice), no integrity checks. The tarball URL comes from GitHub’s API, and you trust them to return the right content for the SHA. A lockfile wouldn’t fix the missing spec, but it would at least give you a concrete record of what resolution produced. Even setting lockfiles aside, Actions has other issues that proper package managers solved long ago. **No registry.** Actions live in git repositories. There’s no central index, no security scanning, no malware detection, no typosquatting prevention. A real registry can flag malicious packages, store immutable copies independent of the source, and provide a single point for security response. The Marketplace exists but it’s a thin layer over repository search. Without a registry, there’s nowhere for immutable metadata to live. If an action’s source repository disappears or gets compromised, there’s no fallback. **Shared mutable environment.** Actions aren’t sandboxed from each other. Two actions calling `setup-node` with different versions mutate the same `$PATH`. The outcome depends on execution order, not any deterministic resolution. **No offline support.** Actions are pulled from GitHub on every run. There’s no offline installation mode, no vendoring mechanism, no way to run without network access. Other package managers let you vendor dependencies or set up private mirrors. With Actions, if GitHub is down, your CI is down. **The namespace is GitHub usernames.** Anyone who creates a GitHub account owns that namespace for actions. Account takeovers and typosquatting are possible. When a popular action maintainer’s account gets compromised, attackers can push malicious code and retag. A lockfile with integrity hashes wouldn’t prevent account takeovers, but it would detect when the code changes unexpectedly. The hash mismatch would fail the build instead of silently running attacker-controlled code. Another option would be something like Go’s checksum database, a transparent log of known-good hashes that catches when the same version suddenly has different contents. ### How Did We Get Here? The Actions runner is forked from Azure DevOps, designed for enterprises with controlled internal task libraries where you trust your pipeline tasks. GitHub bolted a public marketplace onto that foundation without rethinking the trust model. The addition of composite actions and reusable workflows created a dependency system, but the implementation ignored lessons from package management: lockfiles, integrity verification, transitive pinning, dependency visibility. This matters beyond CI/CD. Trusted publishing is being rolled out across package registries: PyPI, npm, RubyGems, and others now let you publish packages directly from GitHub Actions using OIDC tokens instead of long-lived secrets. OIDC removes one class of attacks (stolen credentials) but amplifies another: the supply chain security of these registries now depends entirely on GitHub Actions, a system that lacks the lockfile and integrity controls these registries themselves require. A compromise in your workflow’s action dependencies can lead to malicious packages on registries with better security practices than the system they’re trusting to publish. Other CI systems have done better. GitLab CI added an `integrity` keyword in version 17.9 that lets you specify a SHA256 hash for remote includes. If the hash doesn’t match, the pipeline fails. Their documentation explicitly warns that including remote configs “is similar to pulling a third-party dependency” and recommends pinning to full commit SHAs. GitLab recognized the problem and shipped integrity verification. GitHub closed the feature request. GitHub’s design choices don’t just affect GitHub users. Forgejo Actions maintains compatibility with GitHub Actions, which means projects migrating to Codeberg for ethical reasons inherit the same broken CI architecture. The Forgejo maintainers openly acknowledge the problems, with contributors calling GitHub Actions’ ecosystem “terribly designed and executed.” But they’re stuck maintaining compatibility with it. Codeberg mirrors common actions to reduce GitHub dependency, but the fundamental issues are baked into the model itself. GitHub’s design flaws are spreading to the alternatives. GitHub issue #2195 requested lockfile support. It was closed as “not planned” in 2022. Palo Alto’s “Unpinnable Actions” research documented how even SHA-pinned actions can have unpinnable transitive dependencies. Dependabot can update action versions, which helps. Some teams vendor actions into their own repos. zizmor is excellent at scanning workflows and finding security issues. But these are workarounds for a system that lacks the basics. The fix is a lockfile. Record resolved SHAs for every action reference, including transitives. Add integrity hashes. Make the dependency tree inspectable. GitHub closed the request three years ago and hasn’t revisited it. * * * **Further reading:** * Characterizing the Security of GitHub CI Workflows - Koishybayev et al., USENIX Security 2022 * ARGUS: A Framework for Staged Static Taint Analysis of GitHub Workflows and Actions - Muralee et al., USENIX Security 2023 * New GitHub Action supply chain attack: reviewdog/action-setup - Wiz Research, 2025 * Unpinnable Actions: How Malicious Code Can Sneak into Your GitHub Actions Workflows * GitHub Actions Worm: Compromising GitHub Repositories Through the Actions Dependency Tree * setup-python: Action can be compromised via mutable dependency

Git Source Code Mirror - This is a publish-only repository but pull requests can be turned into patches to the mailing list via GitGitGadget (https://gitgitgadget.github.io/). Please follow Documen...

(See how I cleverly did not mention AI in the title!) You know we have seen more than our fair share of slop reports sent to the curl project so it seems only fair that I also write something about the state of AI when we get to enjoy some positive aspects of this technology. Let’s try doing this in a chronological order. ## The magnitude of things curl is almost 180,000 lines of C89 code, excluding blank lines. About 637,000 words in C and H files. To compare, the original novel War and Peace (a _thick_ book) consisted of 587,000 words. The first ideas and traces for curl originated in the httpget project, started in late 1996. Meaning that there is a lot of history and legacy here. curl does network transfers for 28 URL schemes, it has run on over 100 operating systems and on almost 30 CPU architectures. It builds with a wide selection of optional third party libraries. We have shipped over 270 curl releases for which we have documented a total of over 12,500 bugfixes. More than 1,400 humans have contributed with commits merged into the repository, over 3,500 humans are thanked for having helped out. It is a very actively developed project. ## It started with sleep On August 11, 2025 there was a curl vulnerability reported against curl that would turn out legitimate and it would later be published as CVE-2025-9086. The reporter of this was the Google Big Sleep team. A team that claims they use “an AI agent developed by Google DeepMind and Google Project Zero, that actively searches and finds unknown security vulnerabilities in software”. This was the first ever report we have received that seems to have used AI to accurately spot and report a security problem in curl. Of course, we don’t know how much AI and how much human that were involved in the research and the report. The entire reporting process felt very human. ## krb5-ftp In mid September 2025 we got new a security vulnerability reported against curl from a security researcher we had not been in contact with before. The report which accurately identified a problem, was not turned into a CVE only because of sheer luck: the code didn’t work for other reasons so the vulnerability couldn’t actually be reached. As a direct result of this lesson, we ripped out support for krb5-ftp. ## ZeroPath The reporter of the krb5-ftp problem is called Joshua Rogers. He contacted us and graciously forwarded us a huge list of more potential issues that he had extracted. As I understand it, mostly done with the help of ZeroPath. A code analyzer with AI powers. In the curl project we continuously run compilers with maximum pickiness enabled and we though scan-build, clang-tidy, CodeSonar, Coverity, CodeQL and OSS-Fuzz at it and we always address and fix every warning and complaint they report so it was a little surprising that this tool now suddenly could produce over _two hundred_ new potential problems. But it sure did. And it was only the beginning. ## At three there is a pattern As we started to plow through the huge list of issues from Joshua, we received yet another security report against curl. This time by Stanislav Fort from Aisle (using their own AI powered tooling and pipeline for code analysis). Getting security reports is not uncommon for us, we tend to get 2 -3 every week, but on September 23 we got another one we could confirm was a real vulnerability. Again, an AI powered analysis tool had been used. (At the time I write this blog entry, this particular issue has not been disclosed yet so I can’t link it.) ## A shift in the wind As I was amazed by the quality and insights in some of the issues in Joshua’s initial list he sent over I tooted about it on Mastodon, which later was picked up by Hacker news, The Register, Elektroniktidningen and more. These new reported issues feel quite similar in nature to defects reported by code analyzers typically do: small mistakes, omissions, flaws, bugs. Most of them are just plain variable mixups, return code confusions, small memory leaks in weird situations, state transition mistakes and variable type conversions possibly leading to problems etc. Remarkably few of them complete false positives. The quality of the reports make it feel like a new generation of issue identification. Like in this ladder of tool evolution from the old days. Each new step has taken the notch up a level: 1. At some point I think starting in the early 2000s, the C compilers got better at actually warning and detecting many mistakes they just silently allowed back in the dark ages 2. Then the code analyzers took us from there to the next level and found more mistakes in the code. 3. We added fuzzing to the mix in the mid 2010s and found a whole slew of problems we never realized before we had. 4. Now this new breed, almost like a new category, of analyzers that seem to connect the dots better and see patterns previous tools and analyzers have not been able to. And tell us about the discrepancies. ## 25% something Out of that initial list, we merged about 50 separately identifiable bugfixes. The rest were some false positives but also lots of minor issues that we just didn’t think were worth poking at or we didn’t quite agree with. ## A minor tsunami We (primarily Stefan Eissing and myself) worked hard to get through that initial list from Joshua within only a couple of days. A list we mistakenly thought was “it”. Joshua then spiced things up for us by immediately delivering a _second_ list with 47 additional issues. Follow by a third list with yet another 158 additional potential problems. At the same time Stanislav did the similar thing and delivered to us two lists with a total of around twenty possible issues. Don’t take me wrong. This is good. The issues are of high quality and even the ones we dismiss often have some insights and the rate of obvious false positive has remained low and quite manageable. Every bug we find and fix makes curl better. Every fix improves a software that impacts and empowers a huge portion of the world. The total amount of suspected issues submitted by these two gentlemen are now at over _four hundred_. A fair pile of work for us curl maintainers! Because how these reported issues _might_ include security sensitive problems, we have decided to not publish them but limit access to the reporters and the curl security team. As I write this, we are still working our way through these reports but it feels reasonable to assume that we will get even more soon… ## All code An obvious and powerful benefit this tool seems to have compared to others is that it scans _all_ source code without having a build. That means it can detect problems in all backends used in all build combinations. _Old style_ code analyzers require a proper build to analyze and since you can build curl in countless combinations with a myriad of backend setups (where several are architecture or OS specific), it is literally impossible to have all code analyzed with such tools. Also, these tools can inject (parts of) third party libraries as well and find issues in the borderland between curl and its dependencies. I think this is one primary reason it found so many issues: it checked lots of code barely any other analyzers have investigated. ## A few examples To illustrate the level of “smartness” in this tool, allow me to show a few examples that I think shows it off. These are issues reported against curl in the last few weeks and they have all been fixed. Beware that you might have to understand a thing or two about what curl does to properly follow here. ### A function header comment was wrong It correctly spotted that the documentation in the function header incorrectly said an argument is optional when in reality it isn’t. The fix was to correct the comment. # `Curl_resolv`: NULL out-parameter dereference of `*entry` * **Evidence:** `lib/hostip.c`. API promise: "returns a pointer to the entry in the `entry` argument (**if one is provided**)." However, code contains unconditional writes: `*entry = dns;` or `*entry = NULL;`. * **Rationale:** The API allows `entry == NULL`, but the implementation dereferences it on every exit path, causing an immediate crash if a caller passes `NULL`. I could add that the fact that it takes comments so seriously can also trick it to report wrong things when the comments are outdated and state bad “facts”. Which of course shouldn’t happen because comments should not lie! ### code breaks the telnet protocol It figured out that a piece of telnet code actually wouldn’t comply with the telnet protocol and pointed it out. Quite impressively I might add. **Telnet subnegotiation writes unescaped user-controlled values (tn->subopt_ttype, tn->subopt_xdisploc, tn->telnet_vars) into temp (lines 948–989) without escaping IAC (0xFF)** In lib/telnet.c (lines 948–989) the code formats Telnet subnegotiation payloads into temp using msnprintf and inserts the user-controllable values tn->subopt_ttype (lines 948–951), tn->subopt_xdisploc (lines 960–963), and v->data from tn->telnet_vars (lines 976–989) directly into the suboption data. The buffer temp is then written to the socket with swrite (lines 951, 963, 995) without duplicating CURL_IAC (0xFF) bytes. Telnet requires any IAC byte inside subnegotiation data to be escaped by doubling; because these values are not escaped, an 0xFF byte in any of them will be interpreted as an IAC command and can break the subnegotiation stream and cause protocol errors or malfunction. ### no TFTP address pinning Another case where it seems to know the best-practice for a TFTP implementation (pinning the used IP address for the duration of the transfer) and it detected that curl didn’t apply this best-practice in code so it correctly complained: No TFTP peer/TID validation The TFTP receive handler updates state->remote_addr from recvfrom() on every datagram and does not validate that incoming packets come from the previously established server address/port (transfer ID). As a result, any host able to send UDP packets to the client (e.g., on-path attacker or local network adversary) can inject a DATA/OACK/ERROR packet with the expected next block number. The client will accept the payload (Curl_client_write), ACK it, and switch subsequent communication to the attacker’s address, allowing content injection or session hijack. Correct TFTP behavior is to bind to the first server TID and ignore, or error out on, packets from other TIDs. ### memory leaks no one else reported Most memory leaks are reported when someone runs code and notices that not everything is freed in some specific circumstance. We of course test for leaks all the time in tests, but in order to see them in a test we need to run that exact case and there are many code paths that are hard to travel in tests. Apart from doing tests you can of course find leaks by manually reviewing code, but history and experience tell us that is an error-prone method. # GSSAPI security message: leaked `output_token` on invalid token length * **Evidence:** `lib/vauth/krb5_gssapi.c:205--207`. Short quote: ```c if(output_token.length != 4) { ... return CURLE_BAD_CONTENT_ENCODING; } ``` The `gss_release_buffer(&unused_status, &output_token);` call occurs later at line 215, so this early return leaks the buffer from `gss_unwrap`. * **Rationale:** Reachable with a malicious peer sending a not-4-byte security message; repeated handshakes can cause unbounded heap growth (DoS). This particular bug looks straight forward and in hindsight easy enough to spot, but it has existed like this in plain sight in code for _over a decade_. ## More evolution than revolution I think I maybe shocked some people when I stated that the AI tooling helped us find 22, 70 and then a 100 bugs etc. I suspect people in general are not aware of and does not think about what kind of bugfix frequency we work on in this project. _Fixing several hundred bugs per release is a normal rate for us._ Sure, this cycle we will probably reach a new record, but I still don’t grasp for breath because of this. I don’t consider this new tooling a _revolution_. It does not massively or drastically change code or how we approach development. It is however an excellent new project assistant. A powerful tool that highlights code areas that need more attention. A much appreciated evolutionary step. I might of course be speaking too early. Perhaps it will develop a lot more and it can then turn into a revolution. ## Ethical and moral decisions The AI engines burn the forests and they are built by ingesting other people’s code and work. Is it morally and ethically right to use AI for improving Open Source in this way? It is a question to wrestle with and I’m sure the discussion will go on. At least this use of AI does not generate duplicates of someone else’s code for us to use, but it certainly takes lessons from and find patterns based on others’ code. But so do we all, I hope. ## Starting from a decent state I can imagine that curl is a pretty good source code to use a tool of this caliber on, as curl is old, mature and all the minor nits and defect have been polished away. It is a project where we have a high bar and we want to raise it even higher. We love the opportunity to get additional help and figure out where we might have slipped. Then fix those and try again. Over and over until the end of time. ## AIxCC At the DEF CON 33 conference which took place in August 2025, DARPA ran a competition called the AI Cyber Challenge or AIxCC for short. In this contest, the competing teams used AI tools to find artificially injected vulnerabilities in projects – with zero human intervention. One of the projects used in the finals that the teams looked for problems in, was… curl! I have been promised a report or a list of findings from that exercise, as presumably the teams found something more than just the fake inserted problems. I will report back when that happens. ## Going forward We do not yet have any AI powered code analyzer in our CI setup, but I am looking forward to adding such. Maybe several. We _can_ ask GitHub copilot for pull-request reviews but from the little I’ve tried copilot for reviews it is far from comparable to the reports I have received from Joshua and Stanislav, and quite frankly it has been mostly underwhelming. We do not use it. Of course, that can change and it might turn into a powerful tool one day. We now have an established constructive communication setup with both these reporters, which should enable a solid foundation for us to improve curl even more going forward. I personally still do not use any AI at all during development – apart from occasional small experiments. Partly because they all seem to force me into using VS code and I totally lose all my productivity with that. Partly because I’ve not found it very productive in my experiments. Interestingly, this productive AI development happens pretty much concurrently with the AI slop avalanche we also see, proving that one AI is not necessarily like the other AI.