The Conversation After the Fork
Last week I published The Nuclear Code Fork, documenting how WitFoo killed all 450 external dependencies in our analytics platform and brought every line of code in house. The response was significant and the conversations that followed were even more valuable than the post itself.
A recurring pattern emerged. Smart, experienced engineers pushed back with reasonable-sounding objections that I've heard (and considered) many times. These objections deserve serious engagement, because they represent the conventional wisdom that most of our industry still operates on. And they're wrong. Not wrong because the people making them are uninformed, but wrong because the threat landscape has shifted underneath the assumptions those objections rely on.
This post is an attempt to address those objections head on, with specifics.
"Just Use Mature Projects with Good Processes"
This was the most common response. Use well-established open source projects. Look for clean governance, strong code review, mature release cycles. The implication is that supply chain risk is primarily a quality problem, and quality organizations produce safe code.
I wish that were true.
Trivy is maintained by Aqua Security, a well-funded cybersecurity company with dedicated engineering resources. Their release process involves rigorous code review and automated scanning. They are, by any reasonable measure, a mature project with excellent processes. On March 19, 2026, TeamPCP compromised their GitHub Actions by force-pushing 75 of 76 version tags to malicious commits. The attack didn't exploit sloppy process. It exploited the mechanisms of process: CI/CD pipelines, automated release workflows, and the trust relationships between tools.
But if you want the definitive example, look at Log4j.
Log4j is an Apache Software Foundation project. Apache has one of the most rigorous release processes in open source. Their releases involve staging artifacts, signing JARs, posting code with checksums to a mailing list, and conducting a three-day community vote. The CSRB's post-mortem specifically noted that Apache had "a well-established software development lifecycle with clear roles for vetting, testing, and approval of new code."
The Log4Shell vulnerability (CVE-2021-44228, CVSS 10.0, the highest possible score) sat in Log4j for eight years. From 2013 to 2021. Through countless releases. Through all of Apache's rigorous governance. Through all the code review, all the community oversight, all the signed JARs and mailing list votes. Eight years of a remote code execution vulnerability in one of the most widely deployed Java libraries on earth.
CISA's director called it "one of the most serious I've seen in my entire career, if not the most serious." It affected hundreds of millions of devices. And every process designed to catch it failed to do so.
Process maturity doesn't protect you from a risk that your processes weren't designed to detect. It gives you the appearance of safety, which may actually be worse than having no process at all, because it discourages the kind of skepticism that might lead you to look harder.
"Just Wait 20 Days Before Using a Package"
The logic here is reasonable on its face: let other people find the problems first, then consume the tested version. Don't be a first mover. Wait for the dust to settle.
Log4j settled for eight years.
The vulnerability was introduced in version 2.0-beta9 in 2013 and wasn't discovered until late 2021. Waiting 20 days would have accomplished nothing. Waiting 20 months would have accomplished nothing. The vulnerability survived every audit cycle, every release review, and every production deployment for nearly a decade.
And here's the part that should keep you up at night: with AI-assisted attack tooling, what used to take years to come to a head is now taking hours. The TeamPCP campaign moved from Trivy compromise (March 19) to LiteLLM exploitation (March 24) in five days. The LiteLLM malicious packages were live for three hours before PyPI quarantined them. By the time your 20-day waiting period starts, the damage may have been done, the evidence buried, and the next target already compromised.
I wrote about this dynamic in People > Machines (The OODA Loop Strikes Back). The attackers' decision cycle is accelerating faster than the defenders' observation cycle. A temporal buffer designed for the pace of 2020 is not calibrated for the pace of 2026.
"Repositories Are Immutable"
This is the one that comes up the most, and it's the one most dangerously wrong.
The claim is that package registries enforce immutability. Once version 1.2.3 is published, it cannot be changed. Therefore, pinning to a specific version guarantees you're running the same code every time.
Let's walk through what actually happened.
In the Trivy attack, the attackers didn't create new versions. They force-pushed 75 existing version tags to point to entirely different commits containing their malicious payload. The GitHub release page didn't change. Same tag names. Same dates. Same descriptions. The malicious commits even reused the original author metadata and timestamps. If you were "pinned" to aquasecurity/[email protected], you silently pulled attacker-controlled code on your next build. No notification. No version change. No indication whatsoever that the code behind the tag was different.
This wasn't new. In March 2025, the same technique hit tj-actions/changed-files, a GitHub Action used by over 23,000 repositories. Attackers retroactively modified all version tags to reference a malicious commit. CISA added it to the Known Exploited Vulnerabilities catalog. The chain traced back through reviewdog/action-setup to a single stolen token from a spotbugs workflow. One token. Over 23,000 affected repositories.
During the Trivy incident, malicious Docker Hub images were published under existing version tags, with latest silently pointing to a compromised image during the exposure window. Pull aquasec/trivy:latest during those hours and you got malware.
npm claims its registry data is immutable ("once published, a package cannot change"), and at the version string level, that's technically true. But in the LiteLLM attack, stolen PyPI publishing credentials allowed the attackers to upload entirely new versions (1.82.7 and 1.82.8) that were indistinguishable from legitimate releases. The "immutability" of existing versions is cold comfort when an attacker can publish new ones that your unpinned or loosely-pinned dependencies will happily resolve.
The only truly immutable reference in this ecosystem is a full commit SHA hash. Not a version tag. Not a release name. A 40-character hexadecimal string that maps to a specific commit and cannot be rewritten. As Snyk noted in their Trivy analysis: "Tags are mutable. Anyone with write access can force-push a tag to point to a completely different commit. Commit SHAs are immutable."
Almost nobody pins to full SHA hashes. It's unwieldy, hard to read, and breaks the ergonomic assumptions that every package manager is built around. The entire ecosystem is designed around the convenient fiction that version tags are permanent. They aren't.
"I Don't Run .0 Releases. I Wait for .2"
This is a variant of the 20-day argument, applied to version maturity instead of time. The idea is that early adopters will find the problems and later point releases will be clean.
I understand the instinct. I've followed the same heuristic for years. But it addresses a different class of risk. Waiting for .2 protects you from bugs. It doesn't protect you from supply chain attacks.
A bug is an unintentional defect introduced during development. Supply chain attacks are intentional insertions of malicious code designed to survive exactly the kind of review and testing that patches are subjected to. The attacker isn't hoping you'll notice the problem. The attacker is specifically engineering code that looks legitimate, passes automated scanning, and slips through code review. That's the entire point.
Version 3.4.14 of a respected project isn't safer from supply chain attacks than version 3.5.0. It's been exposed to the public internet for longer, which means attackers have had more time to find ways into it, not less. And if the maintainer's credentials are compromised (as happened with Trivy, LiteLLM, and tj-actions), every version is equally vulnerable to tag rewriting or replacement.
"It's Computationally Impossible to Review 450 Packages"
Here's the one where I get to surprise people.
Using standard approaches, you're right. It is essentially impossible for a human team to meaningfully audit, fork, strip, scan, patch, and maintain 450 packages. The cognitive load alone would be prohibitive, let alone the engineering hours.
That's why I didn't do it with a human team. I did it with Claude.
I wrote about the approach in detail in Teaching Claude the Old Tricks. The broad strokes:
First, detach the packages and strip all test files. Run code coverage against our repositories to identify dead code from imports and remove it. The first pass cut 40%. A more detailed pass reduced it another 35% (75% total reduction).
Second, run static and dynamic code analysis and patch all vulnerabilities. Then run fuzzing and application penetration testing. We hit 8.6 million fuzz executions across 12 targets with zero crashes.
Third, refactor all functions to align with our internal standards (the WitFoo Way). Add code consolidation to the scope. This reduced another 10% of bloat.
Fourth, ensure no code or quality regressions with linters, unit tests, system tests, race detection, integration tests, and end-to-end tests. 10,157 tests, all passing, zero regressions.
Fifth, diff reviews on final commits from pre-fork baselines.
Total cost: approximately $200 in tokens and GitHub Actions compute. Four labor days. Not four engineer-months. Four days.
The capability that makes supply chain attacks more dangerous (AI-assisted development) is the same capability that makes this kind of defensive work feasible. That's the inversion I wrote about in The Closing Window: the economics of AI-assisted development are shifting, and the window of extraordinary value-to-cost ratio won't last forever. But right now, today, a nuclear fork is possible and affordable for most projects if you have a few well-placed licenses (GitHub Enterprise with Advanced Security and a Claude Max subscription, in our case).
"Releases Used to Mean Something"
They did. And I have enormous respect for organizations like Apache that still treat the release process with the gravity it deserves. Staging artifacts, signing JARs, community votes, three-day review periods. These aren't theater. They represent genuine care about the integrity of the software being shipped.
But the threat model has changed, and the release process hasn't kept up.
Apache's rigorous process didn't catch a CVSS 10.0 vulnerability that lived in Log4j for eight years. And that was a bug, not even a deliberate supply chain attack. The processes that worked when the primary risk was accidental defects are not sufficient when the primary risk is adversaries actively engineering code to survive your review process.
Releases still mean something. They're just not sufficient as a trust mechanism anymore. Not when the tools that build the release can be compromised. Not when the credentials that sign the release can be stolen. Not when the tags that identify the release can be rewritten. The ceremony of the release process is intact. The security guarantees it once provided are not.
The Accountability Question
Here's the part of the conversation that usually gets quiet.
When I use an open source package and something goes wrong, who pays? Not the maintainer. Not the foundation. Not the community. The entity that deployed the software into a production environment and delivered it to customers bears the legal liability. That's me. That's WitFoo. And it's you and your company too.
Apache has my deep respect. We use Cassandra in our stack. I know the level of rigor they put into vetting and releasing. But if they miss something (as they did with Log4j), they will not pay the $100 million or more in legal settlements to me or my customers. I can't pull an errors and omissions insurance policy that would cover those damages. The risk is entirely mine to absorb.
That's the calculus that matters. Not whether a process is mature. Not whether a project has good governance. The question is: if this dependency fails catastrophically, can I survive the consequences? And if the answer is no, then the only rational response is to take ownership of the code and be able to answer for every line of it.
I talked about this in my Birthing Perjury-free AI talks last year. In cybersecurity, accountability cannot be delegated. The person who deploys the software is the person who answers for its failures. Open source licenses are explicit about this. Every one of them includes a variation of "provided as-is, without warranty." That's not fine print. That's the deal. And it means the supply chain risk is yours, regardless of how many governance processes the upstream maintainer has in place.
I'm Not Advocating Everyone Fork Everything
I want to be clear about this. The nuclear fork was the right decision for WitFoo because of our specific risk profile. We build cybersecurity software that processes billions of signals daily. A compromised dependency in our stack doesn't just create a security incident for us. It creates a security incident for every organization that depends on our platform to protect them. The maximum destructive impact of a realized supply chain attack on our business is existential.
Not every project carries that risk profile. A marketing website, a personal project, an internal tool that doesn't touch sensitive data: the calculus is different. The conventional mitigations (pinning, scanning, waiting, using mature projects) may be sufficient for your risk tolerance.
But I am advocating that every engineering team have the conversation. Sit down. Look at your dependency graph. Count the packages. Ask yourself: if any one of these is compromised, what happens? And be honest about whether your current mitigations match the threat you actually face, not the threat you faced three years ago.
The things related to capabilities of both development teams and cybercriminals require a fresh set of table top discussions. What took eight years to come to a head with Log4j is taking hours now. The AI-assisted development revolution didn't just change how we build software. It changed how attackers compromise the software we depend on.
Wrap Up
Every objection I've addressed in this post was reasonable once. Use mature projects. Wait for stable releases. Pin your versions. Trust the process. These were good heuristics when the primary supply chain risks were accidental defects and the occasional lone-wolf malicious package.
We are not in that world anymore.
We are in a world where a security scanner becomes the attack vector. Where AI agents are used operationally in supply chain attacks. Where a single stolen token cascades through 23,000 repositories. Where 75 of 76 version tags can be silently rewritten overnight. Where the most respected governance organization in open source can ship a maximum-severity vulnerability for eight years without anyone catching it.
The conventional wisdom isn't wrong in principle. It's wrong in calibration. The defenses are tuned to a threat landscape that no longer exists. And the gap between where those defenses are calibrated and where the threats actually operate is where the next breach lives.
I'm not asking anyone to make the same decision I made. I'm asking everyone to make a decision, with eyes open, about what they're actually trusting and whether that trust is still warranted.
As I wrote in Hypnosis of your Tech nearly a decade ago: it's great to love our tools, but we have to keep our eyes on the mission. The mission hasn't changed. The risk to the mission has.
Charles Herring is co-founder and CEO of WitFoo, a cybersecurity company building collective defense solutions. He is a US Navy veteran and speaks regularly at security conferences including DEFCON, GrrCON, and Secure360. You can find him on LinkedIn.
