H
Harvest
AI Summarized Content

How to Avoid AI Code Slop: Methods Behind the SpaceX Grok Bot Team's 10,000 PRs and 0.5% Rollback Rate

This video explores how Lauren Tan (known as "Potato") and the SpaceX Grok Bot team successfully merged nearly 20,000 pull requests in a single month with a rollback rate of just 0.5%. It introduces the "Potato Pyramid" framework, emphasizing that trust in AI-generated code is built through robust codebases, automated linting, runtime verification, and shifting focus from human code reviews to verifying evidence like screenshots. Ultimately, the talk teaches us how to stop treating AI output as messy "slop" and turn it into reliable software engineering.


1. The Paradox of AI Code and the Potato Pyramid

In October, the Grok Bot team merged an astonishing 19,260 pull requests while keeping the rollback rate down to about 0.5%. This feat was presented by Lauren Tan at the AI Engineer Code Summit under the talk title "All Code Is Slop, Except for Mine."

"All Code Is Slop, Except for Mine."

The term "slop" refers to the low-quality, hastily generated code churned out by AI. To transition from a "low-trust loop"—where humans constantly fix agent-written slop—to a "high-trust loop" where agents write good code and other agents verify it, the team relies on a conceptual framework called the Potato Pyramid.

"To move from left to right, where should you spend your time in an agent's environment? Lorentani answers with a single diagram."

This inverted triangle has five distinct sections, wide at the top and narrow at the bottom:

  1. Codebase (Widest top section)
  2. Static Analysis
  3. Verification
  4. Prose
  5. Human Review (Narrowest bottom section)

The narrow bottom sections are reserved for text-based rule files and agent configuration instructions, emphasizing that human judgment and textual prompts should only occupy a tiny fraction of the workflow.


2. Managing the Codebase and Banning Comments

Starting from the widest section—the codebase—the team realized that quick fixes often get copied repeatedly until they form harmful anti-patterns. To prevent this, the Grok Bot team uses a custom framework called Dune to establish strict structural boundaries, such as separating front-end UI code from backend logic to prevent lagging.

A striking lesson learned by the team involved the dangers of code comments and quick timers:

"We banned the tricky one from the codebase entirely, and separated the mixed code with folders and linting. In both cases, we don't have to see them in code reviews anymore."

Timers like setTimeout were rapidly created to fix minor UI flickering, leading to heavy garbage collection and severe app lag. Agents would then write extensive comments justifying these quick fixes. However, the team discovered a hard truth about comments:

"We decided to just ban 99% of comments because they go stale easily. Humans just really don't bother reading the comments anyway. And again, it gives an agent an escape hatch to justify not doing the right thing because it's hard. And the worst part about comments, though, is that they're not enforceable."

Instead of relying on unenforced comments, the team turns every agent correction into a deterministic lint rule or an architectural fix.

"Our general principle is that every time we correct our agents, it turns into one of these deterministic checks, or we think about how to categorically eliminate that problem in the code through better API design, through better architecture."


3. The Power of Verification and Evidence-Based Review

The third and most crucial level of the Potato Pyramid is verification. When agents are allowed to merge most pull requests autonomously, human reviewers stop looking at raw code and instead review the evidence of verification.

"What humans look at isn't the code, but the evidence of verification. Things like videos, screenshots, and diagrams."

Runtime verification involves the agent running the actual application and capturing screenshots or videos as proof. The team also incorporates advanced formal verification tools:

  • Tla+: Maps out every possible system state to catch missed edge cases.
  • Lean: Verifies algorithm correctness step-by-step like a mathematical proof.
  • Property-based testing: Floods functions with random inputs against pre-defined rules rather than relying on a few manual examples.

This heavy emphasis on testing means that a massive portion of daily resources goes directly into validation.

"70% of daily tokens go toward verification."

Lauren Tan also developed a plugin called Pstack (Potato Stack), which includes tools like State Chart to help visualize state machines from the user's perspective, making them much easier to review alongside videos and screenshots.


4. The Results and the Golden Age of Testing

Investing heavily in codebase hygiene, automated linting, and rigorous verification yielded incredible results. The team achieved rapid daily releases (averaging around 25 to 28 hours per release interval) with massive numbers of contributors and PRs, all while maintaining that remarkably low 0.5% rollback rate.

"Agents drastically cut this cost down, and really, we're entering a golden age of software testing and verification because agents let us test everything to a much greater degree than used to be economical."

However, order of operations matters immensely. If your inner loop generates slop, automating it will only produce more slop at scale. You must first block anti-patterns with linting and clean up existing code before opening the door for full automation.


Conclusion

The presentation concludes by playfully tweaking its original title:

"All Code Is Slop, Except for Yours."

You don't need to completely overhaul your repository overnight. Start from the top of the Potato Pyramid: the next time an agent makes a repeating mistake, turn that correction into a strict lint rule or test rather than adding more text to a rule file or writing stale comments. By shifting your focus from reviewing lines of code to evaluating verification evidence like screenshots and test outputs, you can build a high-trust loop where AI coding agents become genuinely reliable partners in software development.

Summary completed: 11/18/2026, 3:14:00 PM

Need a summary like this?

Get instant summaries with Harvest

⚡
5-second summaries
AI-powered analysis
📱
All devices
Web, iOS, Chrome
🔍
Smart search
Rediscover anytime
Start Summarizing
Try Harvest