Differential execution for Python
RunBoth runs both versions of your code on generated inputs and tells you what actually behaves differently, including the functions nobody touched. No test suite required.
The gap
A refactor is supposed to look different. That is exactly why nobody catches the one that behaves different.
Reviewers read intent, not behaviour. A clean rename and a changed comparison operator look the same at a glance.
Your suite covers what you thought to cover. Prototype code written fast usually has no suite at all.
One line moved in a helper. A function three files away now returns something else, and its own source never changed.
How it works
Both commits are checked out at once. For every changed function, inputs are
generated from its signature and from the constants mined out of its own bytecode, so a
function containing units >= 100 gets tested at 99, 100 and 101. Both versions
then run in separate sandboxed subprocesses on the same inputs, in the same order.
The before version runs a second time, in a third fresh process, against itself. If it
disagrees with itself the comparison is meaningless, so RunBoth abstains with that reason
rather than reporting a difference. That is what catches uuid4,
os.getpid and unseeded randomness.
From what changed, it walks outward through the call graph and executes the callers too, carrying the values already proven to differ. This is the part with no equivalent in a diff-based reviewer: the caller's source did not change, so there is nothing to review, and it still broke.
The exact arguments, what it used to do, what it does now. A fact you can re-run.
N generated inputs found no difference across seven channels. Evidence, never proof.
It could not be checked, and here is why. Never counted as passing.
Three verdicts instead of two is the deliberate part. "Cannot tell" and "no difference" are different claims, and collapsing them into a green check is how tools end up lying. RunBoth never says safe.
What it caught
toolz, and said it hadn't.toolz is a real library you have probably installed.
tail(n, seq) returns the last n elements. A frontier model was
asked to refactor it and to preserve behaviour. It added a guard that reads like exactly
the kind of defensive check a careful engineer writes.
+ if n <= 0:
+ return type(seq)() if hasattr(seq, '__getitem__') else tuple()
It looks obviously correct. It is not. In the
original, seq[-n:] with n = 0 is seq[0:], the whole
sequence.
| call | toolz | after the refactor |
|---|---|---|
| tail(0, [1, 2, 3]) | [1, 2, 3] | [] |
| tail(-1, [1, 2, 3]) | [2, 3] | [] |
| tail(2, [10, 20, 30, 40, 50]) | [40, 50] | [40, 50] |
A correct answer became a silently empty one. No exception, no failing test: the docstring example still passes, which is why the doctest stays green and a reviewer waves it through. Then RunBoth ran both versions.
$ git commit -m "refactor(itertoolz): simplify tail, behaviour unchanged" BLOCKED: your commit message says the behaviour did not change. It did. tail(-2, [1, 2, 3]) used to: return [3] now: return [] That call is the proof. Anything relying on the old result behaves differently now, and your test suite did not stop this commit.
No test was written and no annotation was added. The inputs came from the function's own signature and the constants in its own bytecode.
Across 36 functions from toolz and
markupsafe. One model and one prompt style, so it is a measurement rather than a
law, and the before and after source of every failure is published with it.
Proof
Real historical commits from public projects, with an automated oracle built to catch the tool lying: for every function reported as changed, check whether its source actually changed, and whether a changed callee or constructor explains it. Anything else is a probable false positive.
| Repository | Layout | Functions | Abstained | Median | False pos. |
|---|---|---|---|---|---|
| boltons | flat | 268 | 0.0% | 7.9s | 0 |
| sqlparse | flat | 230 | 0.9% | 14.3s | 0 |
| arrow | flat | 283 | 1.4% | 118s | 0 |
| cachetools | src/ | 325 | 1.8% | 60.9s | 0 |
| more-itertools | flat | 843 | 4.4% | 102.5s | 0 |
| packaging | src/ | 78 | 5.1% | 0.2s | 0 |
| pluggy | src/ | 157 | 6.4% | 33.5s | 0 |
| tenacity | async | 364 | 14.3% | 192.8s | 0 |
arrow is the one that matters most: a datetime library is the worst case for a differential tester, and it produced no false positives across 283 functions. tenacity is the honest other end, a retry library whose functions genuinely sleep, so checking it is slow and it abstains more.
What it is not
Translate code into logic, let inputs be unconstrained symbols, ask a solver whether a bad state is reachable. You get a proof, and you pay in modelling effort and a limited language subset.
Damage your code on purpose to see whether your tests notice, and score the suite. Needs a test suite to be worth anything.
Executes both real versions on concrete inputs and compares seven channels. Mutates nothing, needs no suite, and hands you a witness you can paste into a REPL.
Start
Installs a commit-msg hook. It stays silent unless
behaviour moved, and it never blocks on an abstention.
pip install runboth runboth install-hook
Runs on your own runners, so there is no service to trust and nothing leaves your CI. Comments once, edits itself after.
- uses: runboth/runboth@v1
with:
budget: 60Free for every use except building a competing product, and it converts to plain Apache 2.0 two years after each release.
FAQ
No. It generates its own inputs from each function's signature and from the literal constants already present in that function. That is the point for prototype code written fast, which usually has no suite at all.
No. The shipped package makes no network calls, declares no dependencies and contains no AI model. Every verdict is backed by an execution you can re-run yourself.
It blocks only on a measured behaviour change, never on an abstention, and it reconciles against your commit message: if you said what you were changing, it lets it through. A gate that argues with work you did on purpose gets uninstalled the same afternoon.
No, and it will never say so. Sampling finds differences; it cannot prove their absence.
The verdict is worded no_change at budget 60 for exactly that reason.
The before version is run twice in separate processes and compared against itself. If it disagrees with itself, RunBoth abstains rather than reporting a difference. An adversarial corpus of 22 functions built to trigger false positives produced none.
Function-level checking is Python. Changed files in other languages are named explicitly rather than passed over quietly, because silence is how this tool reports no difference found, and silence must never cover something it did not look at.
Get the release
The commit gate and the Action, plus the write-up of what red-teaming it on eight libraries actually found. No newsletter, no drip sequence, and the address is never passed on.