Backslide joins the two files, scores only the things that can honestly be called
worse, and hands you the rows that broke — with the reason attached. It reads
promptfoo, OpenAI Batch and Evals exports, raw completion dumps, JSONL and CSV.
Run A — before
Drop a file
or
Run B — after
Drop a file
or
case
verdict
reason
Δ chars
steps A/B
Settings
JSON Schema optional
Paste a schema and Backslide reports every row that satisfied it before and does not
now, quoting the exact property that failed.
Thresholds
Both conditions must hold. A ratio alone would flag 100 ms → 300 ms; a floor
alone would flag every long call.
Tool-argument normalisation
Two runs of the same agent never produce the same request ids or timestamps. These rules
decide what counts as “the same step”. Getting them wrong is not symmetric:
too little normalisation floods you with false alarms, which you notice; too much hides
real changes, which you do not. Watch the collision count.
Take it with you
What this cannot tell you
1 · Two files in, one list out
Drop the run from before your change and the run from after. Backslide works out what format
they are, lines the rows up, and shows you only the ones that got worse. Nothing is uploaded;
both files are read by the browser's own file API.
2 · Lining the rows up is the hard part
If your file has case ids, Backslide uses them — and then checks them, because
testIdx and data_source_idx are positions, not identities. Insert
three test cases at the top of your config and every id points at the wrong row. If the ids
and the inputs disagree, Backslide throws the ids away and joins on the prompt text instead,
and tells you it did.
With no ids at all it hashes the prompts. Because you changed one line of the system prompt,
that line is common to every row of the second file — so it is stripped before hashing and
the rest matches exactly. Row order is irrelevant either way.
3 · Only things that can be called “worse”
Did it stop parsing. Did it stop matching your schema. Did an assertion your own eval already
recorded flip from pass to fail. Did it error, or get cut off at the token limit. Did it get
materially slower or dearer. Everything else — how much text changed, how the length moved —
is shown but explicitly
.
4 · Each row is counted once
An errored row has an empty output, so it would also fail a JSON check. Counting both would
make the columns sum to more than the number of rows. The first check that explains a row owns
it and the rest are marked not-counted, so the numbers always add up.
5 · Agent runs are aligned, not zipped
If your rows contain tool calls, Backslide
the two
sequences rather than comparing them position by position. One extra step at index 2 costs one
gap, not fourteen phantom differences — and an extra step is reported as a
divergence, never a regression. A longer path is not a worse path.
6 · You leave with four files
regressions.jsonl — only the broken rows, in your file's own shape.
regression.csv — the audit trail. regression-report.md — for the pull
request. diff-report.html — one self-contained page with no external references.
Protected
Not protected
Trust surface
Backslide takes two runs of the same eval and gives you back only the rows
that got worse, with the reason attached. It runs entirely in your browser.
It exists because the files you need to compare are the ones you least want to upload. An eval
output holds your prompts, your customers' inputs, your model's answers and often a reference
set you paid someone to write. The tools that would answer “what did my change break?” are an
observability platform with an account and a bill, or twenty lines of pandas you write again
every time.
The alignment of tool-call trajectories is the part that is genuinely hard, and the part
nothing else does: every agent-evaluation library scores a run against a hand-written expected
sequence rather than against another real run.
Three libraries ship in the bundle and no more. Schema validation uses
@cfworker/json-schema,
chosen because it compiles no code — the usual choice, ajv, builds its
validators with the Function constructor and cannot run under this page's
content policy at all. The diff view uses
jsdiff and
edit distance uses
fastest-levenshtein.
The adapters, the join, the scorers, the classifier, the JSON-shape analyser and the
trajectory aligner are first-party and unit-tested.
