Prompt DiffPrompt Diff

Prompt Diff started as a shell script we used to compare two versions of a prompt across the same set of inputs. It grew into a small CLI that runs both versions and prints the outputs next to each other.

It highlights what changed between runs, caches responses so repeat comparisons are free, and exits non-zero when a regression check fails, so it also works in CI.

#What we learned

Most prompt changes help one case and quietly hurt two others. Seeing them side by side is the only reliable way to notice.

Caching by prompt hash and input made the tool fast enough that people actually used it before every commit.