For tool builders

A benchmark with no cheat sheet inside

Point a tool at the shop and it has to find the flaws itself. Score what it finds against a held-back key, and a run becomes a number instead of an opinion.

No answer key inside the target

This is what makes the score mean something. The vulnerable app ships with no list of its own flaws: no manifest in the code, no labels, no solutions. A scanner or an AI agent has to actually find things, not read them off a cheat sheet in the repo.

The answer key is held back. For each exercise it records the exact spot, the class, and a proof that fires it, roughly like this:

{
  "id": "FF-001",
  "cwe": "CWE-89",
  "endpoint": "GET /api/sweets/search",
  "parameter": "q",
  "proof": "' UNION SELECT email,password_hash FROM customers-- "
}

Score a run

Run your tool against the shop, then line its findings up against the held-back key: found, missed, or false positive. The human write-up for each exercise lives here on the site, so you can check your work once you are done.

The rules that keep it honest

The key is fixed before any tool runs, and never edited to flatter a score.
Misses are reported next to finds. A tool that catches 13 of 15 and says so is worth more than one that claims 15.
The target carries no hints, so a good score reflects the tool, not the answer key leaking into the code.

FreeFungi is vendor-neutral. Benchmark posts here report every tool the same way, finds and misses alike.