Security benchmarking sits at the intersection of two hard problems: measuring security (which is inherently about the absence of failures) and making those measurements reproducible and comparable across systems and contexts. This talk confronts both problems directly.

Drawing on years of work benchmarking vulnerability detection tools, web application security, and DBMS configurations, this keynote asks what it would take for security benchmarks to be genuinely trustworthy. It examines the failure modes of current practice (cherry-picked workloads, opaque metrics, irreproducible results) and proposes the foundations of a benchmarking methodology that could do better.