Software security is notoriously difficult to measure — partly because security is about the absence of failures, and partly because the space of possible attacks is essentially unbounded. Yet measurement is exactly what benchmarking promises. This seminar, presented at the Federal University of Pernambuco (UFPE), examined how far security benchmarking has come and how far it still needs to go.

Drawing on a decade of work spanning vulnerability detection tools, web application security, and DBMS configurations, the talk reviewed existing approaches to security benchmarking and identified the methodological requirements for making those benchmarks trustworthy: representative workloads, credible vulnerability corpora, meaningful metrics, and transparent experimental procedures. It concluded by sketching a research agenda for the community.