Presented at the LASIGE Workshop at the University of Lisbon, this seminar explored the state of security benchmarking and its inherent challenges. Security is fundamentally about the absence of exploitable weaknesses, a property that resists the straightforward measurement techniques that work well for performance or even dependability.

The talk examined what makes a security benchmark credible: the workload must include realistic attack scenarios, the metrics must capture what actually matters for defenders, and the experimental setup must be transparent enough for others to reproduce and challenge the results. Case studies from web application security and vulnerability detection tool evaluation illustrated both the promise and the persistent difficulties of the enterprise.