Benchmarking has successfully driven performance improvements in systems for decades, but applying it to dependability and security remains exceptionally challenging. This keynote critically examines the state of the art in security and dependability benchmarking.

We explore the essential prerequisites, grand challenges, metrics, and tools required to make security assessments repeatable, reproducible, and trustworthy across both industry and academia. The talk argues that without a principled benchmarking discipline, claims about system security and dependability remain anecdotal, and that the community has both the opportunity and the obligation to change that.