Vulnerability detection tools are evaluated constantly, yet the community had (and to a considerable extent still has) no consensus on which metrics to use. Detection rate? False positive rate? Coverage of specific vulnerability classes? Ranking tools by different metrics produces different rankings, and without a principled framework for choosing among them, comparison is essentially impossible.

This seminar at City University London examined the metrics problem directly, drawing on earlier DSN 2015 work. It presented a systematic analysis of the metrics landscape for vulnerability detection tool evaluation and proposed criteria for selecting and combining metrics in a way that is both technically sound and useful for practitioners who must choose tools for real deployments.