This tutorial, presented at EuroSys 2009, introduced the principles and practice of dependability benchmarking to a systems community accustomed to measuring performance. At the time, the gap between the precision of performance benchmarks and the informality of dependability evaluation was stark, and growing as systems became more complex and interconnected.

The tutorial covered the foundations: what a dependability benchmark must specify (workload, faultload, metrics, and experimental environment), why each element matters for reproducibility, and how the community could build on existing performance benchmarking infrastructure to do for dependability what TPC had done for transaction processing. Several concrete benchmarks for OLTP systems and web servers were used as worked examples.