These days, I received an email that brought me back in time. Nothing dramatic, just a routine thread of the Transaction Processing Performance Council (TPC), but it was enough to surface old memories.
I have spent twenty-five years researching benchmarking, in one form or another, dependability, security, performance, trustworthiness, applied across all kinds of tools and systems along the way. Along that path, I brought the University of Coimbra into both TPC and SPEC to introduce dependability considerations within these communities, and at one point chaired SPEC’s Security Benchmarking Working Group. More on that here.
Here is one small thing that has stayed with me through all those years. TPC and SPEC are two of the most respected names in the field: careful, rigorous, audited. Yet, no one ever taught me how to compare two benchmarks when they disagree, or how to tell whether a benchmark is measuring the right thing in the first place. Twenty-five years in, I am still not sure anyone has taught anyone else how to do that, either.
I do not have a conclusion to offer for this one yet. Just a question that has followed me for twenty-five years and has not gotten any less strange with time.
I’m thinking about it…