Category Archives: publication

Technical report: Benchmarking Crimes: An Emerging Threat in Systems Security

Or: if you can’t do the time, don’t do the crime

Several days ago, we released a technical report entitled Benchmarking Crimes: An Emerging Threat in Systems Security.  The paper was intended for publication at a security conference but was rejected at multiple venues. To let our work be a supporting piece of evidence and analysis for the community to build on, we share our work with the community as a technical report, and we publish it on Arxiv.org.

The results are as revealing as they are damning: we formulate 22 different benchmarking crimes, each of which violates the results of a benchmark in a minor or major fashion. We survey 50 different systems security defense papers. We include papers published by this group in that selection. To gauge reliability, the survey is performed twice – we let two independent readers perform this survey. Their findings are consistent: in this wide study of  accepted papers at top systems security venues, all papers had committed benchmarking crimes in some number and degree of egregiousness.

Most of these are recent papers (2015), but a significant fraction are from 2010. This longitudinal component of the study tells us that not only are benchmarking crimes widespread, but also no better in modern papers than in older ones.

This raises the question of how we can trust benchmarks in research results. We hope our work will contribute to an improvement in this situation.

The Register has coverage.

ASLR^Cache or AnC: A MMU Sidechannel breaking ASLR from Javascript, and media coverage

Today we announce ASLR^Cache, a MMU sidechannel exploiting a micro-architectural property of all modern CPU models. This signal is even visible from Javascript and breaks ASLR in sandboxed environments. The name ASLR^Cache (or simply AnC) is a reference to the fact that ASLR and CPU caches are mutually exclusive on modern architectures. For more information, please see our AnC project page.

Press outlets and other organisations have picked up on this work: wiredarstechnica, ACM Tech NewsNCSCbleepingcomputer.comTom’s Hardwaresecurity.nltheregistertweakers.netdigitaljournal.comCSO Australiahackadayslashdotsecurityweek.comheise.detheinquirer.netitnews.com.au, eejournal.comhabrahabr.ruimpress.co.jppaper.li, boingboing.net.

Also some of our favourite podcasts picked it up: securitynow episode 600,  ISC Internet Storm Center podcast, risky.biz episode #444.