This year, VUSec had 2 papers accepted at USENIX Security ’18: Malicious Management Unit (how to use the MMU to mount indirect cache attacks and bypass software-based defenses) and TLBleed (how to mount TLB side-channel attacks across threads and leak fine-grained information).
Or: if you can’t do the time, don’t do the crime
Several days ago, we released a technical report entitled Benchmarking Crimes: An Emerging Threat in Systems Security. The paper was intended for publication at a security conference but was rejected at multiple venues. To let our work be a supporting piece of evidence and analysis for the community to build on, we share our work with the community as a technical report, and we publish it on Arxiv.org.
The results are as revealing as they are damning: we formulate 22 different benchmarking crimes, each of which violates the results of a benchmark in a minor or major fashion. We survey 50 different systems security defense papers. We include papers published by this group in that selection. To gauge reliability, the survey is performed twice – we let two independent readers perform this survey. Their findings are consistent: in this wide study of accepted papers at top systems security venues, all papers had committed benchmarking crimes in some number and degree of egregiousness.
Most of these are recent papers (2015), but a significant fraction are from 2010. This longitudinal component of the study tells us that not only are benchmarking crimes widespread, but also no better in modern papers than in older ones.
This raises the question of how we can trust benchmarks in research results. We hope our work will contribute to an improvement in this situation.
The Register has coverage.
Press outlets and other organisations have picked up on this work: wired, arstechnica, ACM Tech News, NCSC, bleepingcomputer.com, Tom’s Hardware, security.nl, theregister, tweakers.net, digitaljournal.com, CSO Australia, hackaday, slashdot, securityweek.com, heise.de, theinquirer.net, itnews.com.au, eejournal.com, habrahabr.ru, impress.co.jp, paper.li, boingboing.net.
This year, VUSec had 4 papers accepted at NDSS ’17: AnC (a new side-channel-based ASLR bypass), SafeInit (efficient protection against uninitialized reads), a new evolutionary fuzzer (AFL on steroids), and Marx (uncovering class hierarchies in C++ programs, with @thorstenholz’s group at @ruhrunibochum).
This year, VUSec had 2 papers accepted at EuroS&P ’17: Nucleus (compiler-agnostic function detection) and CodeArmor (how to efficiently re-randomize code every few microseconds).
This year, VUSec had 2 papers accepted at CCS: Drammer (Deterministic Rowhammer attacks) and TypeSan (a practical type confusion detector).
This year, VUSec presented 3 papers at DSN. (1) OSIRIS (efficient and consistent whole-OS crash recovery), (2) HSFI (scalable and representative fault injection), (3) MvArmor (secure and efficient MVX with Dune). All the code is open source. Check it out at https://github.com/vusec. OSIRIS was selected for the Best paper session.