Technical report: Benchmarking Crimes: An Emerging Threat in Systems Security

Or: if you can’t do the time, don’t do the crime

Several days ago, we released a technical report entitled Benchmarking Crimes: An Emerging Threat in Systems Security.  The paper was intended for publication at a security conference but was rejected at multiple venues. To let our work be a supporting piece of evidence and analysis for the community to build on, we share our work with the community as a technical report, and we publish it on

The results are as revealing as they are damning: we formulate 22 different benchmarking crimes, each of which violates the results of a benchmark in a minor or major fashion. We survey 50 different systems security defense papers. We include papers published by this group in that selection. To gauge reliability, the survey is performed twice – we let two independent readers perform this survey. Their findings are consistent: in this wide study of  accepted papers at top systems security venues, all papers had committed benchmarking crimes in some number and degree of egregiousness.

Most of these are recent papers (2015), but a significant fraction are from 2010. This longitudinal component of the study tells us that not only are benchmarking crimes widespread, but also no better in modern papers than in older ones.

This raises the question of how we can trust benchmarks in research results. We hope our work will contribute to an improvement in this situation.

The Register has coverage.

Vusec commentary on Meltdown and Spectre

There has been significant media coverage over the CPU flaws known as meltdown and spectre. In the wake of one of our researchers, using only public information and speculation, reproducing the bug before the embargo got lifted, the vusec group has been asked to comment in several pieces, including,,, nos.nlHBO Vice news tonight (video),

DRAMMER wins CSAW Applied Research Best Paper Award

Drammer won first 2017 CSAW prize in the category Applied Research Best Paper Award.

Luckily, Victor was there to receive this award on behalf of the Drammer team!

To quote the CSAW site:

Accepted papers are presented by one of the student authors in poster-format during CSAW Finals. Industry experts serving as judges evaluate the originality, relevance, and accuracy of the research.

With eligibility limited to previously published papers, this competition has a reputation for drawing some of the best doctoral security research worldwide.

vusec is proud of this academic and industry recognition.

Fortifying Anemic Dutch Cybersecurity Investment

Brain Drain

Prof. Herbert Bos, Prof. Michel van Eeten, and Prof.  Bart Jacobs on the 24th released a joint Dutch statement and proposal on the inadequacy of academic cybersecurity funding in The Netherlands. Funding that is up to 50x higher in neighboring countries is causing a drain of talented researchers away from The Netherlands.

Cybersecurity Investment Proposal

The proposal calls for the development of a three-pronged strategy to maintain the high academic standard of Dutch research organizations, funded by in total a budget of €100 million over 10 years, in a combination of public and private investment.

  1. €40M (public): fund open tenders for non-permanent PhD and postdoc projects, where both pure-CS and interdisciplinary proposals will be considered. Examples might be legal, medical and organizational fields.
  2. €20M (public): a budget of €2M/year for which universities may apply to either (a) hire permanent staff for a newly appointed cybersecurity professor; or (b) retain staff, done by a cybersecurity professor with at least 5 years proven record, to establish areas of new research.
  3. €40M (private): The establishment of a pool of inter-organizational cybersecurity experts. The organizations will be a combination of research, government and industrial organizations that host the members. These members will then share knowledge, deepen knowledge (by following an external or industrial PhD program), and provide operational expertise in emergencies.


This proposal was covered in Computable last week and Prof. Bos was a guest on BNR News Radio at 06:00 AM this morning for discussion.

AnC wins Pwnie Award at Blackhat USA 2017

AnC won the 2017 Blackhat Pwnie award in the category Pwnie for Most Innovative Research. Luckily, Victor was there to receive this award on behalf of the AnC team!

To quote the pwnie award site:

  • Credit: Ben Gras, Kaveh Razavi, Erik Bosman, Herbert Bos, Cristiano Giuffrida

    Exploit writers have been bending over backwards to try to defeat ASLR for the better part of a decade. Usually this requires finding some soon-to-be-patched memory disclosure bug. Of course this is a hard job and needs to be repeated for different browsers/plugins/versions/etc. Then these guys come along with a universal ASLR bypass based on timing of the caching of memory access. Of course this works using Javascript in most browsers by default and isn’t really something you can fix very easy. Seems too easy, I think I’ll keep looking for infoleaks like a real hacker.

Vusec is proud of the industrial and scientific recognition of this work.

DRAMMER wins pwnie award at Blackhat USA 2017

DRAMMER won the 2017 Blackhat Pwnie award in the category  Pwnie for Best Privilege Escalation Bug. Luckily, Victor was there to receive his award in person!

To quote the pwnie award site:

  • Credit: Victor van der Veen, Yanick Fratantonio, Martina Lindorfer, Daniel Gruss, Clementine Maurice, Giovanni Vigna, Herbert Bos, Kaveh Razavi, Cristiano Giuffrida

    Mobile computing row hammer attacks (MC Hammers, for short) are terrifying. You can’t touch them and can only hope that, please, they won’t hurt you.

Vusec is proud of the industrial and scientific recognition of this work.