NetCAT shows that network-based cache side-channel attacks are a realistic threat. Cache attacks have been traditionally used to leak sensitive data on a local setting (e.g., from an attacker-controlled virtual machine to a victim virtual machine that share the CPU cache on a cloud platform). With NetCAT, we show this threat extends to untrusted clients over the network, which can now leak sensitive data such as keystrokes in a SSH session from remote servers with no local access. The root cause of the vulnerability is a recent Intel feature called DDIO, which grants network devices and other peripherals access to the CPU cache. Originally, intended as a performance optimization in fast networks, we show DDIO has severe security implications, exposing servers in local untrusted networks to remote side-channel attacks.
Intel agrees this is a significant vulnerability, having awarded NetCAT a bounty and recommending users to “limit direct access from untrusted networks when DDIO & RDMA are enabled“. This essentially means that in untrusted network environments DDIO and/or RDMA should be disabled to provide security. To the best of our knowledge, this is the first time a major hardware vendor like Intel cautions against using a CPU feature in untrusted local networks.
What is DDIO?
Data-Direct I/O (DDIO) is a performance-enhancing technology on recent Intel server-grade processors. Instead of reading/writing from/to slow memory, DDIO allows peripherals to read/write from/to the fast (last-level) cache. DDIO was specifically introduced to improve the performance of server applications in fast networks.
In traditional architectures, where the network card uses direct memory access (DMA) to talk to the operating system, the memory latency alone quickly becomes the bottleneck on fast (e.g., 10Gb/s) networks. To alleviate the bottleneck, Intel introduced DDIO, an architecture where peripherals can operate direct cache access on the CPU’s (last-level) cache. The DDIO cache region is not dedicated or reserved in the cache, but allocating writes are statically limited to a portion of the cache to avoid thrashing caused by I/O bursts or unconsumed data streams. The figure below illustrates the difference between direct cache access (orange) and direct memory access (blue). Additionally, the available write allocation cache lines for direct cache access in orange versus the others in green.
In this project, we present the first security analysis of DDIO. Based on our analysis, we present NetCAT, the first network-based cache attack on the processor’s last-level cache of a remote machine. We show that NetCAT can break confidentiality of a SSH session from a third machine without any malicious software running on the remote server or client. The attacker machine does this by solely sending network packets to the remote server.
More precisely, with NetCAT, we can leak the arrival time of the individual network packets from a SSH session using a remote cache side channel. Why is this useful? In an interactive SSH session, every time you press a key, network packets are being directly transmitted. As a result, every time a victim you type a character inside an encrypted SSH session on your console, NetCAT can leak the timing of the event by leaking the arrival time of the corresponding network packet. Now, humans have distinct typing patterns. For example, typing ‘s’ right after ‘a’ is faster than typing ‘g’ after ‘s’. As a result, NetCAT can operate statical analysis of the inter-arrival timings of packets in what is known as a keystroke timing attack to leak what you type in your private SSH session.
In our attack, we exploit the fact that the DDIO-enabled application server has a shared resource (the last-level cache) between the CPU cores and the network card. We reverse engineered important properties of DDIO to understand how the cache is shared with DDIO. We then use this knowledge to leak sensitive information from the cache of the application server using a cache side-channel attack over the network. To simplify the attack, similar in spirit to Throwhammer, we rely on Remote Direct Memory Access (RDMA) technology. RDMA allows our exploit to surgically control the relative memory location of network packets on the target server.
The figure below illustrates our target topology, which is common in data centers. The attacker controls a machine which communicates over RDMA to an application server that supports DDIO and also services network requests from a victim client. NetCAT shows that attackers can successfully spy on remote server-side peripherals such as network cards to leak victim data over the network.
We initiated a coordinated disclosure process with Intel and NCSC (the Dutch national CERT) on June 23, 2019. The vulnerability was acknowledged by Intel with a bounty and CVE-2019-11184 was assigned to track this issue. The public disclosure was on September 10, 2019.
- Is DDIO enabled by default?
Yes, DDIO is enabled transparently by default in all Intel server-grade processors since 2012 (Intel Xeon E5, E7 and SP families).
- Does writing side channel-resistant (constant-time) software help?
No, with NetCat, we are remotely side channeling the activity of the hardware (i.e., a network device). As long as the network card creates distinct patterns in the cache, NetCAT will be effective regardless of the software running on the remote server. Nonetheless, Intel’s recommendation to deploy side channel-resistant software may be helpful against future NetCAT-like attacks that target victim software on DDIO-enabled machines.
- I use a public cloud provider. Am I at risk?
If DDIO is available/enabled on your platform, you are affected by the vulnerability. If RDMA is also enabled, the vulnerability immediately exposes your server to practical side-channel attacks over the network, as demonstrated by our NetCAT exploit.
- I am a public cloud provider. How can I protect my customers?
Disable DDIO to eradicate the vulnerability (or at least RDMA to reduce its impact).
- How can I disable DDIO?
You can disable DDIO by adjusting the Integrated I/O (IIO) configuration registers. There are two possibilities, changing it globally (Disable_All_Allocating_Flows bit) or per root PCIe port (NoSnoopOpWrEn and Use_Allocating_Flow_Wr bit). We successfully mitigated NetCAT by setting these bits on our Intel Xeon E5 cluster. For the Intel Xeon SP families, the offsets of these bits are not (yet) publicly documented.
- Why the name NetCAT instead of something more frightening?
We do not believe in fear-mongering with our research. NetCAT stands for Network Cache ATtack. “netcat” is also a famous utility that hackers and system administrators use to send information over the network. cat is the UNIX tool to read the contents a file, netcat is supposed to do the same over the network. NetCAT is a pun on being able to read data from the network without cooperation from the other machine on the network. If that is not enough pun already, DDIO makes use of the Cache Allocation Technology (CAT) internally. Finally, who doesn’t like cats?