Scalable Data Management on Next-Generation Networks beyond RDMA (BI2011/1-2)
PI: Carsten Binnig (TU Darmstadt)
Project Collaborator: Tobias Ziegler
Project website: Scalable Data Management on Next-Generation Networks beyond RDMA
The efficient use of networks plays a substantial role for the scalability of distributed in-memory DBMSs. While in the past, communication networks have been slow and passive, modern networks have significantly changed. Modern networks not only enable high bandwidth and low latency communication via Remote-Direct-Memory-Access (RDMA). While RDMA provides significant performance benefits for scalable in-memory DBMSs, it also has many inherent limitations. A main reason is that the limited set of RDMA operations often leads to complex DBMS protocols.
In Phase II of the SPP, we will thus use the programmability of the next generation of high-speed network cards and switches to address these inherent limitations.
The main directions of the proposal are:
(1) First, we want to leverage the programmability of high-speed networks to develop a set of smart remote memory operations that extend the RDMA protocol.
(2) Second, based on these smart remote memory operations, we will revisit the design of core components of distributed DBMS for OLAP and OLTP.
(3) Finally, we will evaluate the end-to-end efficiency of our new techniques by running benchmarks such as TPC-C or TPC-H.
It is important to note that our approach to offload smart remote memory operations to the network is very different from other recent approaches that offload more coarse-grained operations such as complete DBMS operators (e.g., joins) to the network.
Different from these coarse-grained offloading approaches, the smart remote memory operations are more fine-grained and thus will allow us not only
(1) to reuse the same remote memory primitives for various data management components but also (2) to better reason about which of the remote operations to offload when to best leverage the available network resources.