Persistent key value stores are an important component of many distributed data serving solutions with innovations targeted at taking advantage of growing flash speeds. Unfortunately their performance is hampered by the need to maintain and replicate a write ahead log to guarantee availability in the face of machine and storage failures. Cyclone is a replicated log plug-in for key value stores that systematically addresses various sources of this bottleneck. It uses a small amount of non-volatile memory directly addressable by the CPU - such as in the form of NVDIMMs or Intel 3DXPoint - to remove block oriented IO devices such as SSDs from the critical path for appending to the log. This enables it to address network overheads using an implementation of the RAFT consensus protocol that is designed around a userspace network stack to relieve the CPU of the burden of data copies. Finally, it provides a way to efficiently map the commutativity in key-value store APIs to the parallelism available in commodity NICs. Cyclone is able to replicate millions of small updates per second using only commodity 10 gigabit ethernet adapters. As a practical application, we use it to improve the performance (and availability) of RocksDB, a popular persistent key value store by an order of magnitude when compared to its own write ahead log without replication.
Consensus in a box @cite_26 implements the Zookeeper atomic broadcast protocol together with a NIC on an FPGA. That work ignored durability, focusing purely on network performance for replication. Latency is excellent due to the fact that FPGA cuts out the path from CPU caches to the NIC. The authors report latency as low as 3us for a round trip from leader to follower replica, compared to the 6us we observe with DPDK. Notably however they reported 7us when using TCP IP from the FPGA rather than their own connection oriented network protocol. On the other hand their peak replication throughput is 4M replications sec on 66 Gbps of aggregate network bandwidth for small messages, comparable to the 6M replications sec mark obtained by Cyclone on an aggregate 40 Gbps of network bandwidth when . We believe the general applicability afforded by our software only approach together with the fact that we provide persistence by design makes it a compelling alternative to an FPGA based solution, even given the larger latency. Also, these results demonstrate that a general purpose CPU core can be as efficient as an FPGA in running a consensus protocol using the techniques described in this paper.
@cite_26
2303620077
Consensus mechanisms for ensuring consistency are some of the most expensive operations in managing large amounts of data. Often, there is a trade off that involves reducing the coordination overhead at the price of accepting possible data loss or inconsistencies. As the demand for more efficient data centers increases, it is important to provide better ways of ensuring consistency without affecting performance. In this paper we show that consensus (atomic broadcast) can be removed from the critical path of performance by moving it to hardware. As a proof of concept, we implement Zookeeper's atomic broadcast at the network level using an FPGA. Our design uses both TCP and an application specific network protocol. The design can be used to push more value into the network, e.g., by extending the functionality of middleboxes or adding inexpensive consensus to in-network processing nodes. To illustrate how this hardware consensus can be used in practical systems, we have combined it with a mainmemory key value store running on specialized microservers (built as well on FPGAs). This results in a distributed service similar to Zookeeper that exhibits high and stable performance. This work can be used as a blueprint for further specialized designs.