DGCC protocol has been shown to achieve good performance on multi-core in-memory system. However, distributed transactions complicate the dependency resolution, and therefore, an effective transaction partitioning strategy is essential to reduce expensive multi-node distributed transactions. During failure recovery, log must be examined from the last checkpoint onwards and the affected transactions are re-executed based on the way they are partitioned and executed. Existing approaches treat both transaction management and recovery as two separate problems, even though recovery is dependent on the sequence in which transactions are executed. In this paper, we propose to treat the transaction management and recovery problems as one. We first propose an efficient Distributed Dependency Graph based Concurrency Control (DistDGCC) protocol for handling transactions spanning multiple nodes, and propose a new novel and efficient logging protocol called Dependency Logging that also makes use of dependency graphs for efficient logging and recovery. DistDGCC optimizes the average cost for each distributed transaction by processing transactions in batch. Moreover, it also reduces the effects of thread blocking caused by distributed transactions and consequently improves the runtime performance. Further, dependency logging exploits the same data structure that is used by DistDGCC to reduce the logging overhead, as well as the logical dependency information to improve the recovery parallelism. Extensive experiments are conducted to evaluate the performance of our proposed technique against state-of-the-art techniques. Experimental results show that DistDGCC is efficient and scalable, and dependency logging supports fast recovery with marginal runtime overhead. Hence, the overall system performance is significantly improved as a result.
ARIES @cite_16 is the most widely used logging approach in traditional database systems. By maintaining data in memory, new recovery techniques @cite_20 @cite_39 @cite_4 @cite_29 have been put forward, most of which inherits the idea from ARIES. Logical logging techniques @cite_1 @cite_33 are proposed recently that aim to reduce the log size. While they improve the runtime performance by reducing the number of disk I Os to an extent, they usually incur expensive cost for recovery, especially in distributed environment @cite_21 . Many optimizations are also proposed to increase the efficiency of logging and recovery. @cite_5 makes use of shadow pages to reduce the log size during the runtime. @cite_47 reduces the lock contention on the log buffers and the effects of context switching to improve the logging performance.
@cite_4 @cite_33 @cite_29 @cite_21 @cite_1 @cite_39 @cite_5 @cite_47 @cite_16 @cite_20
2145170453 2431277952 2071414195 2102712231 2118995439 2104954161 2118411843
New hardware platforms, e.g. cloud, multi-core, etc., have led to a reconsideration of database system architecture. Our Deuteronomy project separates transactional functionality from data management functionality, enabling a flexible response to exploiting new platforms. This separation requires, however, that recovery is described logically. In this paper, we extend current recovery methods to work in this logical setting. While this is straightforward in principle, performance is an issue. We show how ARIES style recovery optimizations can work for logical recovery where page information is not captured on the log. In side-by-side performance experiments using a common log, we compare logical recovery with a state-of-the art ARIES style recovery implementation and show that logical redo performance can be competitive. By maintaining the data in main memory, in-memory databases dramatically reduce the I O cost of transaction processing. However, for recovery purposes, in-memory systems still need to flush the log to disk, which incurs a substantial number of I Os. Recently, command logging has been proposed to replace the traditional data log (e.g., ARIES logging) in in-memory databases. Instead of recording how the tuples are updated, command logging only tracks the transactions that are being executed, thereby effectively reducing the size of the log and improving the performance. However, when a failure occurs, all the transactions in the log after the last checkpoint must be redone sequentially and this significantly increases the cost of recovery. In this paper, we first extend the command logging technique to a distributed system, where all the nodes can perform their recovery in parallel. We show that in a distributed system, the only bottleneck of recovery caused by command logging is the synchronization process that attempts to resolve the data dependency among the transactions. We then propose an adaptive logging approach by combining data logging and command logging. The percentage of data logging versus command logging becomes a tuning knob between the performance of transaction processing and recovery to meet different OLTP requirements, and a model is proposed to guide such tuning. Our experimental study compares the performance of our proposed adaptive logging, ARIES-style data logging and command logging on top of H-Store. The results show that adaptive logging can achieve a 10x boost for recovery and a transaction throughput that is comparable to that of command logging. Fine-grained, record-oriented write-ahead logging, as exemplified by systems like ARIES, has been the gold standard for relational database recovery. In this paper, we show that in modern high-throughput transaction processing systems, this is no longer the optimal way to recover a database system. In particular, as transaction throughputs get higher, ARIES-style logging starts to represent a non-trivial fraction of the overall transaction execution time. The impact of updating policy and access pattern on the performance of post-crash log processing with a fuzzy checkpointing main memory database (MMDB) is discussed. The problem of restoring the database to a consistent state and several algorithms for post-crash log processing under the various updating alternatives are reviewed. Using an analytic model, the checkpoint behavior and post-crash log processing performance of these algorithms are examined. Analytic results show that deferred updating always takes less time to process the log after a crash. > The shift to multi-core hardware brings new challenges to database systems, as the software parallelism determines performance. Even though database systems traditionally accommodate simultaneous requests, a multitude of synchronization barriers serialize execution. Write-ahead logging is a fundamental, omnipresent component in ARIES-style concurrency and recovery, and one of the most important yet-to-be addressed potential bottlenecks, especially in OLTP workloads making frequent small changes to data. In this paper, we identify four logging-related impediments to database system scalability. Each issue challenges different level in the software architecture: (a) the high volume of small-sized I O requests may saturate the disk, (b) transactions hold locks while waiting for the log flush, (c) extensive context switching overwhelms the OS scheduler with threads executing log I Os, and (d) contention appears as transactions serialize accesses to in-memory log data structures. We demonstrate these problems and address them with techniques that, when combined, comprise a holistic, scalable approach to logging. Our solution achieves a 20 -69 speedup over a modern database system when running log-intensive workloads, such as the TPC-B and TATP benchmarks. Moreover, it achieves log insert throughput over 1.8GB s for small log records on a single socket server, an order of magnitude higher than the traditional way of accessing the log using a single mutex. DB2 TM , IMS, and Tandem TM systems. ARIES is applicable not only to database management systems but also to persistent object-oriented languages, recoverable file systems and transaction-based operating systems. ARIES has been implemented, to varying degrees, in IBM's OS 2 TM Extended Edition Database Manager, DB2, Workstation Data Save Facility VM, Starburst and QuickSilver, and in the University of Wisconsin's EXODUS and Gamma database machine. Performance needs of many database applications dictate that the entire database be stored in main memory. The Dali system is a main memory storage manager designed to provide the persistence, availability and safety guarantees one typically expects from a diskresident database, while at the same time providing very high performance by virtue of being tuned to support in-memory data. Dali follows the philosophy of treating all data, including system data, uniformly as database files that can be memory mapped and directly accessed updated by user processes. Direct access provides high performance; slower, but more secure, access is also provided through the use of a server process. Various features of Dali can be tailored to the needs of an application to achieve high performance - for example, concurrency control and logging can be turned off if not desired, which enables Dali to efficiently support applications that require non-persistent memory resident data to be shared by multiple processes. Both objectoriented and relational databases can be implemented on top of Dali.