For the architecture community, reasonable simulation time is a strong requirement in addition to performance data accuracy. However, emerging big data and AI workloads are too huge at binary size level and prohibitively expensive to run on cycle-accurate simulators. The concept of data motif, which is identified as a class of units of computation performed on initial or intermediate data, is the first step towards building proxy benchmark to mimic the real-world big data and AI workloads. However, there is no practical way to construct a proxy benchmark based on the data motifs to help simulation-based research. In this paper, we embark on a study to bridge the gap between data motif and a practical proxy benchmark. We propose a data motif-based proxy benchmark generating methodology by means of machine learning method, which combine data motifs with different weights to mimic the big data and AI workloads. Furthermore, we implement various data motifs using light-weight stacks and apply the methodology to five real-world workloads to construct a suite of proxy benchmarks, considering the data types, patterns, and distributions. The evaluation results show that our proxy benchmarks shorten the execution time by 100s times on real systems while maintaining the average system and micro-architecture performance data accuracy above 90 , even changing the input data sets or cluster configurations. Moreover, the generated proxy benchmarks reflect consistent performance trends across different architectures. To facilitate the community, we will release the proxy benchmarks on the project homepage this http URL
In terms of micro-architectural simulation, many previous studies generate synthetic benchmarks as proxies @cite_37 @cite_6 . Statistical simulation @cite_9 @cite_58 @cite_46 @cite_22 @cite_13 @cite_31 generates synthetic trace or synthetic benchmarks to mimic micro-architectural performance of long-running real-world workloads, which targets one workload on a specific architecture with the certain configurations, and thus each benchmark needs to be generated on the other architectures with different configurations @cite_1 . Sampled simulation selects a series of sample units for simulation instead of entire instruction stream, which are sampled randomly @cite_53 , periodically @cite_50 @cite_26 or based on phase behavior @cite_8 . @cite_27 accelerate the full-system simulation through characterizing and predicting the performance behavior of OS services. For emerging big data workloads, PerfProx @cite_16 proposes a proxy benchmark generation framework for real-world database applications through characterizing low-level dynamic execution characteristics. The proxy benchmark of PerfProx also needs to be regenerated under different configurations.
@cite_37 @cite_31 @cite_26 @cite_22 @cite_8 @cite_9 @cite_53 @cite_1 @cite_6 @cite_16 @cite_27 @cite_50 @cite_46 @cite_58 @cite_13
2736232888 1804189410 2109814389 2131667412
Early design space evaluation of computer systems is usually performed using performance models (e.g., detailed simulators, RTL-based models, etc.). However, it is very challenging (often impossible) to run many emerging applications on detailed performance models owing to their complex software-stacks and long run times. To overcome such challenges in benchmarking these complex applications, we propose a proxy generation methodology, PerfProx to generate miniature proxy benchmarks, which are representative of the performance of real-world applications and yet, converge to results quickly and do not need any complex software-stack support. Past proxy generation research utilizes detailed micro-architecture independent metrics derived from detailed simulators, which are often difficult to generate for many emerging applications. PerfProx enables fast and efficient proxy generation using performance metrics derived primarily from hardware performance counters. We evaluate the proxy generation framework on three modern databases (Cassandra, MongoDB and MySQL) running data-serving and data-analytics applications. The proxy benchmarks mimic the performance (IPC) of the original applications with ∼94 accuracy, while significantly reducing the instruction count. Benchmarks set standards for innovation in computer architecture research and industry product development. Consequently, it is of paramount importance that the benchmarks used in computer architecture research and development are representative of real-world applications. However, composing such representative workloads poses practical challenges to application analysis teams and benchmark developers - (1) Benchmarks that are standardized are open-source whereas applications of interest are typically proprietary, (2) Benchmarks are rigid, measure single-point performance, and only represent a sample of the application behavior space, (3) Benchmark suites take several years to develop, but applications evolve at a faster rate, and (4) Benchmarks geared towards temperature and power characterization are difficult to develop and standardize. The objective of this dissertation is to develop an adaptive benchmark generation strategy to construct synthetic benchmarks to address these benchmarking challenges. We propose an approach for automatically distilling key hardware-independent performance attributes of a proprietary workload and capture them into a miniature synthetic benchmark clone. The advantage of the benchmark clone is that it hides the functional meaning of the code, but exhibits similar performance and power characteristics as the target application across a wide range of microarchitecture configurations. Moreover, the dynamic instruction count of the synthetic benchmark clone is substantially shorter than the proprietary application, greatly reducing overall simulation time—for the SPEC CPU 2000 suite, the simulation time reduction is over five orders of magnitude compared to the entire benchmark execution. We develop an adaptive benchmark generation strategy that trades off accuracy to provide the flexibility to easily alter program characteristics. The parameterization of workload metrics makes it possible to succinctly describe an application's behavior using a limited number of fundamental program characteristics. This provides the ability to alter workload characteristics and construct scalable benchmarks that allows researchers to explore a wider range of the application behavior space, conduct program behavior studies, and model emerging workloads. The parameterized workload model is the foundation for automatically constructing power and temperature oriented synthetic workloads. We show that machine learning algorithms can be effectively used to search the application behavior space to automatically construct benchmarks for evaluating the power and temperature characteristics of a computer architecture design. The need for a scientific approach to construct synthetic benchmarks, to complement application benchmarks, has long been recognized by the computer architecture research community, and this dissertation work is a significant step towards achieving that goal. Most research in the area of microarchitectural performance analysis is done using trace-driven simulations. Although trace-driven simulations are fairly accurate, they are both time- and space-consuming which makes them sometimes impractical. Modeling the execution of a computer program by a statistical profile and generating a synthetic benchmark trace from this statistical profile can be used to accelerate the design process. Thanks to the statistical nature of this technique, performance characteristics quickly converge to a steady state solution during simulation, which makes this technique suitable for fast design space explorations. In this paper, it is shown how more detailed statistical profiles can be obtained and how the synthetic trace generation mechanism should be designed to generate syntactically correct benchmark traces. As a result, the performance predictions in this paper are far more accurate than those reported in previous research. Computer architects must determine how tomost effectively use finite computational resources whenrunning simulations to evaluate new architectural ideas.To facilitate efficient simulations with a range of benchmarkprograms, rn have developed the MinneSPEC inputset for the SPEC CPU 2000 benchmark suite. Thisnew workload allows computer architects to obtain simulationresults in a reasonable time using existing sirnulators.While the MinneSPEC workload is derived from thestandard SPEC CPU 2000 warklcad, it is a valid benchmarksuite in and of itself for simulation-based research.MinneSPEC also may be used to run Iarge numbers ofsimulations to find "sweet spots" in the evaluation parameterspace. This small number of promising designpoints subsequently may be investigated in more detailwith the full SPEC reference workload. In the processof developing the MinneSPEC datasets, we quantify itsdifferences in terms of function-level execution patterns,instruction mixes, and memory behaviors compared tothe SPEC programs when executed with the reference inputs.We find that for some programs, the MinneSPECprofiles match the SPEC reference dataset program behaviorvery closely. For other programs, however, theMinneSPEC inputs produce significantly different programbehavior. The MinneSPEC workload has been recognizedby SPEC and is distributed with Version 1.2 andhigher of the SPEC CPU 2000 benchmark suite.