This paper describes an information system designed to support the large volume of monitoring information generated by a distributed testbed. This monitoring information is produced by several subsystems and consists of status and performance data that needs to be federated, distributed, and stored in a timely and easy to use manner. Our approach differs from existing approaches because it federates and distributes information at a low architectural level via messaging; a natural match to many of the producers and consumers of information. In addition, a database is easily layered atop the messaging layer for consumers that want to query and search the information. Finally, a common language to represent information in all layers of the information system makes it significantly easier for users to consume information. Performance data shows that this approach meets the significant needs of FutureGrid and would meet the needs of an experimental infrastructure twice the size of FutureGrid. In addition, this design also meets the needs of existing distributed scientific infrastructures.
A number of information systems have been proposed and deployed on scientific infrastructures. For example, early versions of the Globus software included a Metacomputing Directory Service (MDS) @cite_8 for storing information in a hierarchical directory service accessed via the Lightweight Directory Access Protocol (LDAP). The Globus project then transitioned to a MDS based on web service (SOAP WSDL) technologies @cite_33 .
@cite_33 @cite_8
2123289583 2139126523 2054510130 2161692763
The Globus Toolkit Monitoring and Discovery System (MDS4) defines and implements mechanisms for service and resource discovery and monitoring in distributed environments. MDS4 is distinguished from previous similar systems by its extensive use of interfaces and behaviors defined in the WS-Resource Framework and WS-Notification specifications, and by its deep integration into essentially every component of the Globus Toolkit. We describe the MDS4 architecture and the Web service interfaces and behaviors that allow users to discover resources and services, monitor resource and service states, receive updates on current status, and visualize monitoring results. We present two current deployments to provide insights into the functionality that can be achieved via the use of these mechanisms. High-performance execution in distributed computing environments often requires careful selection and configuration not only of computers, networks, and other resources but also of the protocols and algorithms used by applications. Selection and configuration in turn require access to accurate, up-to-date information on the structure and state of available resources. Unfortunately no standard mechanism exists for organizing or accessing such information. Consequently different tools and applications adopt ad hoc mechanisms, or they compromise their portability and performance by using default configurations. We propose a Metacomputing Directory Service that provides efficient and scalable access to diverse, dynamic, and distributed information about resource structure and state. We define an extensible data model to represent required information and present a scalable, high-performance, distributed implementation. The data representation and application programming interface are adopted from the Lightweight Directory Access Protocol; the data model and implementation are new. We use the Globus distributed computing toolkit to illustrate how this directory service enables the development of more flexible and efficient distributed computing services and applications. In order to reduce high construction cost and energy consumption of distributed base stations in cellular networks, the centralized radio access Cellular network infrastructure with a Super BS(CSBS) architecture has been proposed in which the centralized processing resource can be flexibly organized. However, there is sill the problem of low processing resource efficiency in protocol processing. In this paper, based on a protocol processing resource management framework, distributed load diversity based processing resource allocation (D-LDA) as well as the improved hybrid load diversity based processing resource allocation(H-LDA) are proposed, which take load discrepancy into consideration in the situation that a large number of BSs are supported. Simulation results show that D-LDA and H-LDA algorithm significantly outperforms the conventional method respectively by more than 30 and 55 in the processing resource efficiency. Scientific experiments and large-scale simulations produce massive amounts of data. Many of these scientific datasets are arrays, and are stored in file formats such as HDF5 and NetCDF. Although scientific data management systems, such as SciDB, are designed to manipulate arrays, there are challenges in integrating these systems into existing analysis workflows. Major barriers include the expensive task of preparing and loading data before querying, and converting the final results to a format that is understood by the existing post-processing and visualization tools. As a consequence, integrating a data management system into an existing scientific data analysis workflow is time-consuming and requires extensive user involvement. In this paper, we present the design of a new scientific data analysis system that efficiently processes queries directly over data stored in the HDF5 file format. This design choice eliminates the tedious and error-prone data loading process, and makes the query results readily available to the next processing steps of the analysis workflow. Our design leverages the increasing main memory capacities found in supercomputers through bitmap indexing and in-memory query execution. In addition, query processing over the HDF5 data format can be effortlessly parallelized to utilize the ample concurrency available in large-scale supercomputers and modern parallel file systems. We evaluate the performance of our system on a large supercomputing system and experiment with both a synthetic dataset and a real cosmology observation dataset. Our system frequently outperforms the relational database system that the cosmology team currently uses, and is more than 10X faster than Hive when processing data in parallel. Overall, by eliminating the data loading step, our query processing system is more effective in supporting in situ scientific analysis workflows.