We present a distributed system for storage, processing, three-dimensional visualisation and basic analysis of data from Earth-observing satellites. The database and the server have been designed for high performance and scalability, whereas the client is highly portable thanks to having been designed as a HTML5- and WebGL-based Web application. The system is based on the so-called MEAN stack, a modern replacement for LAMP which has steadily been gaining traction among high-performance Web applications. We demonstrate the performance of the system from the perspective of an user operating the client.
Finally, the German Satellite Data Archive (D-SDA) of the Earth Observation Center at the German Aerospace Center (DLR) @cite_3 at a glance seems to serve the same purpose as our system yet is backed by considerably more resources. There are, however, differences: it is tied to DLR infrastructure, keeps both data and metadata as files in storage rather than in a database, and most importantly follows considerably different philosophy --- its primary goal is to achieve long-term data preservation over more than 20 years with nearly exponentially growing data capacity, whereas our archive has been designed primarily to provide high-performance access to a fixed chunk of data. One could therefore imagine the two systems as complementary --- ours for rapid application-specific processing, D-SDA for long-term storage.
@cite_3
97315372 2137133993 2156719566 1751683294
The Earth Observation Center (EOC) at the German Aerospace Center (DLR) is the center of competence in Germany, providing expertise in earth observation research and development activities, as well as operational tasks for data reception, processing and archiving. We briefly present the German Satellite Data Archive (D-SDA) with systems and activities that make it possible to accomplish successful Long Term Data Preservation (LTDP) over more than 20 years with nearly exponentially growing data capacity. Tables, facts and figures exhibit the path taken at DLR to successfully establish the infrastructure for long term data preservation for Germany and Europe. Our experience is that that LTDP is manageable but requires following the evolution in technology, new usage scenarios and data policies. DLR is working on the continuous evolution of the flexible DIMS components, infrastructure and management services to serve the processing and archiving centers for the next generation earth observation satellites. Archiving is important for scientific data, where it is necessary to record all past versions of a database in order to verify findings based upon a specific version. Much scientific data is held in a hierachical format and has a key structure that provides a canonical identification for each element of the hierarchy. In this article, we exploit these properties to develop an archiving technique that is both efficient in its use of space and preserves the continuity of elements through versions of the database, something that is not provided by traditional minimum-edit-distance diff approaches. The approach also uses timestamps. All versions of the data are merged into one hierarchy where an element appearing in multiple versions is stored only once along with a timestamp. By identifying the semantic continuity of elements and merging them into one data structure, our technique is capable of providing meaningful change descriptions, the archive allows us to easily answer certain temporal queries such as retrieval of any specific version from the archive and finding the history of an element. This is in contrast with approaches that store a sequence of deltas where such operations may require undoing a large number of changes or significant reasoning with the deltas. A suite of experiments also demonstrates that our archive does not incur any significant space overhead when contrasted with diff approaches. Another useful property of our approach is that we use XML format to represent hierarchical data and the resulting archive is also in XML. Hence, XML tools can be directly applied on our archive. In particular, we apply an XML compressor on our archive, and our experiments show that our compressed archive outperforms compressed diff-based repositories in space efficiency. We also show how we can extend our archiving tool to an external memory archiver for higher scalability and describe various index structures that can further improve the efficiency of some temporal queries on our archive. We present the Deep Store archival storage architecture, a large-scale storage system that stores immutable data efficiently and reliably for long periods of time. Archived data is stored across a cluster of nodes and recorded to hard disk. The design differentiates itself from traditional file systems by eliminating redundancy within and across files, distributing content for scalability, associating rich metadata with content, and using variable levels of replication based on the importance or degree of dependency of each piece of stored data. We evaluate the foundations of our design, including PRESIDIO, a virtual content-addressable storage framework with multiple methods for interfile and intra-file compression that effectively addresses the data-dependent variability of data compression. We measure content and metadata storage efficiency, demonstrate the need for a variable-degree replication model, and provide preliminary results for storage performance. Existing file systems, even the most scalable systems that store hundreds of petabytes (or more) of data across thousands of machines, store file metadata on a single server or via a shared-disk architecture in order to ensure consistency and validity of the metadata. This paper describes a completely different approach for the design of replicated, scalable file systems, which leverages a high-throughput distributed database system for metadata management. This results in improved scalability of the metadata layer of the file system, as file metadata can be partitioned (and replicated) across a (shared-nothing) cluster of independent servers, and operations on file metadata transformed into distributed transactions. In addition, our file system is able to support standard file system semantics--including fully linearizable random writes by concurrent users to arbitrary byte offsets within the same file--across wide geographic areas. Such high performance, fully consistent, geographically distributed files systems do not exist today. We demonstrate that our approach to file system design can scale to billions of files and handle hundreds of thousands of updates and millions of reads per second-- while maintaining consistently low read latencies. Furthermore, such a deployment can survive entire datacenter outages with only small performance hiccups and no loss of availability.