The LOFAR radio telescope creates Petabytes of data per year. This data is important for many scientific projects. The data needs to be efficiently processed within the timespan of these projects in order to maximize the scientific impact. We present a workflow orchestration system that integrates LOFAR processing with a distributed computing platform. The system is named Automated Grid-enabled LOFAR Workflows (AGLOW). AGLOW makes it fast and easy to develop, test and deploy complex LOFAR workflows, and to accelerate them on a distributed cluster architecture. AGLOW provides a significant reduction in time for setting up complex workflows: typically, from months to days. We lay out two case studies that process the data from the LOFAR Surveys Key Science Project. We have implemented these into the AGLOW environment. We also describe the capabilities of AGLOW, paving the way for use by other LOFAR science cases. In the future, AGLOW will automatically produce multiple science products from a single dataset, serving several of the LOFAR Key Science Projects.
A workflow is described by a set of tasks. The dependencies between these tasks are encoded in a Directed Acyclic Graph (DAG) @cite_16 . This data structure imposes a strict dependency hierarchy between the tasks @cite_21 . This means that there exists a well-defined execution order and a well-defined list of dependencies for each task. The execution order is typically determined by algorithms such as Kahn's algorithm @cite_3 or a depth-first search @cite_9 .
@cite_9 @cite_16 @cite_21 @cite_3
2134890818 1603699185 2084763084 2063123613
We study the problem of incrementally maintaining a topological sorting in a large DAG. The Discovery Algorithm (DA) of [Proc. 1st Annual ACM-SIAM Symp. on Discrete Algorithms, 1990, pp. 32-42] computes a cover K of nodes such that a solution to the modified problem can be found by changing node priorities within K only. It achieves a runtime complexity that is polynomially bounded in terms of the minimal cover size k.The temporary space complexity of DA grows quickly with increasing number of added edges and cover size. We introduce the Depth-First Discovery Algorithm (DFDA), which uses depth-first search to reduce the temporary space of DA from O(|A| × ||-K||) to O(|A| + ||K||), where |A| is the number of edges to add and ||K|| is the extended size of the cover. DFDA is simpler than DA and performs better in our empirical tests. We consider the problem of maintaining the topological order of a directed acyclic graph (DAG) in the presence of edge insertions and deletions. We present a new algorithm and, although this has inferior time complexity compared with the best previously known result, we find that its simplicity leads to better performance in practice. In addition, we provide an empirical comparison against the three main alternatives over a large number of random DAGs. The results show our algorithm is the best for sparse digraphs and only a constant factor slower than the best on dense digraphs. Topological Sorting is a procedure required for many problems involving analysis of networks. An example of one such problem is PERT. The present paper presents a very general method for obtaining topological order. It permits treatment of larger networks than can be handled on present procedures and achieves this with greater efficiency. Although the procedure can be adapted to any machine, it is discussed in terms of the 7090. A PERT network of 30,000 activities can be ordered in less than one hour of machine time. The method was developed as a byproduct of procedure needed by Westinghouse, Baltimore. It has not been programmed and at present there are no plans to implement it. In regard to the techniques described, Westinghouse's present and anticipated needs are completely served by the Lockheed program, which is in current use.