FIELD

The present disclosure relates generally to data lineage and more particularly to systems and methods for dynamically rendering data lineage to efficiently and interactively search, retrieve, and evaluate the data lineage.

BACKGROUND

The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

Lineage of data (data lineage) shows where the data originates upstream, how the data moves and is modified through its life cycle, and which other data uses it downstream. Data lineage describes what happens to the data as the data goes through diverse processes through its life cycle. Data lineage helps provide visibility into analytics pipeline and simplifies tracing errors back to their sources. Data lineage also enables replaying portions of dataflow for step-wise debugging or regeneration of data.

Data lineage provides a visual representation that helps in discovering and exploring flow/movement of data from its source to destination and various changes it undergoes during its life cycle in an enterprise. Data lineage represents how the data hops between various data points, how the data gets transformed along the way, how the representation and parameters of the data change, and how the data splits or converges after each hop.

Data lineage representations can be useful in many ways. For example, big data analytics can use data lineage representations. Big data analytics is a process of examining large data sets to uncover hidden patterns, unknown correlations, market trends, customer preferences, and other useful business information. For example, the big data analytics may apply machine learning algorithms to the data which transform the data. Due to the sheer size of the data, there could be unknown features in the data including outliers, and it can be difficult for data scientists to debug unexpected results.

Further, the massive scale and unstructured nature of data, the complexity of these analytics pipelines, and long runtimes pose significant manageability and debugging challenges. A single error in these analytics can be difficult to identify and remove. While one may debug errors by re-running the entire analytics through a debugger for step-wise debugging, such a process can be expensive due to the amount of time and resources needed. Auditing and data validation are other major problems due to the growing ease of access to relevant data sources for use in experiments, sharing of data between scientific communities, and use of third-party data in business enterprises. In today's competitive business environment, companies not only have to find and analyze the relevant data they need, but they also need to find it quickly. The challenge is processing the sheer volume of data and accessing the level of detail needed at a high speed. The challenge grows as the degree of granularity increases.

Big data platforms have a complicated structure. Data are distributed among several systems. Debugging of a big data pipeline becomes challenging because of the very nature of the system. It is not an easy task for a data scientist to figure out which system's data has outliers and unknown features causing a particular algorithm to give unexpected results.

Data lineage can be used to make the debugging of big data pipeline easier. This necessitates collection of data about data transformations. Data lineage provides a historical record of the data and its origins. The lineage of data which is generated by complex transformations such as workflows can be valuable to data scientists. From data lineage, one can ascertain the quality of the data based on its ancestral data and derivations, track back sources of errors, allow automated re-enactment of derivations to update data, and provide attribution of data sources, for example. Data lineage is also useful in the business domain where it can be used, for example, to drill down to the source of data in a data warehouse, track the creation of intellectual property, and provide an audit trail for regulatory purposes.

SUMMARY

A system comprises a processor and machine readable instructions stored on a tangible machine readable medium. When executed by the processor, the machine readable instructions configure the system to process data associated with lineage of a selected data asset, the processed data indicating relationship of the selected data asset with data assets upstream or downstream from the selected data asset. The machine readable instructions configure the system to generate, based on the processed data, a first set of data assets for displaying at a first position relative to the selected data asset, the first set of data assets being one level upstream or downstream from the selected data asset, and a first data asset in the first set of data assets to be displayed at a selected position relative to the selected data asset. The machine readable instructions configure the system to generate, based on the processed data, a second set of data assets for displaying at a second position relative to the selected data asset, the second set of data assets being one level upstream or downstream from the first data asset. In response to a second data asset from the first set of data assets being selected, the machine readable instructions configure the system to reposition the second data asset for display at the selected position; and to replace, for display, the second set of data assets with a third set of data assets that are one level upstream or downstream from the second data asset.

In other features, the machine readable instructions further configure the system to capture, from the lineage data including a network of relationships of data assets, information about data assets that are upstream and downstream from the selected data asset, and information about relationship of the selected data asset to the data assets upstream and downstream from the selected data asset. The machine readable instructions further configure the system to generate, from the captured information, number of upstream and downstream relationships of the selected data asset. The machine readable instructions further configure the system to identify, based on the number of upstream and downstream relationships of the selected data asset, the information associated with lineage of the selected data asset, the information including the one or more data assets contributing to the second data asset.

In other features, in the first set of data assets the first data asset is used most to search a catalog of data assets upstream or downstream from the selected data asset; and remaining data assets are arranged in opposite directions from the first data asset in a decreasing order of use of the remaining data assets to search the catalog of data assets.

In other features, the selected data asset identifies the first set of data assets as being one level upstream or downstream from the selected data asset, the first data asset identifies the second set of data assets as being one level upstream or downstream from the first data asset, and the second data asset identifies the third set of data assets as being one level upstream or downstream from the second data asset.

In other features, the data associated with the lineage of the selected data asset includes data from one or more computer programs developed by one or more software vendors.

In other features, in response to access to a data asset in the one or more levels being restricted due to security, the machine readable instructions further configure the system to generate, for displaying, based on the processed data, an indication that access to the data asset is restricted due to security.

In other features, the machine readable instructions further configure the system to generate, for displaying, based on the processed data, a link between the selected data asset and one of the data assets at the one or more levels upstream or downstream from the selected data asset. The machine readable instructions further configure the system to generate, for displaying, based on the processed data, additional information about the lineage of the selected data asset in response to a cursor being positioned over the link.

In other features, the additional information includes identification of a computer program associated with transforming the data assets linked by the link.

In other features, the additional information includes a hyperlink for providing further information about the lineage of the selected data asset, and the machine readable instructions further configure the system to generate, for displaying, based on the processed data, the further information in response to the hyperlink being selected.

In still other features, a method executed by a data processing system for analyzing lineage data comprises processing, by the data processing system, data associated with lineage of a selected data asset, the processed data indicating relationship of the selected data asset with data assets upstream or downstream from the selected data asset. The method further comprises generating, by the data processing system, based on the processed data, a first set of data assets for displaying at a first position relative to the selected data asset, the first set of data assets being one level upstream or downstream from the selected data asset, and a first data asset in the first set of data assets to be displayed at a selected position relative to the selected data asset. The method further comprises generating, by the data processing system, based on the processed data, a second set of data assets for displaying at a second position relative to the selected data asset, the second set of data assets being one level upstream or downstream from the first data asset. The method further comprises, in response to a second data asset from the first set of data assets being selected, repositioning, by the data processing system, the second data asset for display at the selected position; and replacing, by the data processing system, for display, the second set of data assets with a third set of data assets that are one level upstream or downstream from the second data asset.

In other features, the method further comprises, capturing, by the data processing system, from the lineage data including a network of relationships of data assets, information about data assets that are upstream and downstream from the selected data asset, and information about relationship of the selected data asset to the data assets upstream and downstream from the selected data asset. The method further comprises, generating, by the data processing system, from the captured information, number of upstream and downstream relationships of the selected data asset. The method further comprises, identifying, by the data processing system, based on the number of upstream and downstream relationships of the selected data asset, the information associated with lineage of the selected data asset, the information including the one or more data assets contributing to the second data asset.

In other features, in the first set of data assets, the first data asset is used most to search a catalog of data assets upstream or downstream from the selected data asset, and the method further comprises arranging, by the data processing system, remaining data assets in the first set of data assets in opposite directions from the first data asset in a decreasing order of use of the remaining data assets to search the catalog of data assets.

In other features, the method further comprises identifying, by the data processing system, the first set of data assets as being one level upstream or downstream from the selected data asset; the second set of data assets as being one level upstream or downstream from the first data asset; and the third set of data assets as being one level upstream or downstream from the second data asset.

In other features, the method further comprises, in response to access to a data asset in the one or more levels being restricted due to security, generating, by the data processing system, for displaying, based on the processed data, an indication that access to the data asset is restricted due to security.

In other features, the method further comprises generating, for displaying, based on the processed data, by the data processing system, a link between the selected data asset and one of the data assets at the one or more levels upstream or downstream from the selected data asset. The method further comprises generating, for displaying, based on the processed data, by the data processing system, additional information about the lineage of the selected data asset in response to a cursor being positioned over the link.

In other features, the additional information includes identification of a computer program associated with transforming the data assets linked by the link.

In other features, the additional information includes a hyperlink for providing further information about the lineage of the selected data asset in response to the hyperlink being selected, and the method further comprises generating, for displaying, based on the processed data, by the data processing system, the further information in response to the hyperlink being selected.

In still other features, a system comprises a processor and machine readable instructions stored on a tangible machine readable medium. When executed by the processor, the machine readable instructions configure the system to generate, based on lineage data, information for displaying lineage of a first data asset, the information including one or more data assets contributing to the first data asset. The machine readable instructions configure the system to receive an input representing selection of a second data asset. The machine readable instructions configure the system to, in response to receiving the input, identify information associated with lineage of the second data asset; reposition the second data asset to a different position; and convey, for displaying, one or more data assets contributing to the second data asset.

In other features, the machine readable instructions further configure the system to reposition the second data asset by replacing the first data asset with the second data asset.

In other features, the machine readable instructions further configure the system to capture, from the lineage data including a network of relationships of data assets, information about data assets that are upstream and downstream from the second data asset, and information about relationship of the second data asset to the data assets upstream and downstream from the second data asset. The machine readable instructions further configure the system to generate, from the captured information, number of upstream and downstream relationships of the second data asset. The machine readable instructions further configure the system to identify, in response to receiving the input, based on the number of upstream and downstream relationships of the second data asset, the information associated with lineage of the second data asset, the information including the one or more data assets contributing to the second data asset.

In still other features, a system comprises a processor and machine readable instructions stored on a tangible machine readable medium. When executed by the processor, the machine readable instructions configure the system to process data associated with lineage of a selected data asset, the processed data indicating relationship of the selected data asset with data assets upstream or downstream from the selected data asset. The machine readable instructions configure the system to generate, for displaying, based on the processed data, summaries of the selected data asset and data assets at one or more levels upstream or downstream from the selected data asset, the summaries indicating selectable paths upstream or downstream from the selected data asset for searching the lineage of the selected data asset. A summary of a data asset includes an identifier of the data asset, a number of relationships of the data asset with data assets one level upstream, and a number of relationships of the data asset with data assets one level downstream from the data asset.

In other features, the machine readable instructions further configure the system to inspect information stored in a data asset catalog regarding data assets and instruments used to transform the data assets. The machine readable instructions further configure the system to capture information about data assets that are transformed by the instruments and information about the instruments that transformed the data assets. The machine readable instructions further configure the system to track, based on the captured information, relationships of data assets with other data assets and with the instruments used to transform the data assets. The machine readable instructions further configure the system to generate, based on the tracking, a network of relationships of data assets with other data assets and with the instruments used to transform the data assets. The machine readable instructions further configure the system to generate the summaries from the network of relationships.

In other features, the summaries of the selected data asset and the data assets at the one or more levels upstream or downstream from the selected data asset are generated for displaying a portion of the lineage of the selected data asset.

In other features, the machine readable instructions further configure the system to generate, based on the processed data, summaries of a first set of data assets for displaying at a first position relative to the summary of the selected data asset, the first set of data assets being at a selected level from the one or more levels; summary of a first data asset in the first set of data assets to be displayed at a selected position relative to the summary of the selected data asset, the first data asset being used most to search a catalog of data assets upstream or downstream from the selected data asset; and summaries of remaining data assets in the first set of data assets being arranged in opposite directions from the first data asset in a decreasing order of use of the remaining data assets to search the catalog of data assets.

In other features, the machine readable instructions further configure the system to generate, based on the processed data, for displaying at a first position relative to the summary of the selected data asset, summaries of a first set of data assets identified as being one level upstream or downstream in the summary of the selected data asset. The machine readable instructions further configure the system to generate, based on the processed data, for displaying at a selected position relative to the summary of the selected data asset, a summary of a first data asset in the first set of data assets. The machine readable instructions further configure the system to generate, based on the processed data, for displaying at a second position relative to the summaries of the first set of data assets, summaries of a second set of data assets identified as being one level upstream or downstream in the summary of the first data asset.

In other features, the machine readable instructions further configure the system to generate, based on the processed data, summaries of a first set of data assets for displaying at a first position relative to the summary of the selected data asset, a summary of a first data asset in the first set of data assets to be displayed at a selected position relative to the summary of the selected data asset. The machine readable instructions further configure the system to generate, based on the processed data, for displaying at a second position relative to the summaries of the first set of data assets, summaries of a second set of data assets identified as being one level upstream or downstream in the summary of the first data asset. The machine readable instructions further configure the system to, in response to a summary of a second data asset from the first set of data assets being selected, reposition the summary of the second data asset for display at the selected position; and replace, for display, the summaries of the second set of data assets with summaries of a third set of data assets identified as being one level upstream or downstream in the summary of the second data asset.

In other features, the machine readable instructions further configure the system to generate, for displaying, based on the processed data, summaries of data assets at an additional level upstream or downstream from the selected data asset by repositioning the summaries of data assets from the one or more levels.

In other features, the data associated with the lineage of the selected data asset includes data from one or more computer programs developed by one or more software vendors.

In other features, in response to access to a data asset in the one or more levels being restricted due to security, the machine readable instructions further configure the system to generate, for displaying, based on the processed data, a summary of the data asset, the summary of the data asset including an indication that access to the data asset is restricted due to security.

In other features, the machine readable instructions further configure the system to generate, for displaying, based on the processed data, a link between the summary of the selected data asset and a summary of one of the data assets at the one or more levels upstream or downstream from the selected data asset. The machine readable instructions further configure the system to generate, for displaying, based on the processed data, additional information about the lineage of the selected data asset in response to a cursor being positioned over the link.

In other features, the additional information includes identification of a computer program associated with transforming the data assets whose summaries are linked by the link.

In other features, the additional information includes a hyperlink for providing further information about the lineage of the selected data asset, and the machine readable instructions further configure the system to generate, for displaying, based on the processed data, the further information in response to the hyperlink being selected.

In still other features, a method executed by a data processing system for analyzing lineage data comprises processing, by the data processing system, data associated with lineage of a selected data asset, the processed data indicating relationship of the selected data asset with data assets upstream or downstream from the selected data asset. The method further comprises generating, by the data processing system, for displaying, based on the processed data, summaries of the selected data asset and data assets at one or more levels upstream or downstream from the selected data asset, the summaries indicating selectable paths upstream or downstream from the selected data asset for searching the lineage of the selected data asset. A summary of a data asset includes an identifier of the data asset, a number of relationships of the data asset with data assets one level upstream, and a number of relationships of the data asset with data assets one level downstream from the data asset.

In other features, the method further comprises inspecting information stored in a data asset catalog regarding data assets and instruments used to transform the data assets. The method further comprises capturing information about data assets that are transformed by the instruments and information about the instruments that transformed the data assets. The method further comprises tracking, based on the captured information, relationships of data assets with other data assets and with the instruments used to transform the data assets. The method further comprises generating, based on the tracking, a network of relationships of data assets with other data assets and with the instruments used to transform the data assets. The method further comprises generating, the summaries from the network of relationships.

In other features, the method further comprises generating, based on the processed data, summaries of a first set of data assets for displaying at a first position relative to the summary of the selected data asset, the first set of data assets being at a selected level from the one or more levels. The method further comprises a summary of a first data asset in the first set of data assets to be displayed at a selected position relative to the summary of the selected data asset, the first data asset being used most to search a catalog of data assets upstream or downstream from the selected data asset. The method further comprises summaries of remaining data assets in the first set of data assets being arranged in opposite directions from the first data asset in a decreasing order of use of the remaining data assets to search the catalog of data assets.

In other features, the method further comprises generating, based on the processed data, for displaying at a first position relative to the summary of the selected data asset, summaries of a first set of data assets identified as being one level upstream or downstream in the summary of the selected data asset. The method further comprises generating, based on the processed data, for displaying at a selected position relative to the summary of the selected data asset, a summary of a first data asset in the first set of data assets. The method further comprises generating, based on the processed data, for displaying at a second position relative to the summaries of the first set of data assets, summaries of a second set of data assets identified as being one level upstream or downstream in the summary of the first data asset.

In other features, the method further comprises generating, based on the processed data, summaries of a first set of data assets for displaying at a first position relative to the summary of the selected data asset, a summary of a first data asset in the first set of data assets to be displayed at a selected position relative to the summary of the selected data asset. The method further comprises generating, based on the processed data, for displaying at a second position relative to the summaries of the first set of data assets, summaries of a second set of data assets identified as being one level upstream or downstream in the summary of the first data asset. The method further comprises, in response to a summary of a second data asset from the first set of data assets being selected, repositioning the summary of the second data asset for display at the selected position; and replacing, for display, the summaries of the second set of data assets with summaries of a third set of data assets identified as being one level upstream or downstream in the summary of the second data asset.

In other features, the method further comprises generating, for displaying, by the data processing system, based on the processed data, summaries of data assets at an additional level upstream or downstream from the selected data asset by repositioning the summaries of data assets from the one or more levels.

In other features, the method further comprises generate, for displaying, based on the processed data, a link between the summary of the selected data asset and a summary of one of the data assets at the one or more levels upstream or downstream from the selected data asset. The method further comprises generate, for displaying, based on the processed data, additional information about the lineage of the selected data asset in response to a cursor being positioned over the link. The additional information includes identification of a computer program associated with transforming the data assets whose summaries are linked by the link, or the additional information includes a hyperlink for providing further information about the lineage of the selected data asset in response to the hyperlink being selected.

In still other features, a system for analyzing lineage data of a selected data asset comprises a processor and machine readable instructions stored on a tangible machine readable medium. When executed by the processor, the machine readable instructions configure the system to inspect information stored in a data asset catalog regarding data assets and instruments used to transform the data assets. The machine readable instructions configure the system to capture information about data assets that are transformed by the instruments and information about the instruments that transformed the data assets. The machine readable instructions configure the system to track, based on the captured information, relationships of data assets with other data assets and with the instruments used to transform the data assets. The machine readable instructions configure the system to generate, based on the tracking, a network of relationships of data assets with other data assets and with the instruments used to transform the data assets. The machine readable instructions configure the system to generate, for displaying, from the network of relationships, summaries of a selected data asset and data assets at one or more levels upstream or downstream from the selected data asset, the summaries indicating selectable paths upstream or downstream from the selected data asset for searching the lineage of the selected data asset. A summary of a data asset includes an identifier of the data asset and a number of relationships of the data asset with data assets one level upstream and a number of relationships of the data asset with data assets one level downstream from the data asset.

Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

