FIELD

Examples of the present disclosure relate generally to data management, and, more particularly, to systems and methods of creation and execution of data quality management tests based on profile discovery and data profiling.

BACKGROUND

The growth of electronic systems has created an exponential growth of data use and database management issues and concerns. For example, large enterprises can easily maintain hundreds of thousands of data structures, with tens of thousands of large data accesses and transfers daily. In the related art, data quality management is a highly work-intensive and manual process that considers numerous factors including, for example, data domain, business function, criticality, volume, update and access frequency, movement type, source system, data use, performance requirements, technical constraints, and attribute and table considerations. Each time a company creates a new data structure or data process, properly managing the new data structure requires individual decisions about the quality level required, data quality tests must be creation and customization, and then the data quality tests must be properly executed and potential data quality issues resolved. Such processes are labor-intensive and prone to error, both in test creation and selection.

Applying all possible data checks to all data creates enormous overhead. Applying hundreds or thousands of data quality tests to each piece of data creates unnecessary overhead that can render database systems functionally inoperable. Furthermore, even applying just high level data quality checks on the data can create unreasonable overhead that greatly reduces the ability to access the database, and transmit and manipulate data. Data quality errors, whether causing delays, inaccuracy, or lack of verification, create residual negative effects and computational bottlenecks.

Accordingly, there is a need for improved systems and methods for applying, executing, and updating data quality controls. More specifically, a need exists for providing data quality control analysis, testing, and monitoring to improve the functionality of data and networked systems. Aspects of the present disclosure are related to these and other concerns.

