BACKGROUND

In the handling of large data sets (what is frequently referred to as “big data”), the work of preparing data sets for analysis and/or for presentation in reports and/or visualizations can consume more time and/or more processing resources than the work of either of the analyses or the generation of presentations. As the size and number of data sets continues to increase, the correspondingly increasing variety of uses for data sets brings about a growing variety of data preparation operations that may need to be performed and each data preparation operation takes ever longer to perform. As a result, bottlenecks may occur in the preparation of data sets that may greatly delay the availability of properly prepared data sets for subsequent analysis and/or presentation operations.

This has rendered such past practices as choosing to regularly perform a selected battery of data preparation operations on every data set, regardless of which data preparation operations are actually needed, increasingly unfeasible. The task of determining what data preparation operations actually need to be performed on each data set has become increasingly important.

Unfortunately, the increasing size of data sets also increases the difficulty in relying on personnel to manually select the data preparation operations that are to be performed on each data set. Manually inspecting even a large enough portion of a data set to identify what data preparation operations are needed becomes increasingly difficult and requires ever more time per data set. Additionally, the increasing variety of data preparation operations that may need to be performed to accommodate an increasing variety of uses for data sets can become overwhelming.

SUMMARY

This summary is not intended to identify only key or essential features of the described subject matter, nor is it intended to be used in isolation to determine the scope of the described subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

An apparatus includes at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including, receive, from a requesting device via a network, a request to provide a data catalog of at least one data set that meets one or more criteria specified in the request, wherein: the at least one data set is to be selected from multiple data sets; for each data set of the multiple data sets, there exists at least one of corresponding metadata indicative of at least one structural feature of the data set, or corresponding context data indicative of at least one contextual aspect of the data set; and the one or more criteria includes at least one of a specified structural feature, or a specified contextual aspect. The at least one processor, in response to the one or more criteria including the specified structural feature, for each data set of the multiple data sets, is also caused to: analyze the corresponding metadata to determine whether the data set comprises the specified structural feature; and in response to a determination that the data set comprises the specified structural feature, retrieve an indication from the corresponding metadata of a degree of certainty of detection of the specified structural feature within the data set. The at least one processor, in response to the one or more criteria including the specified contextual aspect, for each data set of the multiple data sets, is further caused to: analyze the corresponding context data to determine whether the data set is subject to the specified contextual aspect; and in response to a determination that the data set is subject to the specified contextual aspect, retrieve an indication from the corresponding context data of a degree of certainty concerning the specified contextual aspect. The at least one processor is still further caused to perform operations including: for each data set of the multiple data sets, selectively include the data set in the data catalog based on whether the one or more criteria includes the specified structural feature and whether the data set comprises the specified structural feature, and based on whether the one or more criteria includes the specified contextual aspect and whether the data set is subject to the specified contextual aspect; for each data set included in the data catalog, generate for inclusion in the data catalog, a corresponding score indicative of a likelihood that the data set meets the one or more criteria based on at least one of the degree of certainty of detection of the specified structural feature, or the degree of certainty concerning the specified contextual aspect; and transmit the data catalog to the requesting device.

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium includes instructions operable to cause at least one processor to perform operations including, receive, from a requesting device via a network, a request to provide a data catalog of at least one data set that meets one or more criteria specified in the request, wherein: the at least one data set is to be selected from multiple data sets; for each data set of the multiple data sets, there exists at least one of corresponding metadata indicative of at least one structural feature of the data set, or corresponding context data indicative of at least one contextual aspect of the data set; and the one or more criteria includes at least one of a specified structural feature, or a specified contextual aspect. The at least one processor, in response to the one or more criteria including the specified structural feature, for each data set of the multiple data sets, is also caused to: analyze the corresponding metadata to determine whether the data set comprises the specified structural feature; and in response to a determination that the data set comprises the specified structural feature, retrieve an indication from the corresponding metadata of a degree of certainty of detection of the specified structural feature within the data set. The at least one processor, in response to the one or more criteria including the specified contextual aspect, for each data set of the multiple data sets, is further caused to: analyze the corresponding context data to determine whether the data set is subject to the specified contextual aspect; and in response to a determination that the data set is subject to the specified contextual aspect, retrieve an indication from the corresponding context data of a degree of certainty concerning the specified contextual aspect. The at least one processor is still further caused to perform operations including: for each data set of the multiple data sets, selectively include the data set in the data catalog based on whether the one or more criteria includes the specified structural feature and whether the data set comprises the specified structural feature, and based on whether the one or more criteria includes the specified contextual aspect and whether the data set is subject to the specified contextual aspect; for each data set included in the data catalog, generate for inclusion in the data catalog, a corresponding score indicative of a likelihood that the data set meets the one or more criteria based on at least one of the degree of certainty of detection of the specified structural feature, or the degree of certainty concerning the specified contextual aspect; and transmit the data catalog to the requesting device.

The metadata that corresponds to each data set of the multiple data sets may include a feature vector of multiple storage locations; each storage location of the feature vector may correspond to a different structural feature; each storage location of the feature vector may store an indication of at least one of whether the corresponding structural feature has been detected as incorporated into the corresponding data set, or the degree of certainty of detection of the corresponding structural feature; the context data that corresponds to each data set of the multiple data sets may include a context vector of multiple storage locations; each storage location of the context vector may correspond to a different contextual aspect; and each storage location of the context vector may store an indication of at least one of whether the corresponding data set is subject to the corresponding contextual aspect, or the degree of certainty of the contextual aspect.

Prior to receiving the request for the data catalog, and for each data set of the multiple data sets, the at least one processor may be caused to perform operations including: coordinate execution of instructions of a set of feature routines by a set of processor cores of the at least one processor, wherein execution of the instructions of each feature routine of the set of feature routines causes a processor core of the set of processor cores to search for a structural feature of a pre-selected set of structural features within the data set; receive indications of detected structural features from the set of processor cores; for each detected structural feature, receive a corresponding indication of the degree of certainty of detection of the structural feature from one of the processor cores of the set of processor cores; and generate the corresponding metadata to include indications of the detected structural features, and the corresponding indications of the degree of certainty of detection of the detected structural features.

A structural feature of the pre-selected set of structural features may include a pre-selected type of data identifiable by a data format; and a feature routine of the set of feature routines may include a regular expression engine to employ at least one search pattern written in a pre-selected regular expression syntax to identify data values within a data structure of the pre-selected type of data by the data format. In executing executable instructions of the feature routine, a processor core of the set of processor cores may be caused to perform operations including: retrieve multiple data values of a data set; combine the multiple data values to form a text string; and parse the text string using the regular expression engine to identify instances of a data value of the pre-selected type of data.

A structural feature of the pre-selected set of structural features may include a data structure including multiple storage locations to store multiple data values. The executable instructions of a feature routine of the set of feature routines causes a processor core of the set of processor cores to: search for an instance of the data structure within a data set; and in response to identifying the instance of the data structure, perform operations including, analyze the multiple storage locations to determine a degree of sparsity of data values within the instance of the data structure, derive a degree of certainty of detecting the data structure within the data set at least partially based on the degree of sparsity of data values within the instance of the data structure, and provide, to the at least one processor, an indication of having detected the data structure within the data set, and the degree of certainty of detecting the data structure within the data set.

The degree of certainty of the specified contextual aspect may be based on a degree of trustworthiness associated with a source of an indication of whether a data set is subject to the specified contextual aspect; and prior to receiving the request for the data catalog, and for each data set of the multiple data sets, the at least one processor may be caused to generate the corresponding context data to include the indication of whether the data set is subject to the specified contextual aspect, and the corresponding indication of the degree of certainty of the specified contextual aspect.

The criteria may include a specified subject, and for each data set of the multiple data sets, the at least one processor may be caused to: analyze the corresponding metadata to determine whether the data set covers the specified subject; compare the corresponding score to a minimum threshold score; and selectively include the data set in the data catalog additionally based on the determination of whether the data set covers the specified subject, and based on the comparison of the corresponding score to the minimum threshold score.

For each data set of the multiple data sets that covers the specified subject, the at least one processor may be caused to: analyze the data set to determine a quantity of instances in which the specified subject is referred to within the data set; store, within the corresponding metadata, an indication of the quantity of instances in which the specified subject is referred to within the data set; and generate the corresponding score additionally based on the quantity of instances in which the specified subject is referred to within the data set.

For each data set of the multiple data sets, the at least one processor may be caused to: analyze the data set to identify at least one instance of a type of label from among a pre-selected set of types of label, wherein the at least one instance is indicative of a subject covered by content of the data set; and store a copy of text of each identified instance of the at least one instance within the corresponding metadata. The pre-selected set of types of label may include at least one of: a file name; a document title: a section heading; a sub-heading; a caption of a figure; a caption of a picture; a column heading; a row heading; an end note; a foot note; a name of a person; a name of a place; a name of a corporate, educational, religious or governmental entity; a logo of a corporate, educational, religious or governmental entity; or an address.

The specified contextual aspect may include at least one of: an identity of a source of the data set; a location associated with the source; an industry associated with the source; a time or date of receipt of the data set; an identity of a user of the data set; a location associated with the user; an industry associated with the user; a time or date of receipt of a request from the user to access the data set; or a quantity of instances in which the data set has been accessed. The specified structural feature may include at least one of: a size of the data set; a time or date of generation of the data set; a language used in data values of the data set; a data format of data values of the data set; a range of data values within a row or column of the data set; a statistic descriptive of data values within a row or column of the data set; a type of data structure by which data values are organized within the data set; a type of indexing used in a data structure within the data set; or a quantity of dimensions of indexing used in a data structure within the data set.

A computer-implemented method includes receiving, at a coordinating device of a distributed processing system, and from a requesting device via a network, a request to provide a data catalog of at least one data set that meets one or more criteria specified in the request, wherein: the at least one data set is to be selected from multiple data sets; for each data set of the multiple data sets, there exists at least one of corresponding metadata indicative of at least one structural feature of the data set, or corresponding context data indicative of at least one contextual aspect of the data set; and the one or more criteria includes at least one of a specified structural feature, or a specified contextual aspect. The method also includes, in response to the one or more criteria including the specified structural feature, for each data set of the multiple data sets, performing operations including: analyzing, by at least one processor, the corresponding metadata to determine whether the data set comprises the specified structural feature; and in response to a determination that the data set comprises the specified structural feature, retrieving an indication from the corresponding metadata of a degree of certainty of detection of the specified structural feature within the data set. The method further includes, in response to the one or more criteria including the specified contextual aspect, for each data set of the multiple data sets, performing operations including: analyzing, by the at least one processor, the corresponding context data to determine whether the data set is subject to the specified contextual aspect; and in response to a determination that the data set is subject to the specified contextual aspect, retrieving an indication from the corresponding context data of a degree of certainty concerning the specified contextual aspect. The method still further includes performing operations including: for each data set of the multiple data sets, selectively including, by the at least one processor, the data set in the data catalog based on whether the one or more criteria includes the specified structural feature and whether the data set comprises the specified structural feature, and based on whether the one or more criteria includes the specified contextual aspect and whether the data set is subject to the specified contextual aspect; for each data set included in the data catalog, generating for inclusion in the data catalog, by the at least one processor, a corresponding score indicative of a likelihood that the data set meets the one or more criteria based on at least one of the degree of certainty of detection of the specified structural feature, or the degree of certainty concerning the specified contextual aspect; and transmitting the data catalog to the requesting device via the network.

The metadata that corresponds to each data set of the multiple data sets may include a feature vector of multiple storage locations; each storage location of the feature vector may correspond to a different structural feature; each storage location of the feature vector may store an indication of at least one of whether the corresponding structural feature has been detected as incorporated into the corresponding data set, or the degree of certainty of detection of the corresponding structural feature; the context data that corresponds to each data set of the multiple data sets may include a context vector of multiple storage locations; each storage location of the context vector may correspond to a different contextual aspect; and each storage location of the context vector may store an indication of at least one of whether the corresponding data set is subject to the corresponding contextual aspect, or the degree of certainty of the contextual aspect.

The computer-implemented method may include, prior to receiving the request for the data catalog, and for each data set of the multiple data sets, performing operations including: coordinating execution of instructions of a set of feature routines by a set of processor cores of the at least one processor, wherein execution of the instructions of each feature routine of the set of feature routines causes a processor core of the set of processor cores to search for a structural feature of a pre-selected set of structural features within the data set; receiving indications of detected structural features from the set of processor cores; for each detected structural feature, receiving a corresponding indication of the degree of certainty of detection of the structural feature from one of the processor cores of the set of processor cores; and generating the corresponding metadata to include indications of the detected structural features, and the corresponding indications of the degree of certainty of detection of the detected structural features.

A structural feature of the pre-selected set of structural features may include a pre-selected type of data identifiable by a data format; and a feature routine of the set of feature routines may include a regular expression engine to employ at least one search pattern written in a pre-selected regular expression syntax to identify data values within a data structure of the pre-selected type of data by the data format. In executing executable instructions of the feature routine, a processor core of the set of processor cores may be caused to perform operations including: retrieve multiple data values of a data set; combine the multiple data values to form a text string; and parse the text string using the regular expression engine to identify instances of a data value of the pre-selected type of data.

A structural feature of the pre-selected set of structural features may include a data structure including multiple storage locations to store multiple data values. The executable instructions of a feature routine of the set of feature routines causes a processor core of the set of processor cores to: search for an instance of the data structure within a data set; and in response to identifying the instance of the data structure, perform operations including, analyze the multiple storage locations to determine a degree of sparsity of data values within the instance of the data structure, derive a degree of certainty of detecting the data structure within the data set at least partially based on the degree of sparsity of data values within the instance of the data structure, and provide, to the at least one processor, an indication of having detected the data structure within the data set, and the degree of certainty of detecting the data structure within the data set.

The degree of certainty of the specified contextual aspect may be based on a degree of trustworthiness associated with a source of an indication of whether a data set is subject to the specified contextual aspect; and the method may include, prior to receiving the request for the data catalog, and for each data set of the multiple data sets, generating the corresponding context data to include the indication of whether the data set is subject to the specified contextual aspect, and the corresponding indication of the degree of certainty of the specified contextual aspect.

The criteria may include a specified subject, and the method may include, for each data set of the multiple data sets, performing operations including: analyzing the corresponding metadata to determine whether the data set covers the specified subject; comparing the corresponding score to a minimum threshold score; and selectively including the data set in the data catalog additionally based on the determination of whether the data set covers the specified subject, and based on the comparison of the corresponding score to the minimum threshold score.

The computer-implemented method may include, for each data set of the multiple data sets that covers the specified subject, performing operations including: analyzing the data set to determine a quantity of instances in which the specified subject is referred to within the data set; storing, within the corresponding metadata, an indication of the quantity of instances in which the specified subject is referred to within the data set; and generating the corresponding score additionally based on the quantity of instances in which the specified subject is referred to within the data set.

The computer-implemented method may include performing operations including: analyzing the data set to identify at least one instance of a type of label from among a pre-selected set of types of label, wherein the at least one instance is indicative of a subject covered by content of the data set; and storing a copy of text of each identified instance of the at least one instance within the corresponding metadata. The pre-selected set of types of label may include at least one of: a file name; a document title: a section heading; a sub-heading; a caption of a figure; a caption of a picture; a column heading; a row heading; an end note; a foot note; a name of a person; a name of a place; a name of a corporate, educational, religious or governmental entity; a logo of a corporate, educational, religious or governmental entity; or an address.

The specified contextual aspect may include at least one of: an identity of a source of the data set; a location associated with the source; an industry associated with the source; a time or date of receipt of the data set; an identity of a user of the data set; a location associated with the user; an industry associated with the user; a time or date of receipt of a request from the user to access the data set; or a quantity of instances in which the data set has been accessed. The specified structural feature may include at least one of: a size of the data set; a time or date of generation of the data set; a language used in data values of the data set; a data format of data values of the data set; a range of data values within a row or column of the data set; a statistic descriptive of data values within a row or column of the data set; a type of data structure by which data values are organized within the data set; a type of indexing used in a data structure within the data set; or a quantity of dimensions of indexing used in a data structure within the data set.

The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

