FIELD

The invention relates to the field of large scale data processing. In particular, but not exclusively, it relates to automated analysis of data that may be stored in a decentralized manner.

BACKGROUND

With the growth of the use of the Internet, the growth of data usage in private networks and the growth of data used by companies and other entities, both internal and external data, the need for massive data storage and massive computing power has risen. Therefore, many entities are turning to cloud computing. The terms “the cloud” or “cloud computing” may refer generally to large scale data centers that are maintained by a third party, or a company or entity, for example one that maintains systems and/or software that work with the data center(s), where the storage and computing capabilities of the numerous servers within the data center are offered to internal or external customers through one or more network connections. Because relatively small entities may have access to the large scale storage and computing power of many servers, the entities can have access to large-scale computing power that is flexible and available while lowering or eliminating the costs needed to maintain the data centers. Various databases, such as communications databases and/or databases in a cloud computing data center, may be useful for storing massive amounts of data, but in various database approaches, the data is stored in a decentralized manner, across several servers or nodes, and information regarding the relationships or correlations between the data may not be stored. In various databases, for example, communications databases and/or other forms of databases, large-scale data is formatted or structured to be most easily used for task-specific computations. In other words, data may be analyzed at the outset, for example a particular relationship may be analyzed, and then the data and the resulting conclusion are stored in a specific format. For various databases, this is called a schema. Once the initial analysis is done, it may be very time consuming and difficult to re-structure and/or re-analyze data to find a new value, short of retrieving, harvesting and/or archiving locally all the data the user is interested and then organizing and/or performing computations or routines on the data to analyze relationships.

Data has become a key asset for most modern day enterprises. Managing this data has become a major problem for the IT departments of these companies and organizations. For many years, the changes in business requirements have made it more and more difficult and expensive for enterprises to keep abreast of the changes in data—firstly, because of continuous changes in the tools and standards, and secondly because of the exponential increase in the amount of data that is being made available.

Enterprises may find it difficult to detect business value in the relationships between data points, where many different types of data exist. Trying to convert data to a heterogeneous but flexible format may likely result in incomplete information that is collected from limited points.

Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with the present invention as set forth in the remainder of the present application with reference to the drawings.

BRIEF SUMMARY

Aspects of the present invention are aimed at reducing or eliminating the above problems with the prior art. In particular, but not exclusively, aspects of the invention are directed to large scale data processing modules and methods for automated analysis of data that may be stored in a decentralized manner.

One example embodiment of this invention comprises an enterprise data processing module. The enterprise data processing module comprises at least one collector and at least one analyzer. The collectors may be operable to collect data pieces from a plurality of data sources. The analyzers may be operable to analyze the collected data pieces to determine cross-source relationships that exist between the data pieces collected from the plurality of sources. The analyzed data pieces may be stored in one or more big-data databases as blocks of data according to the cross-source relationships.

In another example embodiment of this invention, the enterprise data processing module, comprises a user interface operable to receive a request from a user to interact with a data group stored in the one or more big-data databases. The request may be attempting to utilize information from the cross-source relationship. The information from the cross-source relationship may comprise conclusion data that supports a schema. If the user has permission to access information from the cross-source relationship, the request may be processed to return the conclusion data without extracting all underlying data required to compute the requested conclusion data.

In another example embodiment of this invention, the cross-source relationship may comprise a degree of correlation that is determined by a correlation intensity algorithm. The correlation intensity algorithm may determine a level of similarity with respect to factors such as, but not limited to: the number of unique concepts in each data piece; a complexity of the data pieces; a size of the data pieces; a spam score of each data piece; and/or a readability score of each data piece.

