Machine learning and classification involves the extraction of structured or unstructured targeted raw data from various data sources and transforming the targeted raw data into a format suitable for performing subsequent learning and/or classification processing on. During machine learning, descriptive patterns within the target data are collected and documented using a consistently applied aggregation method, which typically collects descriptive statistical facts about data of known categories or classification origins. When data of unknown categories or classification origins are used as data input, the consistently applied aggregation method can be used to identify descriptive patterns within the unknown data for comparison and classification against all data of known classification types and origins within the current system.
A parallel data processing implementation for the purposes of machine learning and classification typically comprises the distribution of targeted raw data and a consistently applied aggregation method across multiple processors and disks to perform simultaneous computations, while taking advantage of increased memory, reduced i/o, and the combined processing power of all participating computing devices.
Several systems and programming languages, such as the R Project for Statistical Computing, include programming features for the learning and classification of data. However, current solutions do not provide the functionality to automatically distribute learning and classification processes across multiple processors and disks in a distributed parallel computing environment using a map reduction aggregation method. Furthermore, current learning and classification systems implement a rigid framework which typically requires the use of a single predefined aggregation method and classification metric function. These limitations often result in extensive and sometimes very overhead intensive input data pre-processing in order to transform the targeted data for use within the rigid framework. Furthermore, the rigid framework typically does not support multiple application specific map reduction aggregation methods and classification metric functions created by the application programmer.