BACKGROUND OF THE INVENTION

The present invention is related to a data processing system for processing source data, and is more specifically related to identifying the component parts of one or more tables present within the source data using the data processing system.

Many companies today are required to provide and disclose financial related information to their accounting and financial advisors. The accounting advisors often receive from their client companies voluminous amounts of financial data, in disparate forms and formats, that must be handled and processed accordingly. This creates an enormous strain on the resources of the accounting advisors.

The financial information is often provided to the accounting advisors in both digital and non-digital formats. Regardless of the manner of disclosure, the accounting advisors need to process and review the disclosed information so as to prepare the necessary financial disclosure documents, such as tax returns, 10-K documents and the like. Oftentimes, a significant portion of the disclosed financial data is provided in a tabular format, which is difficult to process in a digital manner and is also difficult to retrieve data therefrom at a later time. This drawback occurs since current software systems have a difficult time recognizing the table data, and in the off chance that portions of the table data are recognized, disassembling the tables into their components parts for further processing is quite difficult.

As such, the conventional processing and handling methodology requires that the accounting advisors manually review the disclosed financial information and then manually input into a computer system the relevant and pertinent financial data. This process is time consuming and expensive.

SUMMARY OF THE INVENTION

An object of the present invention is to digitally process source data that includes financial information by digitally recognizing the presence of tables therein, disassembling the tables into their component parts (e.g., rows and columns), extracting the associated financial data therefrom, and then using the extracted data in preparing the necessary financial documents.

The present invention is directed to a system and method for digitally processing and recognizing financial information presented in a tabular format. The invention includes digitizing the input source data, which can include financial related information presented in a tabular format, pixelating the input information into coordinate data (e.g., X and Y axis coordinate data), clustering together similar data into clusters of pixels, processing the coordinate data along one or more of the coordinate axes, and then classifying the data as corresponding to a row of the table and/or a column of the table. The present invention also contemplates processing the resultant table data to determine the header rows forming part of the table data.

The present invention is thus directed to a data processing system for processing source data that includes table data, wherein the table data has associated therewith a plurality of rows and a plurality of columns. The system includes a conversion unit for converting the table data into machine encoded text data having associated therewith text coordinate data, wherein the coordinate data has a Y-axis component and an X-axis component; a pixelation unit for generating from the machine encoded text data a plurality of pixels along the Y-axis component or the X-axis component, wherein the plurality of pixels can have one or more attributes associated therewith; a clustering unit for receiving the plurality of pixels from the pixelation unit and in response thereto performing a clustering technique thereon to generate a plurality of clusters of pixels by clustering together one or more pixels from the plurality of pixels according to a similarity of the one or more attributes; a computation unit for processing the plurality of clusters of pixels and for determining a maximum extent and a minimum extent for each of the plurality of clusters of pixels; and a classification unit for classifying each of the plurality of clusters of pixels received from the computation unit as a selected one of the plurality of rows of the table or a selected one of the plurality of columns of the table, where the maximum and minimum extents of each of the plurality of clusters of pixels correspond to a width of one of the plurality of columns of the table data or a height of one of the plurality of rows of the table data.

The preprocessing unit is configured for cropping the source data to identify the table data and the conversion unit is configured to convert the table data into the machine encoded text data using an optical character recognition technique.

The pixelation unit comprises a Y-axis converter for selecting and converting the machine encoded data into the plurality of pixels along the Y-axis component, and an X-axis converter for selecting and converting the machine encoded data into the plurality of pixels along the X-axis component. Further, each of the Y-axis converter and the X-axis converter includes a border determination unit for forming a border around the machine encoded text data, and a filling unit for providing additional pixels to the plurality of pixels.

The clustering technique employed by the present invention comprises a density clustering technique, and the one or more attributes associated with the plurality of pixels comprises at least one of coordinate data and distance data representative of a distance between one or more pixels of the plurality of pixels.

The computation unit of the present invention includes a Y-axis calculation unit for determining the maximum and minimum extents for each of the plurality of rows in the table data, and an X-axis calculation unit for determining the maximum and minimum extents for each of the plurality of columns in the table data. The Y-axis converter of the pixelation unit processes the machine encoded text data along the Y-axis component to generate the plurality of pixels along the Y-axis component, wherein the attribute of the plurality of pixels includes the Y-axis component and wherein each of the plurality of clusters of pixels generated by the clustering unit is associated with the plurality of rows of the table data. The Y-axis calculation unit of the computation unit receives the plurality of clusters of pixels and determines the minimum and maximum extents for each of the plurality of rows, and the classification unit classifies each of the plurality of clusters as one of the plurality of rows of the table data.

According to another aspect of the present invention, the classification unit generates row identification data associated with each one of the plurality of rows of the table data, and the pixelation unit receives and processes the row identification data with the X-axis converter along the X-axis component to determine the plurality of pixels along the X-axis component. The clustering unit processes the plurality of pixels along the X-axis component using the clustering technique to generate a plurality of clusters of pixels along the X-axis component by clustering together one or more pixels from the plurality of pixels, and the X-axis calculation unit of the computation unit determines the maximum and minimum extents for each of the plurality of clusters of pixels along the X-axis for determining the maximum and minimum extents for each of the plurality of columns in the table data.

Further, each of the plurality of columns and the plurality of rows of the table data have a text cell with text data associated therewith, and the X-axis calculation unit further determines the text data present in each of the text cells of each of the columns to be clustered together and determines the clusters that belong in a different column. The X-axis calculation unit also determines the number of pixels required to represent a single text character (A) in each row of the plurality of rows using the following formula: A=avg number of pixels/length of token, where A is the number of pixels that represent a single character and the token corresponds to a word portion of the text data in the row. The X-axis calculation unit54is further adapted to determine a pixel length for each token in a row (B), and merge together clusters of pixels within a 2B distance of each other so as to place the tokens within the same column. The X-axis calculation unit further processes the cluster of pixels and determines the maximum extent and the minimum extent for each column of the plurality of columns, and the classification unit classifies each cluster of pixels as a column in the table data.

The classification unit generates row identification data corresponding to the plurality of rows and column identification data corresponding to the plurality of columns, and the system further comprises a table header and identification sub-system for identifying one or more header rows from among the plurality of rows. The table header and identification sub-system comprises an extraction unit for extracting row data from the row identification data and the column identification data and assigning a data type to each row of the plurality of rows, an estimation unit for estimating a similarity between the data types for each row of the plurality of rows, and an identification unit for identifying the rows with similar data types and identifying the rows with non-similar data types, and designating the rows with non-similar data types as header rows. The extraction unit comprises means for implementing a Jaccard similarity technique to extract the row data.

According to another aspect of the present invention, the data processing system processes source data that includes table data, and the table data has associated therewith a plurality of rows and a plurality of columns. The system includes a conversion unit for converting the table data into machine encoded text data having associated therewith text coordinate data, where the coordinate data has a Y-axis component and an X-axis component; a pixelation unit for generating from the machine encoded text data a plurality of pixels along the Y-axis component, where the plurality of pixels can have one or more attributes associated therewith; a clustering unit for receiving the plurality of pixels from the pixelation unit and in response thereto performing a clustering technique thereon to generate a plurality of clusters of pixels by clustering together one or more pixels from the plurality of pixels according to a similarity of the one or more attributes; a computation unit for processing the plurality of clusters of pixels and for determining a maximum extent and a minimum extent for each of the plurality of clusters of pixels; and a classification unit for classifying each of the plurality of clusters of pixels received from the computation unit as a selected one of the plurality of rows of the table, and wherein the maximum and minimum extents of each of the plurality of clusters of pixels correspond to a width of one of the plurality of columns of the table.

The present invention is also directed to a method of processing source data that includes table data, wherein the table data has associated therewith a plurality of rows and a plurality of columns, comprising converting the table data into machine encoded text data having associated therewith text coordinate data, wherein the coordinate data has a Y-axis component and an X-axis component; generating from the machine encoded text data a plurality of pixels along the Y-axis component or the X-axis component, wherein the plurality of pixels can have one or more attributes associated therewith; performing a clustering technique on the plurality of pixels to generate a plurality of clusters of pixels by clustering together one or more pixels from the plurality of pixels according to a similarity of the one or more attributes; processing the plurality of clusters of pixels and determining a maximum extent and a minimum extent for each of the plurality of clusters of pixels; and classifying each of the plurality of clusters of pixels received from the computation unit as a selected one of the plurality of rows of the table or a selected one of the plurality of columns of the table, and wherein the maximum and minimum extents of each of the plurality of clusters of pixels correspond to a width of one of the plurality of columns of the table data or a height of one of the plurality of rows of the table data.

The method can also include cropping the source data to identify the table data, and then selecting and converting the machine encoded data into the plurality of pixels along the Y-axis component, or selecting and converting the machine encoded data into the plurality of pixels along the X-axis component.

The method of the present invention can also include determining the maximum and minimum extents for each of the plurality of rows in the table data, or determining the maximum and minimum extents for each of the plurality of columns in the table data. The method can further include processing the machine encoded text data along the Y-axis component to generate the plurality of pixels along the Y-axis component, wherein the attribute of the plurality of pixels includes the Y-axis component, associating each of the plurality of clusters of pixels with the plurality of rows of the table data, determining the minimum and maximum extents for each of the plurality of rows, and classifying each of the plurality of clusters as one of the plurality of rows of the table data.

According to another aspect, the method still further includes generating row identification data associated with each one of the plurality of rows of the table data, processing the row identification data along the X-axis component to determine the plurality of pixels along the X-axis component, processing the plurality of pixels along the X-axis component to generate a plurality of clusters of pixels along the X-axis component by clustering together one or more pixels from the plurality of pixels, and determining the maximum and minimum extents for each of the plurality of clusters of pixels along the X-axis for determining the maximum and minimum extents for each of the plurality of columns in the table data.

The plurality of columns and the plurality of rows of the table data each have a text cell with text data associated therewith. As such, the method of the present invention further includes determining the text data present in each of the text cells of each of the columns to be clustered together and determining the clusters that belong in a different column.

According to other aspects, the method includes determining the number of pixels required to represent a single text character (A) in each row of the plurality of rows using the following formula: A=avg number of pixels/length of token, where A is the number of pixels that represent a single character and the token corresponds to a word portion of the text data in the row. As such, the method can further include determining a pixel length for each token in a row (B), and merging together clusters of pixels within a 2B distance of each other so as to place the tokens within the same column.

The method of the invention can still further include processing the cluster of pixels and determining the maximum extent and the minimum extent for each column of the plurality of columns, and classifying each cluster of pixels as a column in the table data.

According to another aspect, the classification unit generates row identification data corresponding to the plurality of rows and column identification data corresponding to the plurality of columns. As such, the method further comprises identifying one or more header rows from among the plurality of rows, extracting row data from the row identification data and the column identification data and assigning a data type to each row of the plurality of rows, estimating a similarity between the data types for each row of the plurality of rows, and identifying the rows with similar data types and identifying the rows with non-similar data types, and designating the rows with non-similar data types as header rows.

The present invention is also directed to a table header identification system for processing table data having a plurality of rows and a plurality of columns and identifying one or more header rows from among the plurality of rows. The system includes an extraction unit for extracting row data corresponding to the plurality of rows from the table data and assigning a data type to each row of the plurality of rows; an estimation unit for estimating a similarity between the data types for each row of the plurality of rows; and an identification unit for identifying the rows with similar data types and identifying the rows with non-similar data types, and designating the rows with non-similar data types as header rows. The data types are associated with the type of data set forth in the table data.

The estimation unit comprises means for assigning a similarity coefficient to each row of the plurality of rows, and also includes means for implementing a Jaccard similarity coefficient technique to the row data. The similarity coefficient is a Jaccard coefficient. The identification unit clusters together the rows having similar similarity coefficients.

The present invention is further directed to a method for processing table data having a plurality of rows and a plurality of columns and identifying one or more header rows from among the plurality of rows, comprising extracting row data corresponding to the plurality of rows from the table data with an extraction unit and assigning a data type to each row of the plurality of rows; estimating a similarity between the data types for each row of the plurality of rows; and identifying the rows with similar data types and identifying the rows with non-similar data types, and designating the rows with non-similar data types as header rows.

The step of estimating further comprises assigning a similarity coefficient to each row of the plurality of rows. The step of identifying further comprises clustering together the rows having similar similarity coefficients.

