<jupyter_start><jupyter_text>1. Installation<jupyter_code>!pip install -U ExtractTable<jupyter_output><empty_output><jupyter_text>2. Import and check version<jupyter_code>from ExtractTable import ExtractTable
print(ExtractTable.VERSION)<jupyter_output><empty_output><jupyter_text>3. Create Session & Validate API Key  3.1 **Create Session** with your API Key<jupyter_code>api_key = YOUR_APIKEY_HERE
et_sess = ExtractTable(api_key)
# et_sess.__dict__<jupyter_output><empty_output><jupyter_text>3.2 **Validate** the Key and check the plan usage<jupyter_code>usage = et_sess.check_usage()<jupyter_output><empty_output><jupyter_text>*If there is no error encountered in the above cell, it means we have a valid API key. Now, lets get started by checking the usage and trigger the file for processing*<jupyter_code># et_sess.server_response
print(usage)<jupyter_output><empty_output><jupyter_text>3.3 Check Usage Details**credits**: Total number credits attached to the API Key**queued** : Number of triggered jobs that are still processing in the queue**used**   : Number of credits already used   4. Trigger the extraction process> Note: We will use the session, `et_sess`, created earlier in step 3.1, to save the session data and retrieve when needed  4.1 Accepted Input Types**Allowed input formats** are:- Image  - JPG/JPEG  - PNG- PDF  - Text PDF  - Scan PDF  - Image PDF**Input Location Options**- Location can be a file from the local drive- Accessible remote URL - *the file object will be locally downloaded and deleted once sent to the process*<jupyter_code># image_location = "local_image_path_OR_remote_image_url_with_tables"
# image_location = r'samples/BlurryImage.jpg'
image_location = "https://raw.githubusercontent.com/ExtractTable/ExtractTable-py/master/samples/QualityImage.jpg"<jupyter_output><empty_output><jupyter_text>4.2 Process an IMAGE Input<jupyter_code>table_data = et_sess.process_file(filepath=image_location, output_format="df")
table_data<jupyter_output><empty_output><jupyter_text>4.3 Process a PDF Input<jupyter_code># pdf_location = "local_image_path_OR_remote_image_url_with_tables"
# pdf_location = r'samples/BlurryImage.jpg'
pdf_location = "https://raw.githubusercontent.com/ExtractTable/ExtractTable-py/master/samples/QualityImage.jpg"
table_data = et_sess.process_file(filepath=Location_of_PDF_with_Tables, pages="all", output_format="df")<jupyter_output><empty_output><jupyter_text>Below are the sample values ```pages``` accepts **string** type| pages  	| Explanation                                     	||----------	|-------------------------------------------------	|| "1"      	| [Default] considers only 1st page of the PDF    	|| "1,3,5"  	| considers only 1st, 3rd and 5th page of the PDF 	|| "1, 3-5" 	| considers 1st, 3rd, 4th and 5th page of the PDF 	|| "all"    	| considers complete PDF                          	|  4.4 Table Output options> By default, the `process_file()` returns **only** the table data. Output depends on the `output_format` , explained belowExplore the available options with `ExtractTable._OUTPUT_FORMATS`| output_format 	| Explanation                                	||---------------	|--------------------------------------------	|| "df"          	| [Default] Array of Pandas dataframes        	|| "dataframe"   	| same as "df"; Array of Pandas dataframes    	|| "json"        	| JSON data with index orientation           	|| "dict"        	| Similar to JSON data but python dictionary 	|| "csv"         	| Array of locally saved CSV file locations   	|| "xlsx"         	| To save multiple tables as sheets into a single excel	|| "excel"         | same as "xlsx"; output is an array of excel location	|Default output is an array of pandas dataframes, with which you can change to any other format like excel, html etc. Follow https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html  4.5 Explore session objects> **Explore** all objects of the latest file processing with `et_sess.__dict__.keys()` - Depends on the plan type of your API Key<jupyter_code>et_sess.__dict__.keys()<jupyter_output><empty_output><jupyter_text>Based on the API Key PLAN type, the et_sess contains below objects| Object          	| Explanation                                                                            	||-----------------	|----------------------------------------------------------------------------------------	|| api_key         	| Your API Key                                                                           	|| _session        	| Session data of the **latest** performed request/action                                          	|| input_filename  	| Name of the processed input file || ServerResponse  	| Complete ServerResponse, along with response code and headers                          	|| server_response 	| complete server response content; equivalent to `ServerResponse.json()`                	|| JobStatus       	| Job Status of the triggered process                                                    	|| Pages           	| Number of pages in the input; also number of credits consumed on the triggered process 	|| Tables          	| Tabular Data in JSON format with index orientation; ordered table wise                 	|| Lines           	| Text Data in JSON format, ordered page wise                                            	|  4.6 Save Table & Text to LOCAL```pythonet_sess.save_output(output_folder, output_format="csv")````output_format` param is relavant only for the table data, with options "csv" or "xlsx"> Note: As the `et_sess` contains the latest action performed, make sure this call is right after the `process_file()`  5. Explore the Output  5.1 Output Structure> **Understand the output**: The response of a triggered job is a JSON object in the below format. Note that the response depends on the plan type of the API Key.```javascript{    "JobStatus": ,                               Status of the triggered Process  @ JOB-LEVEL    "Pages": ,                                  Number of pages processed in this request @ PAGE-LEVEL    "Tables": [      List of all tables found @ TABLE-LEVEL        {            "Page": ,                               Page number in which this table is found            "CharacterConfidence": ,                  Accuracy of Characters recognized from the input-page            "LayoutConfidence": ,                     Accuracy of table layout's design decision            "TableJson": ,                             Table Cell Text in key-value format with index orientation - {row: {col: }}            "TableCoordinates": ,                      Top-left & Bottom-right Cell Coordinates - {row: {col: }}            "TableConfidence":                         Cell level accuracy of detected characters - {row: {col: }}        },    {...}                                                ... more "Tables" objects    ],    "Lines": [                Pagewise Line details @ PAGE-LEVEL        {            "Page": ,                           Page number in which the lines are found            "CharacterConfidence": ,              Average Accuracy of all Characters recognized from the input-page            "LinesArray": [                      Ordered list of lines in this page @ LINE-LEVEL                {                    "Line": ,                           Detected text of the complete line                    "WordsArray": [                                  Word level datails in this line @ WORD-LEVEL                        {                            "Conf": ,                     Accuracy of recognized characters of the word                            "Word": ,                       Detected text of the word                            "Loc": [x1, y1, x2, y2]              Top-left & Bottom-right coordinates, w.r.t the input-page width-height dimensions                        },                    {...}                                    More "WordsArray" objects                    ]                },            {...}                                        More "LinesArray" objects            ]        },    {...}                                                More Pagewise "Lines" details    ]}```  5.2 Output DetailsOutput objects are based on the API Key Plan type. Available plan types are **Purchased Plans***   "LITE"   - **only table data** in the output*   "FULL"   - **table and text data** in the output*   "EXTRA"  - **table, text data along with cell & word coordintates and character detection accuracy****Promotional Plans**: Any plan other than Purchased plans are promotional*   "free_trial", "camelotpro" - these are promotional API Keys, gives only table data equivalent to "LITE" plan typeOutput objects detail below| Key Name 	| Parent 	| Type 	| Description 	| Availability 	||-	|-	|-	|-	|-	|| JobStatus 	| Job 	| String 	| Status of the triggered process 	| ALL Plans 	|| Pages 	| Job 	| Integer 	| Number of pages processed in the request 	| ALL Plans 	|| Tables 	| Job 	| Array 	| List of all tables found 	| ALL Plans 	|| Tables[0].Page 	| Table 	| Integer 	| Page number in which the table is found 	| ALL Plans 	|| Tables[0].CharacterConfidence 	| Table 	| Decimal 	| Accuracy of Characters recognized from the image 	| ALL Plans 	|| Tables[0].LayoutConfidence 	| Table 	| Decimal 	| Accuracy of table layout's design decision 	| ALL Plans 	|| Tables[0].TableJson 	| Table 	| Json/dict 	| Table Cell Text in key-value format with index orientation - {row: {col: }} 	| ALL Plans 	|| Tables[0].TableCoordinates 	| Table 	| Json/dict 	| Top-left & Bottom-right Cell Coordinates - {row: {col: }} 	| EXTRA Plan 	|| Tables[0].TableConfidence 	| Table 	| Json/dict 	| Cell level accuracy of detected characters - {row: {col: }} 	| EXTRA Plan 	|| Lines 	| Job 	| Array 	| List of page-wise lines text 	| FULL, EXTRA	|| Lines[0].Page 	| Page 	| Integer 	| Page number in which the lines are found 	| Full Plan 	|| Lines[0].CharacterConfidence 	| Page 	| Decimal 	| Average Accuracy of all Characters recognized from the input-page 	| Full Plan 	|| Lines[0].LineArray 	| Page 	| Array 	| Ordered list of lines of the page 	|  	|| Lines[0].LineArray[0].Line 	| Line 	| String 	| Detected text of the complete line 	| Full Plan 	|| Lines[0].LineArray[0].WordsArray 	| Line 	| Array 	| Word level datails in this line 	| EXTRA Plan 	|| Lines[0].LineArray[0].WordsArray[0].Conf 	| Word 	| Decimal 	| Accuracy of recognized characters of the word 	| EXTRA Plan 	|| Lines[0].LineArray[0].WordsArray[0].Word 	| Word 	| String 	| Detected text of the word 	| EXTRA Plan 	|| Lines[0].LineArray[0].WordsArray[0].Loc 	| Word 	| Array 	| Top-left & Bottom-right coordinates, w.r.t the input-page width-height dimensions 	| EXTRA Plan 	|  6. Make Corrections> **Objective**: To ease corrections on the most common issues with the `MakeCorrections` module.**Details:** The service relies on OCR (Optical Character Recognition) for character detection and deep learning models to detect tabular structures on the input. There may be a chance for merged rows or columns or incorrect type detections on low-quality inputs with a complex table layout or tightly packed columns. With those in mind, we want to offer the built-in service at the client-side to give control and ease in making corrections on the output. The module, `MakeCorrections`, currently supports below functionalities| Functionality        	| Explanation                                    	||----------------------	|------------------------------------------------	|| Split Merged Rows    	| Works well on cell values with no spaces       	|| Split Merged Columns 	| Works well on cell values with no spaces       	|| Fix Decimal Format   	| To fix thousand and decimal separators         	|| Fix Date Format      	| To handle and modify incorrect date separators 	|| Fix Characters       	| To fix incorrect character detections          	|<jupyter_code># First things first lets import the module and prepare for corrections

from ExtractTable.common import MakeCorrections

corrections = MakeCorrections(et_resp=et_sess.server_response)<jupyter_output><empty_output><jupyter_text>6.1 Split Merged Rows        """        To split the merged rows into possible multiple rows        :return: reformatted list of dataframes        """<jupyter_code>corrected_table_dataframes = corrections.split_merged_rows()<jupyter_output><empty_output><jupyter_text>6.2 Split Merged Columns        """        To split the merged columns into possible multiple columns        :param columns_idx: user preferred columns indices.                Default loops through all columns to find numeric or decimal columns        :param force_split: To force split through the columns        :return: reformatted list of dataframes        """<jupyter_code>corrected_table_dataframes = corrections.split_merged_columns()<jupyter_output><empty_output><jupyter_text>6.3 Fix Decimal Format        """        To fix decimal and thousands separator values. Often commas as detected as period        :param columns_idx: user preferred columns indices.                Default loops through all columns to find numeric or decimal columns        :param decimal_separator: preferred decimal separator        :param thousands_separator: preferred thousands separator        :param decimal_position: preferred decimal position        :return: corrected list of dataframes        """<jupyter_code>corrected_table_dataframes = corrections.fix_decimal_format(decimal_separator=".", thousands_separator=",")<jupyter_output><empty_output><jupyter_text>6.4 Fix Date Format        """        To fix date formats of the column        Eg: 12|1212020 as 12/12/2020        :param columns_idx: user preferred columns indices.                Default loops through all columns to find Date Columns        :param delimiter: "/" or "-" whatelse you prefer        :return: correted list of dataframes        """<jupyter_code>corrected_table_dataframes = corrections.fix_date_format(delimiter="/")<jupyter_output><empty_output><jupyter_text>6.5 Fix Characters        """        To replace incorrect character detections        Eg: $123,45.0I as $123,45.01        :param columns_idx: user preferred columns indices.                Default loops through all columns to find Date Columns        :param replace_ref: the replacement dictionary for reference                Eg: {"I": "1"}        :return: correted list of dataframes        """<jupyter_code>corrected_table_dataframes = corrections.fix_characters(self, columns_idx=[0, 1], replace_ref={"I": "1", "S": "$"})<jupyter_output><empty_output><jupyter_text>7. Helpful Code SnippetsExtra code snippets that are useful to perform some actions on the output. Based on the frequently asked questions.  7.1 Get text data<jupyter_code># If your API Key supports "Lines"

all_page_lines = []
for each_page in et_sess.Lines:
  for each_line in each_page['LinesArray']:
    all_page_lines.append(each_line['Line'])
  
print("\n".join(all_page_lines))<jupyter_output><empty_output><jupyter_text>7.2 All tables output to a single excel<jupyter_code>table_data = et_sess.process_file(filepath=Location_of_PDF_with_Tables, output_format="df", pages="all")
 
import pandas as pd
accumulate_all_dfs = pd.DataFrame()

for each_df in table_data:
    accumulate_all_dfs = accumulate_all_dfs.append(each_df, ignore_index=True)
    # print(each_df.shape, accumulate_all_dfs.shape)

print("Shape of all tables accumulated together is", accumulate_all_dfs.shape)


output_excel_location = <location_for_the_excel_output.xlsx>
# Save the accumulated output to a single excel file
accumulate_all_dfs.to_excel(output_excel_location, index=False, header=False)<jupyter_output><empty_output><jupyter_text>8. View transactionsTo view all the transactions that were triggered in the last 24 hours<jupyter_code>et_sess.view_transactions()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09944647125424581, 'fraction_numerical': 0.00773682224179142, 'mean_word_length': 2.6109470815353166, 'pattern_counts': {'":': 21, '<': 74, '<?xml version=': 0, '>': 80, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '597513', 'license': 'Apache-2.0', 'n_tokens_mistral': 4365, 'n_tokens_neox': 4166, 'n_words': 1473, 'path': 'example-code.ipynb', 'repo_name': 'RakhithJK/ExtractTable-py', 'size': '15898'}