<jupyter_start><jupyter_text>Convert PDF files to TXT files The code in this Jupyter Notebook converts PDF files into plain text files (*.txt) by extracting text from each page of the documents, then writing that text to newly created TXT files.   Setup First, we need to install the libraries/packages required to run the code in this notebook. You only need to run the cell below the first time you use this notebook.<jupyter_code># Uninstall the fitz
%pip uninstall fitz
%pip install -r requirements.txt<jupyter_output><empty_output><jupyter_text>Next, we import the libraries and modules we'll be using.<jupyter_code># Import required modules 
import os
import fitz<jupyter_output><empty_output><jupyter_text>Running the cell above may have given you a warning messesage, "WARNING: Skipping fitz as it is not installed." If so, no worries! That just means that there wasn't already a package called fitz installed. If you didn't get a warning message, fitz was uninstalled in order to avoid interference with importing the PyMuPDF library in the import fitz line. For more information, see the following GitHub issue for the PyMuPDF project: Unable to use fitz with python 3.8 The last setup steps are to specify the folder/directory containing the PDF files you want to convert and the directory where you want to save the newly created TXT files. This notebook assumes that the PDF files you want to convert are in a folder/directory called 'pdf' that is in the same directory as the notebook. If the PDF files are in a different directory, you will need to update the value for the pdf_dir variable accordingly in the cell below. You may use either a relative path or an absolute path (for more information, see What is the difference between a relative and absolute path?).<jupyter_code># Set the directory with PDF documents for text extraction
pdf_dir = 'pdf'

# Validate directory path
if not os.path.exists(pdf_dir):
    raise Exception ("Directory '%s' not found!" % (pdf_dir))<jupyter_output><empty_output><jupyter_text>If you'd like to store your TXT files in a different directory, make sure to to update the value for the txt_dir variable accordingly in the cell below.<jupyter_code># Create a subdirectory called 'txt' for plain text files
txt_dir = 'txt'

try:
    os.stat(txt_dir)
except:
    os.mkdir(txt_dir)<jupyter_output><empty_output><jupyter_text>Create TXT files with extracted text from PDF documents Finally, we can convert our PDF files into TXT files!**Resources**- [Document — PyMuPDF 1.19.4 documentation](https://pymupdf.readthedocs.io/en/latest/document.html)- [Page — PyMuPDF 1.19.4 documentation](https://pymupdf.readthedocs.io/en/latest/page.html)<jupyter_code># Create a list for files in which text is not recognized
exceptions = []

# Iterate through all files in the directory with PDFs
for filename in os.listdir(pdf_dir):
        
    # Check if the file type is PDF before trying to extract text
    if filename.endswith('.pdf'):

        # Create a PDF Document object
        try:
            doc = fitz.Document('%s/%s' % (pdf_dir, filename))
        except:
            continue

        # Extract text from the document page-by-page
        text = []

        for page in doc:
            page_text = page.get_text('text')
            if page_text:
                text.append(page_text)

        # If no text is extracted, add the filename to the exceptions list and move on to the next file
        if not text:
            exceptions.append(filename)
            continue
        
        # Create a TXT file for each document using the PDF filename
        # Write each page of document text to the TXT file
        with open('%s/%s' % (txt_dir, filename.replace('pdf', 'txt')), 'w', encoding='utf-8') as file:
            for page in text:
                file.write(page)

# Create a TXT file that lists all files in the exceptions list and save to the directory with TXT files
if exceptions:
    with open('%s/exceptions.txt' % (txt_dir), 'w', encoding='utf-8') as file:
        for filename in exceptions:
            file.write('%s\n' % (filename))
# Check the exceptions list for any files from which text was not extracted successfully
exceptions<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0684285040113261, 'fraction_numerical': 0.002831524303916942, 'mean_word_length': 3.642935377875137, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '530734', 'license': 'MIT', 'n_tokens_mistral': 1160, 'n_tokens_neox': 1129, 'n_words': 549, 'path': 'pdf_to_txt_coverter/pdf_txt_converter.ipynb', 'repo_name': 'tyt3/concordancers', 'size': '4238'}