<jupyter_start><jupyter_text><jupyter_code>!pip install transformers
!pip install torch
!pip install sentencepiece
!pip install pdfminer.six<jupyter_output>Requirement already satisfied: transformers in /usr/local/lib/python3.6/dist-packages (4.0.0)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.5)
Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers) (0.0.43)
Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.8)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)
Requirement already satisfied: tokenizers==0.9.4 in /usr/local/lib/python3.6/dist-packages (from transformers) (0.9.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages [...]<jupyter_text>Importing required packages<jupyter_code>import torch
import json 
from transformers import T5Tokenizer, T5ForConditionalGeneration, T5Config
#import pdfminer
import os
import io
import io
from pdfminer.converter import TextConverter
from pdfminer.pdfinterp import PDFPageInterpreter
from pdfminer.pdfinterp import PDFResourceManager
from pdfminer.pdfpage import PDFPage
#print(pdfminer.__version__)
def pre_process_text(text):
  preprocess_text = text.strip().replace("\n","")
  t5_prepared_Text = "summarize: "+preprocess_text
  #print ("original text preprocessed: \n", preprocess_text)
  return preprocess_text,t5_prepared_Text
def get_summary(t5_prepared_Text):
  tokenized_text = tokenizer.encode(t5_prepared_Text, return_tensors="pt").to(device)
  
  # summmarize 
  summary_ids = model.generate(tokenized_text,
                                      num_beams=4,
                                      no_repeat_ngram_size=2,
                                      min_length=30,
                                      max_length=100,
                                      early_stopping=True)
  output = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
  return output
  
#using the t5-base pretrained checkpoint trained on 220 million parameters
model = T5ForConditionalGeneration.from_pretrained('t5-base')
tokenizer = T5Tokenizer.from_pretrained('t5-base')
device = torch.device('cpu')

def extract_text_by_page(pdf_path):
    with open(pdf_path, 'rb') as fh:
        for page in PDFPage.get_pages(fh, 
                                      caching=True,
                                      check_extractable=True):
            resource_manager = PDFResourceManager()
            fake_file_handle = io.StringIO()
            converter = TextConverter(resource_manager, fake_file_handle)
            page_interpreter = PDFPageInterpreter(resource_manager, converter)
            page_interpreter.process_page(page)
            
            text = fake_file_handle.getvalue()
            yield text
    
            # close open handles
            converter.close()
            fake_file_handle.close()

def extract_text(pdf_path):
  final_summary = ""
  page_count = 1
  for page in extract_text_by_page(pdf_path):
    #print(page)
    #print(type(page))
    text = page
    preprocess_text, t5_prepared_Text = pre_process_text(text)
    #print ("original text preprocessed: \n", preprocess_text)
    output = get_summary(t5_prepared_Text)
    #print ("\n\nSummarized text: \n",output)
    print()
    final_summary += output
    #print(final_summary)
    final_summary += "\n\n"
    print("Done Summarizing Page----------->>> " + str(page_count))
    page_count += 1
  print("\n\nSummarization done. Printing summary.\n")
  return final_summary

def read_pdf_paths(folder_name, output_folder):
  path_to_files = os.listdir(os.path.abspath('.') + '/' + folder_name)
  for entry in path_to_files:
    if entry.split('.')[-1] == 'pdf':
      print("Processing Pdf : " + entry)
      final_summary = extract_text(os.path.abspath('.') + '/' + folder_name + '/' + entry)
      with open(os.path.abspath('.') + '/' + output_folder + '/Summary_' + entry.split('.')[-2] + '.txt', 'w' ) as f:
        f.write("Summary of " + entry.split('.')[-2]+":\n\n" )
        f.write("\n\n===========================================\n\n" )
        f.write(final_summary)
      #print(entry)
      print("Summary of " + entry.split('.')[-2]+" is :\n\n")
      print(final_summary)<jupyter_output><empty_output><jupyter_text>Output: The code below created the input and outputs folders for the model to take inputs from and then reads each pdf page by page to extract summaries and then saves the summaries as plain text files. Steps for running the script:1. Run all the cells above this point. 2.   Run the cell below to create the input folder containing the Pdfs with the name `pdf_folder` and the output folder that'll contain the output summaries as text files with the name `summary_folder`.Note : This step could be skipped if these folders are already present or if you wish to manually create these folders.3.   Populate the `pdf_folder` with the Pdfs to be summarized.<jupyter_code>pdf_folder = '/pdf_folder'
if not os.path.exists(os.path.abspath('.') + pdf_folder):
  os.mkdir(os.path.abspath('.') + pdf_folder)
summary_folder = '/summary_folder'
if not os.path.exists(os.path.abspath('.') + summary_folder):
  os.mkdir(os.path.abspath('.') + summary_folder)<jupyter_output><empty_output><jupyter_text>3.   Run the code below with the appropriate folder names created in the previous step. The program will read all the Pdfs present in the `pdf_folder` and output the results in the `summary_folder`.<jupyter_code>read_pdf_paths('pdf_folder', 'summary_folder')<jupyter_output>Processing Pdf : The-COVID-19-recovery-will-be-digital-A-plan-for-the-first-90-days-vF_677b26fd2626e24c8ab1bf28ed70410e.pdf

Done Summarizing Page----------->>> 1

Done Summarizing Page----------->>> 2

Done Summarizing Page----------->>> 3

Done Summarizing Page----------->>> 4

Done Summarizing Page----------->>> 5

Done Summarizing Page----------->>> 6

Done Summarizing Page----------->>> 7

Done Summarizing Page----------->>> 8


Summarization done. Printing summary.

Summary of The-COVID-19-recovery-will-be-digital-A-plan-for-the-first-90-days-vF_677b26fd2626e24c8ab1bf28ed70410e is :


COVID-19 recovery will be digital: A plan for the first 90 days. rapid migration to digital technologies driven by the pandemic will continue into the recovery.

digital will undoubtably play a center-stage role during the recovery and beyond. banks have transitioned to remote sales and service teams, schools have pivoted to online learning and digital classrooms, doctors have begun delivering telem[...]
{'fraction_non_alphanumeric': 0.12232591529073941, 'fraction_numerical': 0.023115577889447236, 'mean_word_length': 3.48840206185567, 'pattern_counts': {'":': 1, '<': 16, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '483280', 'license': 'MIT', 'n_tokens_mistral': 2211, 'n_tokens_neox': 2121, 'n_words': 556, 'path': 'T5_Abstractive_Summarizer.ipynb', 'repo_name': 'null-buster/PDF_Summarizer', 'size': '6965'}