<jupyter_start><jupyter_text><jupyter_code>!pip install torch==1.8.2+cu111 torchvision==0.9.2+cu111 torchaudio===0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html
!pip install transformers
from transformers import PegasusForConditionalGeneration, PegasusTokenizer<jupyter_output><empty_output><jupyter_text><jupyter_code>pip install sentencepiece
tokenizer = PegasusTokenizer.from_pretrained("google/pegasus-xsum")
model = PegasusForConditionalGeneration.from_pretrained("google/pegasus-xsum")

text = """  Nigeria (/naɪˈdʒɪəriə/ (About this soundlisten)), officially the Federal Republic of Nigeria, is a country in West Africa. It is the most populous country in Africa; geographically situated between the Sahel to the north, and the Gulf of Guinea to the south in the Atlantic Ocean; covering an area of 923,769 square kilometers, with a population of over 211 million. It borders Niger in the north, Chad in the northeast, Cameroon in the east, and Benin in the west. Nigeria is a federal republic comprising 36 states and the Federal Capital Territory, where the capital, Abuja, is located. The largest city in Nigeria is Lagos, one of the largest metropolitan areas in the world and second largest in Africa.

Nigeria has been home to several indigenous pre-colonial states and kingdoms since the second millennium BC, with the Nok civilization in the 15th century BC marking the first internal unification in the country. The modern state originated with British colonialization in the 19th century, taking its present territorial shape with the merging of the Southern Nigeria Protectorate and Northern Nigeria Protectorate in 1914 by Lord Lugard. The British set up administrative and legal structures while practising indirect rule through traditional chiefdoms.[8] Nigeria became a formally independent federation on October 1, 1960. It experienced a civil war from 1967 to 1970, followed by a succession of democratically-elected civilian governments and military dictatorships, until achieving a stable democracy in the 1999 presidential election; the 2015 election was the first time an incumbent president had lost re-election.[9]  """
tokens = tokenizer(text, truncation=True, padding="longest", return_tensors="pt")
tokens = tokenizer(text, truncation=True, padding="longest", return_tensors="pt")
tokens
summary = model.generate(**tokens)
summary[0]
tokenizer.decode(summary[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06265356265356266, 'fraction_numerical': 0.024979524979524978, 'mean_word_length': 6.164222873900293, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '619511', 'license': 'MIT', 'n_tokens_mistral': 687, 'n_tokens_neox': 617, 'n_words': 289, 'path': 'Text_Sumirization_Using_Pegasus_Paraphras.ipynb', 'repo_name': 'Layusmen/Deep-Learning', 'size': '2442'}