<jupyter_start><jupyter_text>What is NLP?NLP is a field of linguistics and machine learning focused on understanding everything related to human language. The aim of NLP tasks is not only to understand single words individually, but to be able to understand the context of those words.The following is a list of common NLP tasks, with some examples of each:- Classifying whole sentences: Getting the sentiment of a review, detecting if an email is spam, determining if a sentence is grammatically correct or whether two sentences are logically related or not - Classifying each word in a sentence: Identifying the grammatical components of a sentence (noun, verb, adjective), or the named entities (person, location, organization)- Generating text content: Completing a prompt with auto-generated text, filling in the blanks in a text with masked words- Extracting an answer from a text: Given a question and a context, extracting the answer to the question based on the information provided in the context- Generating a new sentence from an input text: Translating a text into another language, summarizing a textNLP isn’t limited to written text though. It also tackles complex challenges in speech recognition and computer vision, such as generating a transcript of an audio sample or a description of an image.  PipelineThe most basic object in the 🤗 Transformers library is the **pipeline**. It connects a model with its necessary preprocessing and postprocessing steps, allowing us to directly input any text and get an intelligible answer<jupyter_code>from transformers import pipeline
classifier = pipeline('sentiment-analysis')
classifier("I've been waiting for a HuggingFace course my whole life.")
## passing multiple sentences

classifier([
    "I've been waiting for a HuggingFace course my whole life.", 
    "I hate this so much!"
])<jupyter_output><empty_output><jupyter_text>Some of the currently [available pipelines](https://huggingface.co/transformers/main_classes/pipelines.html) are:- feature-extraction (get the vector representation of a text)- fill-mask- ner (named entity recognition)- question-answering- sentiment-analysis- summarization- text-generation- translation- zero-shot-classification<jupyter_code>## Zero Shot Classification

zero_shot_classifer = pipeline('zero-shot-classification')
zero_shot_classifer(
    "This is a sample course about Hugging face and transformers.",
    candidate_labels = ['education','politics','sports']
)<jupyter_output><empty_output><jupyter_text>This pipeline is called zero-shot because you don’t need to fine-tune the model on your data to use it. It can directly return probability scores for any list of labels you want!<jupyter_code>## Text generation

text_gen_classifier = pipeline("text-generation")
text_gen_classifier("In this course, we will teach you how to")
text_gen_classifier("In this course, we will teach you how to", num_return_sequences=2, max_length=15)<jupyter_output>Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.<jupyter_text>Using models from Hugging Face Hub in a pipelineModel hub: https://huggingface.co/models<jupyter_code>## using distilgpt2 to generate text

generator = pipeline('text-generation', model='distilgpt2')
generator("My name is Rishu and i", max_length=30, num_return_sequences=2)
## Filling masks. Predicting new masks words in the models

unmasker = pipeline("fill-mask")
unmasker("This course will teach you all about <mask> models.", top_k=2)
### Named entity recognition

ner = pipeline('ner', grouped_entities=True)
ner("My name is Sylvain and I work at Hugging Face in Brooklyn.")
## Question Answering

question_answerer = pipeline("question-answering")
question_answerer(
    question="Where do I work?",
    context="My name is Sylvain and I work at Hugging Face in Brooklyn"
)
# Summarization

summarizer = pipeline("summarization")
summarizer("""
    America has changed dramatically during recent years. Not only has the number of 
    graduates in traditional engineering disciplines such as mechanical, civil, 
    electrical, chemical, and aeronautical engineering declined, but in most of 
    the premier American universities engineering curricula now concentrate on 
    and encourage largely the study of engineering science. As a result, there 
    are declining offerings in engineering subjects dealing with infrastructure, 
    the environment, and related issues, and greater concentration on high 
    technology subjects, largely supporting increasingly complex scientific 
    developments. While the latter is important, it should not be at the expense 
    of more traditional engineering.

    Rapidly developing economies such as China and India, as well as other 
    industrial countries in Europe and Asia, continue to encourage and advance 
    the teaching of engineering. Both China and India, respectively, graduate 
    six and eight times as many traditional engineers as does the United States. 
    Other industrial countries at minimum maintain their output, while America 
    suffers an increasingly serious decline in the number of engineering graduates 
    and a lack of well-educated engineers.
""")
## Translation

translator = pipeline("translation", model="Helsinki-NLP/opus-mt-fr-en")
translator("Ce cours est produit par Hugging Face.")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06139205075573801, 'fraction_numerical': 0.0026124276917335325, 'mean_word_length': 5.254375729288214, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '33738', 'license': 'Apache-2.0', 'n_tokens_mistral': 1363, 'n_tokens_neox': 1302, 'n_words': 649, 'path': 'hugging-face/1. Introduction-hugging-face (Pipeline).ipynb', 'repo_name': 'rishuatgithub/MLPy', 'size': '5359'}