<jupyter_start><jupyter_text>How can I leverage State-of-the-Art Natural Language Models with only one line of code ? Newly introduced in transformers v2.3.0, **pipelines** provides a high-level, easy to use,API for doing inference over a variety of downstream-tasks, including: - Sentence Classification (Sentiment Analysis): Indicate if the overall sentence is either positive or negative. _(Binary Classification task or Logitic Regression task)_- Token Classification (Named Entity Recognition, Part-of-Speech tagging): For each sub-entities _(**tokens**)_ in the input, assign them a label _(Classification task)_.- Question-Answering: Provided a tuple (question, context) the model should find the span of text in **content** answering the **question**.- Mask-Filling: Suggests possible word(s) to fill the masked input with respect to the provided **context**.- Feature Extraction: Maps the input to a higher, multi-dimensional space learned from the data.Pipelines encapsulate the overall process of every NLP process:  1. Tokenization: Split the initial input into multiple sub-entities with ... properties (i.e. tokens). 2. Inference: Maps every tokens into a more meaningful representation.  3. Decoding: Use the above representation to generate and/or extract the final output for the underlying task.The overall API is exposed to the end-user through the `pipeline()` method with the following structure:```pythonfrom transformers import pipeline Using default model and tokenizer for the taskpipeline("") Using a user-specified modelpipeline("", model="") Using custom model/tokenizer as strpipeline('', model='', tokenizer='')```<jupyter_code>!pip install transformers
from __future__ import print_function
import ipywidgets as widgets
from transformers import pipeline<jupyter_output><empty_output><jupyter_text>1. Sentence Classification - Sentiment Analysis<jupyter_code>nlp_sentence_classif = pipeline('sentiment-analysis')
nlp_sentence_classif('Such a nice weather outside !')<jupyter_output><empty_output><jupyter_text>2. Token Classification - Named Entity Recognition<jupyter_code>nlp_token_class = pipeline('ner')
nlp_token_class('Hugging Face is a French company based in New-York.')<jupyter_output><empty_output><jupyter_text>3. Question Answering<jupyter_code>nlp_qa = pipeline('question-answering')
nlp_qa(context='Hugging Face is a French company based in New-York.', question='Where is based Hugging Face ?')<jupyter_output><empty_output><jupyter_text>4. Text Generation - Mask Filling<jupyter_code>nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in <mask>')<jupyter_output><empty_output><jupyter_text>5. Projection - Features Extraction<jupyter_code>import numpy as np
nlp_features = pipeline('feature-extraction')
output = nlp_features('Hugging Face is a French company based in Paris')
np.array(output).shape   # (Samples, Tokens, Vector Size)<jupyter_output><empty_output><jupyter_text>Alright ! Now you have a nice picture of what is possible through transformers' pipelines, and there is moreto come in future releases. In the meantime, you can try the different pipelines with your own inputs<jupyter_code>task = widgets.Dropdown(
    options=['sentiment-analysis', 'ner', 'fill_mask'],
    value='ner',
    description='Task:',
    disabled=False
)

input = widgets.Text(
    value='',
    placeholder='Enter something',
    description='Your input:',
    disabled=False
)

def forward(_):
    if len(input.value) > 0: 
        if task.value == 'ner':
            output = nlp_token_class(input.value)
        elif task.value == 'sentiment-analysis':
            output = nlp_sentence_classif(input.value)
        else:
            if input.value.find('<mask>') == -1:
                output = nlp_fill(input.value + ' <mask>')
            else:
                output = nlp_fill(input.value)                
        print(output)

input.on_submit(forward)
display(task, input)
context = widgets.Textarea(
    value='Einstein is famous for the general theory of relativity',
    placeholder='Enter something',
    description='Context:',
    disabled=False
)

query = widgets.Text(
    value='Why is Einstein famous for ?',
    placeholder='Enter something',
    description='Question:',
    disabled=False
)

def forward(_):
    if len(context.value) > 0 and len(query.value) > 0: 
        output = nlp_qa(question=query.value, context=context.value)            
        print(output)

query.on_submit(forward)
display(context, query)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09874366321357725, 'fraction_numerical': 0.0033061494379545957, 'mean_word_length': 4.80306905370844, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '52568', 'license': 'Apache-2.0', 'n_tokens_mistral': 1309, 'n_tokens_neox': 1257, 'n_words': 424, 'path': 'notebooks/03-pipelines.ipynb', 'repo_name': 'mittalgovind/transformers', 'size': '4537'}