<jupyter_start><jupyter_text>InstallationBefore we start we should install asent this can be done simply by commenting the following lines out:<jupyter_code>!pip install asent
!python -m spacy download en_core_web_lg<jupyter_output>Defaulting to user installation because normal site-packages is not writeable
Collecting asent
  Downloading asent-0.1.3-py3-none-any.whl (241 kB)
[K     |████████████████████████████████| 241 kB 4.5 MB/s 
[?25hRequirement already satisfied: spacy<3.3.0,>=3.0.0 in /home/coder/.local/lib/python3.7/site-packages (from asent) (3.2.1)
Requirement already satisfied: matplotlib<4.0.0,>=3.5.0 in /home/coder/.local/lib/python3.7/site-packages (from asent) (3.5.1)
Requirement already satisfied: numpy>=1.15.0 in /home/coder/.local/lib/python3.7/site-packages (from spacy<3.3.0,>=3.0.0->asent) (1.19.1)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/coder/.local/lib/python3.7/site-packages (from spacy<3.3.0,>=3.0.0->asent) (4.49.0)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from spacy<3.3.0,>=3.0.0->asent) (40.8.0)
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /home/coder/.local/lib/python3.7/site-packages (from spacy<3.3.0,>=3.0.0->a[...]<jupyter_text>Tutorial> *Note*: This tutorial is English but the library also allows for multiple other languages to see all languages available check out the [Languages section](https://kennethenevoldsen.github.io/asent/languages/index.html) on the website.Asent is a package for fast and transparent sentiment analysis. The package applied uses a dictionary of words rated as either positive or negative and a series of rules to determine whether a word, sentence or a document is positive or negative. The current rules account for negations (i.e. "not happy"), intensifiers ("very happy") and account for contrastive conjugations (i.e. "but") as well as other emphasis markers such as exclamation marks, casing and question marks. The following will take you through how the sentiment is calculated in a step by step fashion.To start of with we will need a spaCy pipeline as well as we will need to add the asent pipeline `asent_en_v1` to it, where `en` indicate that it is the English pipeline and that `v1` indicate that it is version 1.<jupyter_code>import asent
import spacy

# load spacy pipeline
nlp = spacy.load("en_core_web_lg")

# add the rule-based sentiment model
nlp.add_pipe("asent_en_v1")<jupyter_output><empty_output><jupyter_text>If you want to see all the available components you can simply run:<jupyter_code>for c in asent.components.get_all():
    print(c)<jupyter_output>asent_da_v1
asent_en_v1
asent_no_v1
asent_sv_v1<jupyter_text>Token valence and polarityAs seen in figure 1. token valence is simply the value gained from a lookup in a rated dictionary. For instance if the have the example sentence "I am not very happy" the word "happy" has a positive human rating of 2.7 which is not amplified by the word being in all-caps.    Figure 1: Calculation of token polarity and valence  We can extract valence quite easily using the `valence` extension:<jupyter_code>doc = nlp("I am not very happy.")

for token in doc:
    print(token, "\t", token._.valence)<jupyter_output>I 	 0.0
am 	 0.0
not 	 0.0
very 	 0.0
happy 	 2.7
. 	 0.0<jupyter_text>Naturally, in this context happy should not be perceived positively as it is negated, thus we should look at token polarity. Token polarity examines if a word is negated and, if so, multiplies the values by a negative constant. This constant is emperically derived to be 0.74 [(Hutto and Gilbert, 2014)](https://ojs.aaai.org/index.php/ICWSM/article/view/14550). Similarly with the specific example we chose we can also see that "happy" is intensified by the word "very", while increases it polarity. The constant 0.293 is also emperically derived by Hutto and Gilbert. We can similarly extract the polarity using the `polarity` extension:<jupyter_code>for token in doc:
    print(token._.polarity)<jupyter_output>polarity=0.0 token=I span=I
polarity=0.0 token=am span=am
polarity=0.0 token=not span=not
polarity=0.0 token=very span=very
polarity=-2.215 token=happy span=not very happy
polarity=0.0 token=. span=.<jupyter_text>Notice that here we even get further information, that token "happy", has a polarity of -2.215 and that this includes the span (sequence of tokens) "not very happy".  Visualizing polarityAsent also include a series of methods to visualize the token polarity:<jupyter_code>asent.visualize(doc)<jupyter_output><empty_output><jupyter_text>Document and Span PolarityWe want to do more than simply calculate the polarity of the token, we want to extract information about the entire sentence (span) and aggregate this across the entire document.    Figure 2: Calculation of document polarity  The calculation of the sentence polarity includes a couple of steps. First, it checks if the sentence contains a contrastive conjugation (e.g. "but"), then overweighs things after the but and underweighs previous elements. This seems quite natural for e.g. the sentence "The movie was great, but the acting was horrible", where the second statement is noticeably more important. This has also been shown empirically by [(Hutto and Gilbert, 2014)](https://ojs.aaai.org/index.php/ICWSM/article/view/14550). Afterwards, the model takes into account question marks and exclamations marks, which both increase the polarity of the sentence – negative sentences become more negative and positive sentences become less negative. Lastly, the polarity is normalized between approximately -1 and 1.You can easily extract the sentence polarity and the document polarity using:<jupyter_code>for sentence in doc.sents:
    print(sentence._.polarity)
# or for multiple sentences:
print(doc._.polarity)<jupyter_output>neg=0.391 neu=0.609 pos=0.0 compound=-0.4964
{'fraction_non_alphanumeric': 0.09083162917518746, 'fraction_numerical': 0.030845262440354466, 'mean_word_length': 4.9765784114052956, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 38, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '105739', 'license': 'MIT', 'n_tokens_mistral': 1903, 'n_tokens_neox': 1761, 'n_words': 712, 'path': 'tutorials/introduction.ipynb', 'repo_name': 'martbern/asent', 'size': '5868'}