<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/ABSA_Inference.ipynb)  **Aspect Based Sentiment Analysis in Spark NLP** Model Details: https://nlp.johnsnowlabs.com/2020/12/29/ner_aspect_based_sentiment_en.html  Spark NLP documentation and instructions:https://nlp.johnsnowlabs.com/docs/en/quickstart You can find details about Spark NLP annotators here:https://nlp.johnsnowlabs.com/docs/en/annotators You can find details about Spark NLP models here:https://nlp.johnsnowlabs.com/models  1. Colab Setup Install Dependencies and Libraries<jupyter_code>!wget http://setup.johnsnowlabs.com/colab.sh -O - | bash
# !bash colab.sh
# -p is for pyspark
# -s is for spark-nlp
# !bash colab.sh -p 3.1.1 -s 3.0.1
# by default they are set to the latest

# Install Spark NLP Display lib
! pip install --upgrade -q spark-nlp-display<jupyter_output>openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.18.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.18.04, mixed mode, sharing)
setup Colab for PySpark 3.1.1 and Spark NLP 3.0.0
[K     |████████████████████████████████| 212.3MB 62kB/s 
[K     |████████████████████████████████| 143kB 53.7MB/s 
[K     |████████████████████████████████| 204kB 41.3MB/s 
[?25h  Building wheel for pyspark (setup.py) ... [?25l[?25hdone
[K     |████████████████████████████████| 102kB 3.8MB/s 
[K     |████████████████████████████████| 71kB 4.4MB/s 
[?25h<jupyter_text>Import and start the Spark session<jupyter_code>import pandas as pd
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

import sparknlp
from sparknlp.annotator import *
from sparknlp.base import *

spark = sparknlp.start()

# manually start session
'''
spark = SparkSession.builder \
    .appName('Spark NLP Licensed') \
    .master('local[*]') \
    .config('spark.driver.memory', '16G') \
    .config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') \
    .config('spark.kryoserializer.buffer.max', '2000M') \
    .config('spark.jars.packages', 'com.johnsnowlabs.nlp:spark-nlp_2.11:' +sparknlp.version()).getOrCreate()
'''<jupyter_output><empty_output><jupyter_text>2. Build Pipeline<jupyter_code>document_assembler = DocumentAssembler() \
    .setInputCol('text')\
    .setOutputCol('document')

sentence_detector = SentenceDetector() \
    .setInputCols(['document'])\
    .setOutputCol('sentence')

tokenizer = Tokenizer()\
    .setInputCols(['sentence']) \
    .setOutputCol('token')

word_embeddings = WordEmbeddingsModel.pretrained("glove_6B_300", "xx")\
    .setInputCols(["document", "token"])\
    .setOutputCol("embeddings")
    
ner_model = NerDLModel.pretrained("ner_aspect_based_sentiment")\
    .setInputCols(["document", "token", "embeddings"])\
    .setOutputCol("ner")

ner_converter = NerConverter()\
    .setInputCols(['sentence', 'token', 'ner']) \
    .setOutputCol('ner_chunk')

nlp_pipeline = Pipeline(stages=[
    document_assembler, 
    sentence_detector,
    tokenizer,
    word_embeddings,
    ner_model,
    ner_converter])

empty_df = spark.createDataFrame([['']]).toDF('text')
pipeline_model = nlp_pipeline.fit(empty_df)
light_pipeline = LightPipeline(pipeline_model)<jupyter_output>glove_6B_300 download started this may take some time.
Approximate size to download 426.2 MB
[OK!]
ner_aspect_based_sentiment download started this may take some time.
Approximate size to download 21.3 MB
[OK!]<jupyter_text>3. Create example inputs<jupyter_code># Enter examples as strings in this array
input_list = [
    """From the beginning, we were met by friendly staff members, and the convienent parking at Chelsea Piers made it easy for us to get to the boat."""]<jupyter_output><empty_output><jupyter_text>4. Run the pipeline Full Pipeline (Expects a spark Data Frame)<jupyter_code>df = spark.createDataFrame(pd.DataFrame({"text": input_list}))
result = pipeline_model.transform(df)<jupyter_output><empty_output><jupyter_text>Light Pipeline (Expects a list of string)<jupyter_code>lresult = light_pipeline.fullAnnotate(input_list)<jupyter_output><empty_output><jupyter_text>5. Visualize results Full Pipeline Result<jupyter_code># Using display lib
from sparknlp_display import NerVisualizer

NerVisualizer().display(result.collect()[0], 'ner_chunk', 'document')
# Process manually
exploded = F.explode(F.arrays_zip('ner_chunk.result', 'ner_chunk.metadata'))
select_expression_0 = F.expr("cols['0']").alias("chunk")
select_expression_1 = F.expr("cols['1']['entity']").alias("ner_label")
result.select(exploded.alias("cols")) \
    .select(select_expression_0, select_expression_1).show(truncate=False)
result = result.toPandas()<jupyter_output>+-------------+---------+
|chunk        |ner_label|
+-------------+---------+
|staff members|POS      |
+-------------+---------+<jupyter_text>Light Pipeline Result<jupyter_code># Using display lib
from sparknlp_display import NerVisualizer

NerVisualizer().display(lresult[0], 'ner_chunk', 'document')
# Process manually
for example in lresult:
  for res in example['ner_chunk']:
    print ('Token/Phrase:', res.result, 'Sentiment: ', res.metadata['entity'])<jupyter_output>Token/Phrase: staff members Sentiment:  POS
{'fraction_non_alphanumeric': 0.17215376126540372, 'fraction_numerical': 0.024094169578811844, 'mean_word_length': 5.130777903043969, 'pattern_counts': {'":': 1, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '186430', 'license': 'Apache-2.0', 'n_tokens_mistral': 2142, 'n_tokens_neox': 1913, 'n_words': 376, 'path': 'tutorials/streamlit_notebooks/ABSA_Inference.ipynb', 'repo_name': 'hatrungduc/spark-nlp-workshop', 'size': '5437'}