<jupyter_start><jupyter_text>Noisy Channel Model Spell Checker - Medical   Initial SetupAs it's usual in Spark-NLP let's start with building a pipeline; a _spell correction pipeline_. We will use a pretrained model from our library.<jupyter_code>!export SPARK_NLP_LICENSE="<KEY>"
from sparknlp.annotator import *
from sparknlp.common import *
from sparknlp.base import *
import sparknlp_jsl

from IPython.utils.text import columnize
from sparknlp_jsl.annotator import *
import credentials

beautify = lambda annotations: [columnize(sent['checked']) for sent in annotations]

# Two options for the Spark session
# spark = sparknlp_jsl.start("XXXXXXXXX") # with license key

jars = "/home/jose/spark-nlp/python/lib/sparknlp.jar,/home/jose/spark-nlp-internal/python/lib/sparknlp-jsl.jar" 
from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("Spell checking") \
    .master("local[*]") \
    .config("spark.driver.memory", "8G") \
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")\
    .config("spark.jars", jars)\
    .config("spark.driver.extraClassPath", jars)\
    .config("spark.executor.extraClassPath", jars)\
    .config("spark.kryoserializer.buffer.max", "500m")\
    .getOrCreate()

documentAssembler = DocumentAssembler()\
  .setInputCol("text")\
  .setOutputCol("document")

tokenizer = RecursiveTokenizer()\
  .setInputCols(["document"])\
  .setOutputCol("token")\
  .setPrefixes(["\"", "(", "[", "\n"])\
  .setSuffixes([".", ",", "?", ")","!", "'s"])

spellModel = ContextSpellCheckerModel\
    .pretrained('spellcheck_clinical', 'en', 'clinical/models')\
    .setInputCols("token")\
    .setOutputCol("checked")
finisher = Finisher()\
    .setInputCols("checked")

pipeline = Pipeline(
    stages = [
    documentAssembler,
    tokenizer,
    spellModel,
    finisher
  ])

empty_ds = spark.createDataFrame([[""]]).toDF("text")
lp = LightPipeline(pipeline.fit(empty_ds))<jupyter_output>com.johnsnowlabs.nlp.Finisher
org.apache.spark.ml.PipelineModel
com.johnsnowlabs.nlp.LightPipeline<jupyter_text>Ok!, at this point we have our spell checking pipeline as expected. Let's see what we can do with it, see these errors,___Witth__ the __hell__ of __phisical__ __terapy__ the patient was __imbulated__ and on posoperative, the __impatient__ tolerating a post __curgical__ soft diet.__With __paint__ __wel__ controlled on __orall__ pain medications, she was discharged __too__ __reihabilitation__ __facilitay__.__She is to also call the __ofice__ if she has any __ever__ greater than 101, or __leeding__ __form__ the surgical wounds.__Abdomen is __sort__, nontender, and __nonintended__.__Patient not showing pain or any __wealth__ problems._            _No __cute__ distress_Check that some of the errors are valid English words, only by considering the context the right choice can be made.<jupyter_code># check for the different occurrences of the word "siter"
example1 = ["Witth the hell of phisical terapy the patient was imbulated and on posoperative, the impatient tolerating a post curgical soft diet.",
            "With paint wel controlled on orall pain medications, she was discharged too reihabilitation facilitay.",
            "She is to also call the ofice if she has any ever greater than 101, or leeding form the surgical wounds.",
            "Abdomen is sort, nontender, and nonintended.",
            "Patient not showing pain or any wealth problems.",
            "No cute distress"
            
]
lp.annotate(example1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10667798528579513, 'fraction_numerical': 0.003395585738539898, 'mean_word_length': 4.413476263399693, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '637541', 'license': 'Apache-2.0', 'n_tokens_mistral': 1139, 'n_tokens_neox': 1087, 'n_words': 324, 'path': 'tutorials/Certification_Trainings/Healthcare/6.Noisy_Channel_Spell_Checker_Medical.ipynb', 'repo_name': 'nabinkhadka/spark-nlp-workshop', 'size': '3989'}