<jupyter_start><jupyter_text>Installing Python Packages to Existing EMR Clusters Notebook Scope LibrariesNotebook-scoped libraries provide you the following benefits:* Runtime installation – You can import your favorite Python libraries from PyPI repositories and install them on your remote cluster (driver and executors) on the fly when you need them. These libraries are instantly available to your Spark runtime environment. There is no need to restart the notebook session or recreate your cluster.* Dependency isolation – The libraries you install using EMR Notebooks are isolated to your notebook session and don’t interfere with bootstrapped cluster libraries or libraries installed from other notebook sessions. These notebook-scoped libraries take precedence over bootstrapped libraries. Multiple notebook users can import their preferred version of the library and use it without dependency clashes on the same cluster.* Portable library environment – The library package installation happens from your notebook file. This allows you to recreate the library environment when you switch the notebook to a different cluster by re-executing the notebook code. At the end of the notebook session, the libraries you install through EMR Notebooks are automatically removed from the hosting EMR cluster. This functionality is available for clusters running EMR release >= 5.26.0. More info can be found in the [docs here](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-installing-libraries-and-kernels.html)<jupyter_code>%%info<jupyter_output><empty_output><jupyter_text>To enable notebook scoped libraries, we must set the configuration to use a virtualenv<jupyter_code>%%configure -f
{ "conf":{
          "spark.pyspark.python": "python3",
          "spark.pyspark.virtualenv.enabled": "true",
          "spark.pyspark.virtualenv.type":"native",
          "spark.pyspark.virtualenv.bin.path":"/usr/bin/virtualenv"
         }
}
sc.list_packages()<jupyter_output><empty_output><jupyter_text>Notice that there is no "translate" package available on our cluster<jupyter_code># Install translate from given PyPI repository
sc.install_pypi_package("translate", "https://pypi.org/simple") 
from translate import Translator
from pyspark.sql.functions import udf, col

@udf
def translate_to_german(sentence):
    translator= Translator(to_lang="de")
    return translator.translate(sentence)
columns = ["TaskID","Sentence"]
data = [("1", "This is a pen"),
    ("2", "This is a chair")]

df = spark.createDataFrame(data=data,schema=columns)

df.show(truncate=False)
df.select(col("TaskID"), \
    translate_to_german(col("Sentence")).alias("Sentence") ) \
   .show(truncate=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08150584795321637, 'fraction_numerical': 0.0025584795321637425, 'mean_word_length': 5.455188679245283, 'pattern_counts': {'":': 5, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '446311', 'license': 'MIT-0', 'n_tokens_mistral': 742, 'n_tokens_neox': 708, 'n_words': 279, 'path': 'interactive_notebooks/install-py-libraries.ipynb', 'repo_name': 'ericzbeard/sagemaker-studio-emr', 'size': '2736'}