<jupyter_start><jupyter_text>Using Jupyter Notebooks with HDInsight Spark KernelsApache Spark cluster on HDInsight (Linux) includes Jupyter notebooks that you can use to test your applications. By default Jupyter notebook comes with **pyspark**. A kernel is a program that runs and interprets your code. The ones provided in the notebook service are the language kernel (python, scala and R, respectively) along with the paths and environmental variables defined for Spark Sessions. HDInsight Spark clusters provide two additional kernels that you can use with the Jupyter notebook. These are:1. **PySpark 3** (for applications written in Python 3). PySpark kernel exposes the Spark programming model to Python2. **Spark** (for applications written in Scala)3. **SparkR** for spark applications written in R    - note, this is **NOT** the environment to use for `RxSpark`. Those applications must be submitted through the edge node, which can be accessed through RStudio Server or through a remote context. ------------ How do I use the new kernels? 1. Create a notebook with the new kernels. Click **New**, and then click **PySpark** or **Spark**. ![Create notebooks with new kernels](https://mysstorage.blob.core.windows.net/notebookimages/overview/jupyter-kernels.png "Create notebooks with new kernels") 2. This should open a new notebook with the kernel you selected. ---------- Notebook setupWhen using PySpark kernel notebooks on HDInsight, there is no need to create a SparkContext or a SparkSession; a SparkSession which has the SparkContext is created for you automatically when you run the first code cell, and you'll be able to see the progress printed. The contexts are created with the following variable names:- SparkSession (spark)To run the cells below, place the cursor in the cell and then press **SHIFT + ENTER**. Everytime you run a cell, your web browser window title will show a **(Busy)** status along with the notebook title. You will also see a solid circle next to the **PySpark** text in the top-right corner. After the job completes, this will change to a hollow circle.![Status of a Jupyter notebook job](https://mysstorage.blob.core.windows.net/notebookimages/overview/HDI.Spark.Jupyter.Job.Status.png "Status of a Jupyter notebook job") ---------- PySpark magics The PySpark kernel provides some predefined “magics”, which are special commands that you can call with `%%` (e.g. `%%MAGIC` ). The magic command must be the first word in a code cell and allow for multiple lines of content. You can’t put comments before a cell magic.For more information on magics, see [here](http://ipython.readthedocs.org/en/stable/interactive/magics.html).  Help magic (%%help)This magic gives you information about the different supported magics in PySpark kernel and the usage for each. bb<jupyter_code>%%help<jupyter_output><empty_output><jupyter_text>Session information (%%info)[Livy](https://github.com/hdinsight/livy) is an open source REST server for Spark. When you execute a code cell in a PySpark notebook, it creates a Livy session to execute your code. You can use the `%%info` magic to display the current Livy session information.<jupyter_code>%%info<jupyter_output><empty_output><jupyter_text>Session configuration (%%configure) Use the `%%configure` magic to configure new or existing Livy sessions.* If a session is already running, you can change the configuration by using the `-f` argument with `%%configure` magic. This will delete the current session and recreate it with the applied configurations. If you don't provide the `-f` argument, an error will be displayed and no configuration changes will be applied.* If you haven't already started the session, then the `-f` argument is not mandatory. Even if you use it with a session that you are just creating, it will not delete any currently running sessions.These are some session attributes that can be used for configuration - **"name"**: Name of the application- **"driverMemory"**: Memory for driver (e.g. 1000M, 2G) - **"executorMemory"**: Memory for executor (e.g. 1000M, 2G) For more attributes for session configuration see  the Livy documentation.> **TIP**: The application name should start with `remotesparkmagics` to allow sessions to get automatically cleaned up if an error happened. If you provide a name that does not start with `remotesparkmagics` it will not result in an error but the cleanup won't occur.<jupyter_code>%%configure -f 
{"name":"remotesparkmagics-sample", "executorMemory": "4G", "executorCores":4}<jupyter_output><empty_output><jupyter_text>---------- Automatic visualization of queries The Pyspark kernel automatically visualizes the output of SQL (HiveQL) queries. You are given the option to choose between several different types of visualizations:- Table- Pie- Line - Area- Bar>**TIP**: When you perform a SQL query, the number of rows of data that will be included in the result data set will be limited by default to 2500 rows.   SQL magic (%%sql)The HDInsight PySpark kernel supports easy inline HiveQL queries against the SparkSession. The (`-o VARIABLE_NAME`) argument persists the output of the SQL query as a Pandas dataframe on the Jupyter server (e.g. `-o query1` in the example below). This means it'll be available in the local mode which will be explained later. The output will be automatically visualized after you run the cell below.<jupyter_code>%%sql -o query1
SELECT clientid, querytime, deviceplatform, querydwelltime 
FROM hivesampletable
WHERE state = 'Washington' AND devicemake = 'Microsoft'<jupyter_output><empty_output><jupyter_text>In addition to `-o`, a number of other configuration parameters for SQL queries are available (as described in the `%%help` output above). These include:* `-q`. This causes the magic to return `None`, turning off visualizations for that cell. If you don't want to auto-visualize the content of a cell and just want to capture it as a dataframe, then use `-q -o VARIABLE`. If you want to turn off visualizations without capturing the results (e.g. for running a SQL query with side effects, like a `CREATE TABLE` statement), just use `-q` without specifying a `-o` argument.> Remember that the kernel by default limits the output of a SQL query to 2500 rows. If you want to adjust this default behavior, use these options.* `-m METHOD`, where `METHOD` is either `take` or `sample` (defaults to `take`). If the method is `take`, the kernel will take elements from the top of the result data set; if the method is `sample`, the kernel will randomly sample elements of the data set according to `-r`, described below.* `-r FRACTION`, where `FRACTION` is some floating-point number between 0.0 and 1.0. If the sample method for the SQL query is `sample`, then the kernel will randomly sample this fraction of the elements of the result set for you; e.g. if you run a SQL query with the arguments `-m sample -r 0.01`, then 1% of the result rows will be randomly sampled.* `-n MAXROWS`, where `MAXROWS` is some integer. The kernel will limit the number of output rows to `MAXROWS`. If `MAXROWS` is a negative number such as `-1`, then the number of rows in the result set will not be limited.> **WARNING**: Be careful with the `-n` option, as it is possible to cause the Jupyter server to run out of memory if you try to collect too many result rows. We recommend only using `-n -1` if you are certain that the result dataset will not be too large.As a final example, this SQL query randomly samples 10% of the rows in the **hivesampletable**, limits the size of the result set to 500, and saves the output into a dataframe called `query2` without doing any visualizations.<jupyter_code>%%sql -q -m sample -r 0.1 -n 500 -o query2 
SELECT * FROM hivesampletable<jupyter_output><empty_output><jupyter_text>Running locally (%%local)You can use the `%%local` magic to run your code locally on the Jupyter server, which is the headnode of the HDInsight cluster. Here's a typical use case for this scenario. By default, the output of any code snippet that you run from a Jupyter notebook is available within the context of the session that is persisted on the worker nodes. However, if you want to save a trip to the worker nodes for every computation, and all the data that you need for your computation is available locally on the Jupyter server node (which is the headnode), you can use the `%%local` magic to run the code snippet on the Jupyter server. Typically, you would use `%%local` magic in conjunction with the `%%sql` magic with `-o` parameter. The `-o` parameter would persist the output of the SQL query locally and then `%%local` magic would trigger the next set of code snippet to run locally against the output of the SQL queries that is persisted locally.> **TIPS**: > * Working against locally persisted data is especially useful when you want the flexibility of using visualization libraries such as **matplotlib**. If you work directly against the data on the remote worker nodes, the output received through Livy is always text that limits the options of visual representation.> * Remember that SQL queries, by default, limit the number of result rows to 2500 -- and that it is possible to get Out of Memory errors if you accidentally collect too many results from your SQL query. Therefore, if your dataset is large, it is considered a best practice to do your computation or number-crunching on the cluster or in the SQL query rather than in local mode.When you use `%%local` all subsequent lines in the cell will be executed locally. The code in the cell must be valid Python code.This code block prints the length of the result set of the dataframe `query2` -- remember, we used the `-n 500` option to limit the size of the dataframe to 500.<jupyter_code>%%local  
print(len(query2))<jupyter_output><empty_output><jupyter_text>Session logs (%%logs)You can get the logs of your current Livy session to debug any issues you encounter. From the logs, you can retrieve the YARN application id and then use the YARN UI to look at the YARN logs for that application id. Yarn is the resource manager for Spark. URL for Yarn UI is `https://.azurehdinsight.net/yarnui`. For instructions on how to access the YARN UI for the cluster, see [Access YARN application logs on Linux-based HDInsight](https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-access-yarn-app-logs-linux/).<jupyter_code>%%logs<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07174329501915709, 'fraction_numerical': 0.00603448275862069, 'mean_word_length': 4.711706783369803, 'pattern_counts': {'":': 3, '<': 29, '<?xml version=': 0, '>': 36, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '48359', 'license': 'MIT', 'n_tokens_mistral': 2739, 'n_tokens_neox': 2615, 'n_words': 1550, 'path': 'Student-Resources/Labs/ADLS_Samples/.ipynb_checkpoints/00-Using-Jupyter-on-HDI-Effectively-checkpoint.ipynb', 'repo_name': 'nthacker/LearnAnalytics-SparkML', 'size': '10440'}