<jupyter_start><jupyter_text>Demonstrating sparkmagic  This notebook will demonstrate how we can use the spark magic to interspere our Python code with code that is running against a Spark cluster Let’s say we’re working in an IPython notebook and we want to use Spark to analyze some data. So, we'll load `sparkmagic` in order to be able to talk to Spark from our Python notebook.<jupyter_code>%load_ext sparkmagic.magics<jupyter_output><empty_output><jupyter_text>With it, the `%manage_spark` line magic and the `%%spark` magic are available.The %%manage_spark line magic lets you manage Livy endpoints and Spark sessions. Let's start by adding an Endpoint.An Endpoint is a [Livy](https://github.com/cloudera/livy) installation running on a Spark cluster. `sparkmagic` allows us to specify the Livy endpoint along with a username and password to authenticate to it. If the Livy endpoint is on your local machine or has no password, simply leave the text fields for username and password blank.<jupyter_code>%manage_spark<jupyter_output><empty_output><jupyter_text>![add_endpoint](images/addendpoint.PNG) Now, add a session to the endpoint you added. The name you give to the session will be used with the `%%spark` magic to run Spark code. You can also specify the configuration you want to start the session with. You can create either Python (PySpark) or Scala (Spark) sessions.Creating a session will create a `SparkContext` with the name `sc` and a `HiveContext` with the name `sqlContext`.We'll start by adding a PySpark session. ![add_session](images/addsession.PNG) You can now run Spark code against your Livy session.  For information on the available commands, run %spark?<jupyter_code>%spark?<jupyter_output><empty_output><jupyter_text>Pyspark You can run code against your Spark session by adding `%%spark` at the beginning of the cell.  Since we’ve only created a single session, we don’t need to specify the session name.In the following cell, I'll create a Resilient Distributed Dataset (RDD) called fruits, and print its first element.<jupyter_code>%%spark
numbers = sc.parallelize([1, 2, 3, 4])
print('First element of numbers is {} and its description is:\n{}'.format(numbers.first(), numbers.toDebugString()))<jupyter_output>First element of numbers is 1 and its description is:
b'(1) ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:194 []'<jupyter_text>Now, you've created your session and executed some statements. If you want to look at the Livy logs for this session, simply run a cell like so:<jupyter_code>%spark logs<jupyter_output>16/06/09 23:43:28 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/06/09 23:43:28 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/06/09 23:43:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/09 23:43:30 INFO TimelineClientImpl: Timeline service address: http://localhost:8188/ws/v1/timeline/
16/06/09 23:43:30 INFO Client: Requesting a new application from cluster with 10 NodeManagers
16/06/09 23:43:30 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (25600 MB per container)
16/06/09 23:43:30 INFO Client: Will allocate [...]<jupyter_text>SparkSQL You can run SQL queries by passing the arguments `-c sql` to the %%spark magicFirst, let's create a table:<jupyter_code>%%spark
df = spark.read.json("/apps/spark-2.3.3/examples/src/main/resources/people.json")
df.createOrReplaceTempView("people")<jupyter_output><empty_output><jupyter_text>Now we can see what tables we have:<jupyter_code>%%spark -c sql
SHOW TABLES<jupyter_output><empty_output><jupyter_text>Now, let's query one of the available tables.Notice that we are passing the `--output` or `-o` parameter with a value of `df_hvac` so that the output of our SQL query is saved in the `df_hvac` variable in the IPython kernel context as a [Pandas](http://pandas.pydata.org/) DataFrame.<jupyter_code>%%spark -c sql -o df_people --maxrows 10
SELECT * FROM people<jupyter_output><empty_output><jupyter_text>>SQL queries also have other parameters you can pass in, like `--samplemethod`, `--maxrows`, `--samplefraction`, and `--quiet`. We can now simply use the Pandas dataframe from the IPython notebook.<jupyter_code>df_people.head()<jupyter_output><empty_output><jupyter_text>If you want to visualize the data in the Pandas dataframe, you can write your own code to do so, or you can use our autovisualization widget:<jupyter_code>from autovizwidget.widget.utils import display_dataframe
display_dataframe(df_people)<jupyter_output>/home/itamarst/Devel/sparkmagic/autovizwidget/autovizwidget/widget/utils.py:50: FutureWarning:

A future version of pandas will default to `skipna=True`. To silence this warning, pass `skipna=True|False` explicitly.<jupyter_text>>You could also choose to have this widget display by default for *all* Pandas dataframes from here on by running this piece of code:```ip = get_ipython()ip.display_formatter.ipython_display_formatter.for_type_by_name('pandas.core.frame', 'DataFrame', display_dataframe)```  Server-side renderingYou can also have images rendered on the server, and then display them locally. This prevents the need to ship large amounts of data locally to do visualizations. First, we render a PNG, in this case using matplotlib:<jupyter_code>%%spark
import matplotlib.pyplot as plt
ax = df.toPandas().plot.bar(x='name',y='age')<jupyter_output><empty_output><jupyter_text>And now we can view the resulting image using the `%matplot` magic:<jupyter_code>%%spark
%matplot plt<jupyter_output><empty_output><jupyter_text>Scala support If you want to write your Spark code in Scala, you can easily do that. Let's add a Scala session:<jupyter_code>%manage_spark<jupyter_output>Creating SparkContext as 'sc'
Creating HiveContext as 'sqlContext'<jupyter_text>![add_session](images/addsession_s.PNG) And just run some Spark code. Notice that we now specify the session we want to use, `-s my_spark`.<jupyter_code>%%spark -s my_spark
val hvacText = sc.parallelize(Array(1, 2, 3, 4))
hvacText.first()<jupyter_output>res0: Int = 1<jupyter_text>Now, we can query the table with **SparkSQL** too:<jupyter_code>%%spark -s my_spark -c sql -o my_df_from_scala --maxrows 10
SELECT * FROM hivesampletable<jupyter_output><empty_output><jupyter_text>And we can still access the result of the Spark query from Scala as a Pandas dataframe!<jupyter_code>my_df_from_scala.head()<jupyter_output><empty_output><jupyter_text>Cleaning upNow that you’re done with your Livy sessions, you should clean them up.Simply click on the `Delete` buttons!<jupyter_code>%manage_spark<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08600313256443115, 'fraction_numerical': 0.01779866154065214, 'mean_word_length': 6.024, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 66, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '5454', 'license': 'RSA-MD', 'n_tokens_mistral': 2286, 'n_tokens_neox': 2120, 'n_words': 823, 'path': 'examples/Magics in IPython Kernel.ipynb', 'repo_name': 'fbalicchia/sparkmagic', 'size': '7023'}