<commit_before>---
layout: documentation
displayTitle: Notebook
title: Notebook
description: Notebook
includeOperationsMenu: true
---

A `Notebook` is a special type of operation. It gives a user a capability to quickly explore the data.
In the `Notebook` the user can write the code using Apache Spark Python API.
Blocks of Python code can be executed and the results are immediately returned to the `Notebook`.

To use the `Notebook` the user has to drag and drop the `Notebook` operation onto the canvas
and connect a [DataFrame](../classes/dataframe.html) to its input port. The user can access the connected `DataFrame` from within
the `Notebook` by calling the `dataframe()` function.

The user can start editing the code by clicking the `Open notebook` button in the `Notebook` operation's
parameters panel.

The variables and functions available in the `Notebooks`' global scope:

* <code>dataframe()</code> - a function that returns the input DataFrame for this `Notebook`.
Everytime the input `Dataframe` changes, the `dataframe()` returns the updated `DataFrame`.

* <code>sc</code> - Spark Context

* <code>sqlContext</code> - SQL Context

To visualize the data the user can use <a target="_blank" href="{{ site.PANDAS_LIBRARY_ADDRESS }}">pandas</a> library.

**Remark**: In case of big `DataFrames`, the user should sample the DataFrame before using pandas.

**Since**: Seahorse 1.0.0

####Examples

{% highlight python %}
from pyspark.sql.types import *
from pyspark.sql import SQLContext, Row

schemaString = "datetime,weather2,weather3,windspeed,hum,temp,atemp"
fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split(",")]
schema = StructType(fields)

lines = sc.textFile("weather.csv")
split = lines.map(lambda s: s.split(","))
df = sqlContext.createDataFrame(split, schema)
df.show()
{% endhighlight %}

    +--------------------+--------+--------+---------+----+----+------+
    |            datetime|weather2|weather3|windspeed| hum|temp| atemp|
    +--------------------+--------+--------+---------+----+----+------+
    |2011-01-01T00:00:...|       0|       0|        0|0.81|0.24|0.2879|
    |2011-01-01T01:00:...|       0|       0|        0| 0.8|0.22|0.2727|
    |2011-01-01T02:00:...|       0|       0|        0| 0.8|0.22|0.2727|
    |2011-01-01T03:00:...|       0|       0|        0|0.75|0.24|0.2879|
    |2011-01-01T04:00:...|       0|       0|        0|0.75|0.24|0.2879|
    |2011-01-01T05:00:...|       1|       0|   0.0896|0.75|0.24|0.2576|
    |2011-01-01T06:00:...|       0|       0|        0| 0.8|0.22|0.2727|
    |2011-01-01T07:00:...|       0|       0|        0|0.86| 0.2|0.2576|
    |2011-01-01T08:00:...|       0|       0|        0|0.75|0.24|0.2879|
    |2011-01-01T09:00:...|       0|       0|        0|0.76|0.32|0.3485|
    |2011-01-01T10:00:...|       0|       0|   0.2537|0.76|0.38|0.3939|
    |2011-01-01T11:00:...|       0|       0|   0.2836|0.81|0.36|0.3333|
    |2011-01-01T12:00:...|       0|       0|   0.2836|0.77|0.42|0.4242|
    |2011-01-01T13:00:...|       1|       0|   0.2985|0.72|0.46|0.4545|
    |2011-01-01T14:00:...|       1|       0|   0.2836|0.72|0.46|0.4545|
    |2011-01-01T15:00:...|       1|       0|   0.2985|0.77|0.44|0.4394|
    |2011-01-01T16:00:...|       1|       0|   0.2985|0.82|0.42|0.4242|
    |2011-01-01T17:00:...|       1|       0|   0.2836|0.82|0.44|0.4394|
    |2011-01-01T18:00:...|       0|       1|   0.2537|0.88|0.42|0.4242|
    |2011-01-01T19:00:...|       0|       1|   0.2537|0.88|0.42|0.4242|
    +--------------------+--------+--------+---------+----+----+------+

{% highlight python %}
import matplotlib.pyplot as plt
%matplotlib inline
df = dataframe().toPandas()
df['atemp'].plot()
{% endhighlight %}
<img class="img-responsive" src="../img/notebook_plot.png" />

## Input

<table>
<thead>
<tr>
<th style="width:15%">Port</th>
<th style="width:15%">Type Qualifier</th>
<th style="width:70%">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>0</code></td>
<td><code><a href="../classes/dataframe.html">DataFrame</a></code></td>
<td>The input <code>DataFrame</code> that can be used in the operation.</td>
</tr>
</tbody>
</table>

## Output

The `Notebook` operation does not produce any output.
<commit_msg>Fix notebook documentation<commit_after>---
layout: documentation
displayTitle: Notebook
title: Notebook
description: Notebook
includeOperationsMenu: true
---

A `Notebook` is a special type of operation.
It gives Spark shell capabilities in a web browser, so the user can quickly explore the data.
In the `Notebook`, the user can write the code using Apache Spark Python API and accessing objects available in Spark context.
Blocks of Python code can be executed and the results are immediately returned to the `Notebook`.

To use the `Notebook` the user has to drag and drop the `Notebook` operation onto the canvas
and connect a [DataFrame](../classes/dataframe.html) to its input port. The user can access the connected `DataFrame` from within
the `Notebook` by calling the `dataframe()` function.

The user can start editing the code by clicking the `Open notebook` button in the `Notebook` operation's
parameters panel.

The variables and functions available in the `Notebooks`' global scope:

* <code>dataframe()</code> - a function that returns the input `DataFrame` for this `Notebook`.
Everytime the input `Dataframe` changes, the `dataframe()` returns the updated `DataFrame`.

* <code>sc</code> - Spark Context

* <code>sqlContext</code> - SQL Context

To visualize the data the user can use <a target="_blank" href="{{ site.PANDAS_LIBRARY_ADDRESS }}">pandas</a> library.

**Remark**: In case of big `DataFrames`, the user should sample the DataFrame before using pandas.

**Since**: Seahorse 1.0.0

####Examples

{% highlight python %}
from pyspark.sql.types import *
from pyspark.sql import SQLContext, Row

schemaString = "datetime,weather2,weather3,windspeed,hum,temp,atemp"
fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split(",")]
schema = StructType(fields)

lines = sc.textFile("weather.csv")
split = lines.map(lambda s: s.split(","))
df = sqlContext.createDataFrame(split, schema)
df.show()
{% endhighlight %}

    +--------------------+--------+--------+---------+----+----+------+
    |            datetime|weather2|weather3|windspeed| hum|temp| atemp|
    +--------------------+--------+--------+---------+----+----+------+
    |2011-01-01T00:00:...|       0|       0|        0|0.81|0.24|0.2879|
    |2011-01-01T01:00:...|       0|       0|        0| 0.8|0.22|0.2727|
    |2011-01-01T02:00:...|       0|       0|        0| 0.8|0.22|0.2727|
    |2011-01-01T03:00:...|       0|       0|        0|0.75|0.24|0.2879|
    |2011-01-01T04:00:...|       0|       0|        0|0.75|0.24|0.2879|
    |2011-01-01T05:00:...|       1|       0|   0.0896|0.75|0.24|0.2576|
    |2011-01-01T06:00:...|       0|       0|        0| 0.8|0.22|0.2727|
    |2011-01-01T07:00:...|       0|       0|        0|0.86| 0.2|0.2576|
    |2011-01-01T08:00:...|       0|       0|        0|0.75|0.24|0.2879|
    |2011-01-01T09:00:...|       0|       0|        0|0.76|0.32|0.3485|
    |2011-01-01T10:00:...|       0|       0|   0.2537|0.76|0.38|0.3939|
    |2011-01-01T11:00:...|       0|       0|   0.2836|0.81|0.36|0.3333|
    |2011-01-01T12:00:...|       0|       0|   0.2836|0.77|0.42|0.4242|
    |2011-01-01T13:00:...|       1|       0|   0.2985|0.72|0.46|0.4545|
    |2011-01-01T14:00:...|       1|       0|   0.2836|0.72|0.46|0.4545|
    |2011-01-01T15:00:...|       1|       0|   0.2985|0.77|0.44|0.4394|
    |2011-01-01T16:00:...|       1|       0|   0.2985|0.82|0.42|0.4242|
    |2011-01-01T17:00:...|       1|       0|   0.2836|0.82|0.44|0.4394|
    |2011-01-01T18:00:...|       0|       1|   0.2537|0.88|0.42|0.4242|
    |2011-01-01T19:00:...|       0|       1|   0.2537|0.88|0.42|0.4242|
    +--------------------+--------+--------+---------+----+----+------+

{% highlight python %}
import matplotlib.pyplot as plt
%matplotlib inline
df = dataframe().toPandas()
df['atemp'].plot()
{% endhighlight %}
<img class="img-responsive" src="../img/notebook_plot.png" />

## Input

<table>
<thead>
<tr>
<th style="width:15%">Port</th>
<th style="width:15%">Type Qualifier</th>
<th style="width:70%">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>0</code></td>
<td><code><a href="../classes/dataframe.html">DataFrame</a></code></td>
<td>The input <code>DataFrame</code> that can be used in the operation.</td>
</tr>
</tbody>
</table>

## Output

The `Notebook` operation does not produce any output.

{'fraction_non_alphanumeric': 0.21504373177842565, 'fraction_numerical': 0.13387755102040816, 'mean_word_length': 2.355242566510172, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1752519', 'n_tokens_mistral': 3775, 'n_tokens_neox': 3215, 'n_words': 747}