<jupyter_start><jupyter_text>Building a pipeline  OutlineWe are going to create a pipeline with two steps:For each step, the process of creating it will be the following:1. Create notebook2. Commit changes to git3. Run notebook reproduciblyBy chaining the outputs of one notebook into the inputs of the next, we will create a pipeline. To run a notebook reproducibly, we will use the command `renku run` together with papermill.`renku run [shell command]` takes any shell command as an argument. Renku * takes note of the inputs  * detects outputs of the command* remembers the command as a workflow step.<jupyter_code>%cd renku-tutorial-flights<jupyter_output><empty_output><jupyter_text>Create a notebooks directoryWe want to follow a standard project organization structure. We will put our notebooks in the folder notebooks.<jupyter_code>%mkdir notebooks<jupyter_output><empty_output><jupyter_text>Filter Flights  Create a notebookThere are notebooks prepared in the templates folder. We are going to use a these to build our pipeline.<jupyter_code>!cp ../templates/00-FilterFlights.ipynb ./notebooks/<jupyter_output><empty_output><jupyter_text>Step through the [FilterFlights notebook](./renku-tutorial-flights/notebooks/00-FilterFlights.ipynb).  Commit changes to gitWe've done something new that we need to track in version control.<jupyter_code>!git status
!git add notebooks/00-FilterFlights.ipynb
!git commit -m"Created notebook to filter flights to AUS, TX."
!git status<jupyter_output><empty_output><jupyter_text>**Tip**: `git status` should now report that the working tree is clean; if it's not, you can run `!git clean -f` to make it clean.  Run reproduciblyStepping through the notebook as we did is not guaranteed to be reproducible. For this, we will use papermill in combination with renku to run the notebook.Papermill passes parameters provided with `-p` on to the notebook. And Papermill takes as an argument the notebook to run (`notebooks/00-FilterFlights.ipynb` in this case) and a filename for the resulting run notebook (`notebooks/00-FilterFlights.ran.ipynb`, following the standard convention of [Notebook Name].ran.ipynb)<jupyter_code>%%bash
mkdir -p data/output
renku run papermill \
  -p input_path data/flights/2019-01-flights.csv.zip \
  -p output_path data/output/2019-01-flights-filtered.csv \
  notebooks/00-FilterFlights.ipynb \
  notebooks/00-FilterFlights.ran.ipynb<jupyter_output><empty_output><jupyter_text>Before we continue, let us take a look at the [FilterFlights.ran notebook](./renku-tutorial-flights/notebooks/00-FilterFlights.ran.ipynb).  Count Flights  Create a notebookAgain, you can use the notebook prepared in the templates folder.<jupyter_code>!cp ../templates/01-CountFlights.ipynb ./notebooks/<jupyter_output><empty_output><jupyter_text>Step through the [CountFlights notebook](./renku-tutorial-flights/notebooks/01-CountFlights.ipynb).  Commit changes to gitWe've done something new that we need to track in version control.<jupyter_code>!git status
!git add notebooks/01-CountFlights.ipynb
!git commit -m"Created notebook to count flights to AUS, TX."
!git status<jupyter_output><empty_output><jupyter_text>Again, `git status` should report a clean working tree. Remove any lingering files if this is not the case.  Run reproduciblyLet's run the CountFlights notebook reproducibly as well.<jupyter_code>%%bash
mkdir -p data/output
renku run papermill \
  -p input_path data/output/2019-01-flights-filtered.csv  \
  -p output_path data/output/2019-01-flights-count.txt \
  notebooks/01-CountFlights.ipynb \
  notebooks/01-CountFlights.ran.ipynb<jupyter_output><empty_output><jupyter_text>Now we have a pipeline in Renku!  Viewing the pipelineIf you have dot available, you can visualize the pipeline.<jupyter_code>graph = !renku log --format dot | dot -Tsvg -Gsize=12,10
SVG("\n".join(graph))<jupyter_output><empty_output><jupyter_text>Viewing the resultWe can also look at the file that was the final [2019-01-flights-count.txt](renku-tutorial-flights/data/output/2019-01-flights-count.txt).  ExercisesSolutions are provided in the commented `%load` statements.Enter the renku-tutorial-diamonds repo to do the exercises.<jupyter_code>%cd ../renku-tutorial-diamonds<jupyter_output><empty_output><jupyter_text>Ex 2.0Create a pipeline for the `renku-tutorial-diamonds` project using the `00-FilterDiamonds.ipynb` and `01-CountDiamonds.ipynb` notebooks from the `templates` directory. To get started, enter the renku-tutorial-diamonds and make folders for the notebooks and data outputs, and "write" the 00-FilterDiamonds notebook (by copying it).If something fails, restore the working directory to a clean state (e.g., using `git clean -f`), fix the problem and try again.<jupyter_code>!mkdir -p notebooks
!mkdir -p data/output
!cp ../templates/00-FilterDiamonds.ipynb ./notebooks/<jupyter_output><empty_output><jupyter_text>Ex 2.0.0First add the notebook to git and provide a nice commit message.<jupyter_code># %load ../solutions/ex2-0-0.fragment<jupyter_output><empty_output><jupyter_text>Ex 2.0.1Now run the notebook using papermill using `data/diamonds/diamonds.csv` as input and writing to `data/output/diamonds-filtered.csv` as output.<jupyter_code># %load ../solutions/ex2-0-1.fragment<jupyter_output><empty_output><jupyter_text>Now "write" the 01-CountDiamonds notebook (by copying it).<jupyter_code>!cp ../templates/01-CountDiamonds.ipynb ./notebooks/<jupyter_output><empty_output><jupyter_text>Ex 2.0.2Then add the notebook to git and provide a nice commit message.<jupyter_code># %load ../solutions/ex2-0-2.fragment<jupyter_output><empty_output><jupyter_text>Ex 2.0.3Now run the notebook using papermill<jupyter_code># %load ../solutions/ex2-0-3.fragment<jupyter_output><empty_output><jupyter_text>As earlier, if you have dot available, you can view the graph visually<jupyter_code>graph = !renku log --format dot | dot -Tsvg -Gsize=12,10
SVG("\n".join(graph))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09429530201342282, 'fraction_numerical': 0.018624161073825505, 'mean_word_length': 5.638084632516704, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '374819', 'license': 'Apache-2.0', 'n_tokens_mistral': 2013, 'n_tokens_neox': 1891, 'n_words': 590, 'path': 'notebooks/03-renku/02-1-BuildPipeline.ipynb', 'repo_name': 'SwissDataScienceCenter/r10e-ds-py', 'size': '5960'}