<jupyter_start><jupyter_text>APCOMP 295 Advanced Practical Data Science Exercise 3: Dask**Harvard University****Fall 2020****Instructors**: <NAME> **Each assignment is graded out of 5 points.  The topic for this assignment is Dask.****Due:** 09/29/2020 10:15 AM EDT**Submit:** We won't be re running your notebooks, please ensure output is visible in the notebook.  Question 1: Reflection on Exercise 2  (1 points)In exercise 2, we developed our app, tested it locally, afterward we deploy it with docker and finally with minikube and/or gcloud.  Please answer following questions in 1-3 sentences. (A) For the local code files:1.  Which port is `maindb.py` listening at?   80822. Which port is `task1.py` listening at?  8081(B) For the Docker implementation we built two images: `webapp:db` and `task1:frontend` 1. For `webapp:db` which port is exposed by the container? 8082 Are you binding any port from host (i.e your laptop) to container port ? no    2. For `task1:frontend` which port is exposed by the container? 8081 Are you binding any port from host (i.e your laptop) to container port ? 30000  3.  There's an environment variable in `Docker_task1frontend` named `DB_HOST`. How is that getting utilized? Reminder:  We used these 2 commands to run our images (or used docker-compose) -  `docker run --name mywebdb -d --network appNetwork webapp:db` `docker run --name fe -d -p 5000:8081 -e DB_HOST=mywebdb --network appNetwork task1:frontend`  Hint: Check `task1.py` the environment var `DB_HOST` is passed in as a command line argument to `task1.py`. `task1.py` uses it as part of `db_url` for sending requests to it, and it is defaulted to `0.0.0.0`.(C) For the minikube/gcloud implementation, 1. How is the environment variable `DB_HOST`  getting populated ? it is read from a `database_host` setting from k8s config map    2. How are we able to access our front end via browser ? by loading `http://localhost:30000`    3. What is the purpose of `webapp-db-service` in `webapp_db_deployment_k8s.yaml` ?it is specifying the logically name of the db service to be matched with config map. it is then passed into web frontend.  Question 2: Compute Pi with a Slowly Converging Series (1 points)Leibniz published one of the oldest known series in 1676.  While this is easy to understand and derive, it converges very slowly.https://en.wikipedia.org/wiki/Leibniz_formula_for_%CF%80 $$\frac{\pi}{4} = 1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} ...$$While this is a genuinely cruel way to compute the value of $\pi$, it’s a fun opportunity to use brute force on a problem instead of thinking.Compute $\pi$ using at least four billion terms in this sequence. Compare your time taken with numpy and dask.  On my mac, with numpy this took 44 seconds and with dask it took 5.7 seconds. *Hint:* Use dask array<jupyter_code>import numpy as np
num_terms = 100000000
def pi_arr(arr):
    return (-2*(arr%2 ==0)+1)/(2*arr-1)*4
%%time

import dask.array as da
pi_arr(da.arange(1, num_terms, chunks=num_terms/4)).sum().compute()
%%time

pi_arr(np.arange(1, num_terms)).sum()<jupyter_output>CPU times: user 2.58 s, sys: 1.34 s, total: 3.91 s
Wall time: 4.11 s<jupyter_text>Question 3: Filter Parking Tickets Dataset (1 points)Please download the data set from https://www.kaggle.com/new-york-city/nyc-parking-tickets. According to the documentation for the parking tickets data set, the column called ‘Plate Type’ consists mainly of two different types: ‘PAS’ and ‘COM’, presumably for passenger and commercial vehicles, respectively. Maybe the rest are the famous parking tickets from the UN diplomats, who take advantage of diplomatic immunity not to pay their fines.Create a filtered Dask DataFrame with only the commercial plates.Persist it, so it is available in memory for future computations. Count the number of summonses in 2017 issued to commercial plate types. Compute them as a percentage of the total data set. *Hint*: This is easy; it is only about 5-7 lines of code.<jupyter_code>import dask
import dask.dataframe as dd
df_2017 = dd.read_csv('data/Parking_Violations_Issued_-_Fiscal_Year_2017.csv')
df_2017_COM = df_2017[df_2017['Plate Type'] == 'COM']
df_2017_COM.persist()

print(f'commerical vehicle parking ticket percentage {len(df_2017_COM)*100/len(df_2017)}%')<jupyter_output>commerical vehicle parking ticket percentage 17.02794809010955%
{'fraction_non_alphanumeric': 0.08888378502526413, 'fraction_numerical': 0.04455672944418925, 'mean_word_length': 4.730263157894737, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '146509', 'license': 'MIT', 'n_tokens_mistral': 1434, 'n_tokens_neox': 1281, 'n_words': 596, 'path': 'exercises/exercise3/exercise3.ipynb', 'repo_name': 'jackxxu/2020F-AC295', 'size': '4365'}