<jupyter_start><jupyter_text>CS109B Data Science 2: Advanced Topics in Data Science  Lab 01 - Coding Environment Setup **Harvard University****Spring 2022****Instructors:** <NAME> and <NAME>**Lab Instructor:** <NAME>---<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Learning GoalsThe purpose of this lab is to help you set up the coding environment for CS109B  1. Getting class material Option 1A: Download directly from Ed * Use the >> to download. Option 1B: Cloning the class repo and then copying the contents in a different directory so you can make changes.You may access the code used in class by cloning the class repo: [https://github.com/Harvard-IACS/2022-CS109B](https://github.com/Harvard-IACS/2022-CS109B)* Open the Terminal in your computer and go to the Directory where you want to clone the repo. Then run `git clone https://github.com/Harvard-IACS/2022-CS109B.git`* If you have already cloned the repo, OR if new material is added (happens every day), go inside the '/2022-CS109B/' directory and run `git pull`* **Caution:** If you change the notebooks and then run `git pull` your changes will be overwritten. So create a `playground` folder and copy the folder with the notebook with which you want to work. Option 1C: Fork a respositorySee the following file for details:D:\Dropbox\CS109B\Spring2022\2022-CS109B\content\labs\lab01\ForkRepo_Joel.pdfNote that Joel made some changes in the ForkRepo_Joel.pdf, based on the ForkRepo.pdf  2. Running code: Option 2A: Using your local environment     Use Virtual Environments: we cannot stress this enough!Isolating your projects inside specific environments helps you manage dependencies and therefore keep your sanity. You can recover from mess-ups by simply deleting an environment. Sometimes certain installation of libraries conflict with one another. The two most popular tools for setting up environments are:- `conda` (a package and environment manager)- `pip` (a Python package manager) with `virtualenv` (a tool for creating environments)We recommend using `conda` package installation and environments. `conda` installs packages from the Anaconda Repository and Anaconda Cloud, whereas `pip` installs packages from PyPI. Even if you are using `conda` as your primary package installer and are inside a `conda` environment, you can still use `pip install` for those rare packages that are not included in the `conda` ecosystem. See here for more details on how to manage [Conda Environments](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html).   Use the cs109b.yml file to create an environment:``` $ conda env create -f cs109b.yml (cs109b.yml was posted on Ed and is stored here: D:\Dropbox\CS109B\Spring2022\cs109b.yml)$ conda activate cs109b```   We have included the packages that you will need in the `cs109b.yml` file.   Option 2B: Using Cloud Resources Using FAS OnDemand (supported by CS109b)FAS provides a platform, accessible via the `FAS OnDemand` menu link in **Canvas**. Most of the libraries such as keras, tensorflow, pandas, etc., are pre-installed. If a library is missing you may install it via the Terminal.**NOTE**: The AWS platform is funded by FAS for the purposes of the class.  You are not allowed to use it for purposes not related to this course. Make sure you stop your instance as soon as you do not need it.Information on how to use the platform is displayed when you click the link. For more see [Fas OnDemand Guide](https://canvas.harvard.edu/courses/84598/pages/fas-ondemand-guide).  Using Google Colab (on your own)Google's Colab platform [https://colab.research.google.com/](https://colab.research.google.com/) offers a GPU enviromnent to test your ideas, it's fast, free, with the only caveat that your files persist only for 12 hours (last time we checked). The solution is to keep your files in a repository and just clone it each time you use Colab.    Using AWS in the Cloud (on your own)For those of you who want to have your own machines in the Cloud to run whatever you want, Amazon Web Services is a (paid) solution. For more see: [https://docs.aws.amazon.com/polly/latest/dg/setting-up.html](https://docs.aws.amazon.com/polly/latest/dg/setting-up.html)Remember, AWS is a paid service, so if you let your machine run for days you will get charged!  3. Ensuring everything is installed correctly Some of the packages we will need for this class    - **Clustering**: - Sklearn - [https://scikit-learn.org/stable/](https://scikit-learn.org/stable/) - scipy - [https://www.scipy.org](https://www.scipy.org) - gap_statistic (by <NAME>) - [https://anaconda.org/milesgranger/gap-statistic/notebook](https://anaconda.org/milesgranger/gap-statistic/notebook)- **Bayes**: - pymc3 - [https://docs.pymc.io](https://docs.pymc.io) - **Neural Networks**: - keras - [https://www.tensorflow.org/guide/keras](https://www.tensorflow.org/guide/keras)   Exercise 1:  Run the following cells to make sure these packages load correctly in our environment.<jupyter_code>from sklearn import datasets
iris = datasets.load_iris()
digits = datasets.load_digits()
digits.target # you should see [0, 1, 2, ..., 8, 9, 8]
from scipy import misc
import matplotlib.pyplot as plt

face = misc.face()
face.shape, type(face)
face[1:3, 1:3]
plt.imshow(face)
plt.show() # you should see a racoon
import pymc3 as pm
print('Running PyMC3 v{}'.format(pm.__version__)) # you should see 'Running on PyMC3 v3.8'
# making sure you have gap_statistic
from gap_statistic import OptimalK<jupyter_output><empty_output><jupyter_text>4. Plotting   `matplotlib` and `seaborn`- `matplotlib` - [seaborn: statistical data visualization](https://seaborn.pydata.org/). `seaborn` works great with `pandas`.  It can also be customized easily.  Here is the basic `seaborn` tutorial: [Seaborn tutorial](https://seaborn.pydata.org/tutorial.html).  Plotting a function of 2 variables using contoursIn optimization, our objective function will often be a function of two or more variables. While it's hard to visualize a function of more than 3 variables, it's very informative to plot one of 2 variables. To do this we use contours. First we define the $x$ and $y$ variables and then construct their pairs using `meshgrid`.  Plot the function $f(x,y) = x^2+y^2$<jupyter_code>import seaborn as sn
x = np.linspace(-0.1, 0.1, 50)
y = np.linspace(-0.1, 0.1, 100)
xx, yy = np.meshgrid(x, y)
z = np.sqrt(xx**2+yy**2)
plt.contour(x,y,z);<jupyter_output><empty_output><jupyter_text>5. We will be using `keras` via `tensorflow` **[TensorFlow](https://www.tensorflow.org)** is a framework for representing complicated ML algorithms and executing them in any platform, from a phone to a distributed system using GPUs. Developed by Google Brain, TensorFlow is used very broadly today. **[Keras](https://keras.io/)**, is a high-level API, created by <NAME>, and used for fast prototyping, advanced research, and production. `tf.keras` is now maintained by Tensorflow. Exercise 2: Run the following cells to make sure you have the basic libraries to do deep learning<jupyter_code>from __future__ import absolute_import, division, print_function, unicode_literals

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras import models
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.regularizers import l2

tf.keras.backend.clear_session()  # For easy reset of notebook state.

# You should see a >=2.3.0 here! 
# If you do not, upgrade your env to tensorflow==2.3.0
print(tf.__version__)  
print(tf.keras.__version__)
# Check if your machine has NVIDIA GPUs. 
hasGPU = tf.config.list_physical_devices()
print(f'My computer has the following devices: {hasGPU}')<jupyter_output>My computer has the following devices: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
{'fraction_non_alphanumeric': 0.09389379430419102, 'fraction_numerical': 0.018530033577913194, 'mean_word_length': 4.882955376737381, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 24, 'https://': 23, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '39487', 'license': 'MIT', 'n_tokens_mistral': 2483, 'n_tokens_neox': 2333, 'n_words': 994, 'path': 'content/labs/lab01/cs109b_lab01_intro_Joel.ipynb', 'repo_name': 'zhonglizhang/2022-CS109B', 'size': '8083'}