<jupyter_start><jupyter_text>IntroA breif intro to commonly used data science tools in Python  Table of Content- [Python's Data Science Stack](python-datascience)    - [Jupyter](jupyter)    - [NumPy](numpy)    - [pandas](pandas)    - [matplotlib](matplotlib)    - [Bokeh](bokeh)    - [scikit-learn](scikit-learn)- [Read More](read-more)  Python's Data Science Stack [Python's Data Science Stack (JSM 2016) by <NAME>](https://speakerdeck.com/jakevdp/pythons-data-science-stack-jsm-2016)![datascience-stack](img/datascience-stack.png) ---  Jupyter* [jupyter notebook](https://jupyter.org/)- [jupyter lab](https://github.com/jupyterlab/jupyterlab)    - [JupyterLab is Ready for Users](https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906) (2018/2/20)- [nbviewer](https://nbviewer.jupyter.org)  Why Jupyter notebook is not all that good?* Jupyter notebook is great.    * But sometimes it can be troublesome......    * [I Don't Like Notebooks - <NAME> - JupyterCon 2018](https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/editslide=id.g362da58057_0_1)  Features (Jupyter Notebook)  1. In-browser code editing and executing<jupyter_code>print('Hello World')<jupyter_output>Hello World<jupyter_text>2. Magics<jupyter_code>%lsmagic
%%time
print('Hello world')<jupyter_output>Hello world
CPU times: user 401 µs, sys: 366 µs, total: 767 µs
Wall time: 423 µs<jupyter_text>3. Supporting plenty of  languages[Jupyter kernels](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels)<jupyter_code>%%js
element.text('Hello')
%%ruby
puts "Hello World!"<jupyter_output>Hello World!<jupyter_text>4. Supporting rich media representations`This is markdown`Latex: $P(A|B)=\frac{P(B|A)P(A)}{P(b)}$  Extensions- [jupyter_nbextensions_configurator](https://github.com/Jupyter-contrib/jupyter_nbextensions_configurator)- [RISE](https://github.com/damianavila/RISE)    - Quickly turn your Jupyter Notebooks into a live presentation. ---  NumPy<jupyter_code>import numpy as np

np.random.seed(42)
# initial an array
a = np.arange(8)
a
# reshpae an array
a.reshape(2, 4)
# reshape method is not inplace change
a
# inplace reshape
a.shape = 4, 2
a
# indexing
a[:,1]
b = np.random.randint(1, 100, (4, 2))
b
# element-wise add
a + b
# element-wies multiply
a * b
# dot
a.dot(b.T)
print('mean of a', np.mean(a))
print('var of a', np.var(a))<jupyter_output>mean of a 3.5
var of a 5.25<jupyter_text>---  pandas<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Series<jupyter_code>ser = pd.Series(np.random.randint(1, 100, 5))
ser
ser.describe()
another_ser = pd.Series(np.random.randint(1, 100, 5))
# correlation
ser.corr(another_ser)
# covariance
ser.corr(another_ser)
time_ser = pd.date_range('20180227', periods=30, freq='D')
time_ser<jupyter_output><empty_output><jupyter_text>DataFrame<jupyter_code>df = pd.DataFrame(np.random.randint(1, 100, (10, 5)))
df
feature_name = ['a', 'b', 'c', 'd', 'e']
df.columns = feature_name
df
df['a']
df[['a', 'e']]
df[df['a'] > 50]<jupyter_output><empty_output><jupyter_text>---  matplotlib<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
x1 = np.linspace(start=10, stop=100, num=10)
x2 = np.linspace(start=100, stop=10, num=10)
y = np.linspace(start=5, stop=10, num=10)<jupyter_output><empty_output><jupyter_text>Line Chart<jupyter_code>plt.plot(x1, y, marker='o', color='r', label='x1')
plt.plot(x2, y, marker='*', color='b', label='x2')
plt.title('Line chart example')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Scatter Plot<jupyter_code>plt.scatter(x1, y, label='x1')
plt.scatter(x2, y, label='x2')
plt.title('Scatter Plot Example')
plt.xlabel('x')
plt.ylabel('y')
plt.legend(loc=6)
plt.show()<jupyter_output><empty_output><jupyter_text>Bar Chart<jupyter_code>x = [1, 2]
y = [100, 101]
plt.bar(x, y)
plt.xticks(x)

# Wrong axis might causes bias
plt.axis([0.5, 2.5, 99.5, 101.5])
plt.show()
plt.bar(x, y)
plt.xticks(x)
plt.axis([0.5, 2.5, 0, 120])
plt.show()<jupyter_output><empty_output><jupyter_text>---  Bokeh<jupyter_code>from bokeh.io import output_notebook
from bokeh.plotting import figure, show
output_notebook()
x = np.random.random(10)
y = np.random.random(10)
fig = figure(title="bokeh example", x_axis_label='x', y_axis_label='y')
fig.line(x, y, legend="random")
t = show(fig, notebook_handle=True)<jupyter_output><empty_output><jupyter_text>scikit-learn<jupyter_code>from sklearn import linear_model
X_train = np.random.random((1000, 2))
y_train = np.random.random(1000)
reg = linear_model.LinearRegression()
reg.fit(X_train, y_train)
reg.predict(
    [
        [1, 2], [3, 4], [5, 6]
    ]
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16584949904071628, 'fraction_numerical': 0.03943722020891068, 'mean_word_length': 6.055639097744361, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '562090', 'license': 'MIT', 'n_tokens_mistral': 2108, 'n_tokens_neox': 1937, 'n_words': 359, 'path': '1 - Intro.ipynb', 'repo_name': 'Lee-W/sklearn_intro', 'size': '4703'}