<jupyter_start><jupyter_text>통계적 사고와 분석 기법  (배경훈) 강의 전 Prep 자료 subsection 1.1 subsection 1.2 Section 2 subsection 2.1 subsection 2.2 Section 3실제강의에서 다룰회귀 분석및시계열분석의이해를돕기위해 간단한 수학 및 확률에 대한 기초 prep 자료를 준비하였습니다.관련 내용을 숙지한 후, 첨부된 엑셀 파일의 체크리스트에서 본인이 강의 전 prep 자료 내용을 얼마나 이해 했는지 간단한 자가진단을 통해 확인하여 주시길 부탁드립니다. (제출하실 필요는 없습니다.) 감사합니다! * Bullet1* Bullet2 [링크 텍스트](https://)  $$ y = \beta_{0} + \beta_{1} x_1 + \beta_{2} x_2 $$ $$\cos (2\theta) = \cos^2 \theta - \sin^2 \theta $$<jupyter_code>print("Hello")<jupyter_output>Hello<jupyter_text>파이썬에서 패키지를 불러오는 방법<jupyter_code>import package_name as abb<jupyter_output><empty_output><jupyter_text>* Numpy는 행렬을 계산을 쉽게 해주는 패키지<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text><jupyter_code>x = np.random.normal(loc=10,scale=1,size=100)
X[2]<jupyter_output><empty_output><jupyter_text>loop 문<jupyter_code>for i in range(0,10):
  i**2
tmp = [i**2 for i in range(0,10)]
tmp = [(i+1)*2 for i in range(0,100)]
type(tmp)
tmp
total = 0
for i in X:
  total = total + i
total<jupyter_output><empty_output><jupyter_text>Object.Action()Object가 action을 수행한다.x.sum()x가 sum()을 수행한다.<jupyter_code>x.sum()
np.sum(X)

import numpy as np
np.random.normal(size=100)
X = np.random.normal(size=100)<jupyter_output><empty_output><jupyter_text>$$ \Sigma_{i=1}^{N} X_{i} $$$$ \Sigma_{i=1}^{N} \frac{X_{i}}{Y_{i}}  $$LaTex라는 언어를 조금 배우시면 수식을 쉽게 쓸 수 있습니다.<jupyter_code>X.sum()
total = 0
for i in X:
  total = total + i
print(total)
X.sum()
X
X.sum()
np.sum(X)
X.sum()
Object.Action()
X.std()
np.sum(X)
x = np.random.normal(size=100)
y = np.random.normal(size=100)
(2*x + 3*y).sum()
2* x.sum() + 3*y.sum()
len(x)
x = np.random.normal(size=99)
len(x)
x.sum()/len(x)
x.mean()
x.mean()
x = np.random.normal(loc=0,scale=1,size=100)
x.mean()<jupyter_output><empty_output><jupyter_text>그래프 그리는 방법https://python-graph-gallery.com/갤러리에서 찾아서 코드를 재활용한다.<jupyter_code>from matplotlib import pyplot as plt
import seaborn as sns
df = sns.load_dataset('iris')

# Large bandwidth
sns.kdeplot(df['sepal_width'], shade=True, bw=.5, color="olive")
#sns.plt.show()

# Narrower bandwidth
sns.kdeplot(df['sepal_width'], shade=True, bw=.05, color="olive")
#sns.plt.show()<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/distributions.py:1659: FutureWarning: The `bw` parameter is deprecated in favor of `bw_method` and `bw_adjust`. Using 0.05 for `bw_method`, but please see the docs for the new parameters and update your code.
  warnings.warn(msg, FutureWarning)<jupyter_text>numpy에 있는 linspace<jupyter_code>xsp = np.linspace(0.01,10,num=100)
xsp
ysp = np.log(xsp)
plt.plot(xsp,ysp)
plt.hist(x)
x = np.random.normal(size=10000)
plt.hist(x,bins=50)
# Normal random variable
x = np.random.normal(size=10000)
plt.hist(x,bins=100)
## toss coin random variable
x2 = (np.random.normal(size=10000) >=0) * 1
plt.hist(x2)
x = np.random.normal(size=10000)
x[0:20]
## -1<= x <= 1
(((x >= -1) & (x <= 1))*1).sum()
len(x)
(((x >= -1) & (x <= 1))*1).sum() / len(x)
(((x <= 2))*1).sum() / len(x)
x<jupyter_output><empty_output><jupyter_text>함수를 정의하는 방법<jupyter_code>def function_name(input1,input2):
  #compute something
  print(x)
  output = input1 * input2 + input1
  return output
def cdf(a):
  out = (((x <= a))*1).sum() / len(x)
  return out
cdf(2)
asp = np.linspace(-100,100,num=100000)
asp
ysp = [cdf(i) for i in asp]
asp.shape
ysp2 = np.array(ysp)
ysp2.shape
plt.plot(asp[49000:51000],ysp2[49000:51000])
((x<=1)*1).sum()/len(x)
cdf(2)
x
1/len(x)
x.mean()
x = np.random.normal(size=10)
x
x.mean()
np.median(x)
x.sort()
x
x[4]
x[5]
(x[4]+x[5])*0.5
np.median(x)
x.mean() ## 분포에서 왜곡될 확률이 높다
x.mean()
x.median()
np.median(x)
(x.std())**2
x.std()
## variance
var_x = ((x-x.mean())**2).sum()/len(x)
var_x
std_x
x.std() ** 2
std_x = var_x ** 0.5
std_x
x.std()
x.mean()
x.median()<jupyter_output><empty_output><jupyter_text>dir 함수를 쓰면 모든 action이 보인다.<jupyter_code>dir(x)
x.var()
x.median()
x.mean()
x.min()
dir(x)
x_var
dir(x)
((x - x.mean())**3).sum()/len(x)
x = np.random.normal(size=100)
y = np.random.normal(size=100)
np.cov(x,y)
(x.std()**2) * 100/99
y.std()**2 * 100/99
np.corrcoef(x,y)
x = np.random.normal(size=10000)
y = np.random.normal(size=10000)
np.corrcoef(x,y)
y = 2*(x**2) + 3
np.cov(x,y)
#### correlation 
np.corrcoef(x,y)
### Summary stats
## Mean, std, median, min, max
#### correlation 
np.corrcoef(x,y)<jupyter_output><empty_output><jupyter_text>Summary stats  Correlation 분석<jupyter_code>import numpy as np
from vega_datasets import data
from matplotlib import pyplot as plt
data.list_datasets()
df = data.cars(use_local=True)
type(df)
import numpy as np
import pandas as pd
df.head()
df.tail()
df.Miles_per_Gallon
df['Miles_per_Gallon']<jupyter_output><empty_output><jupyter_text>Data Frame을 다루는 방법<jupyter_code>df['']<jupyter_output><empty_output><jupyter_text>df<jupyter_code>df[['var1','var2']]
df[['Acceleration','Miles_per_Gallon']]
   
df[ (df['Cylinders'] >= 4) or ()  ]
df.iloc[0:10,2:4]<jupyter_output><empty_output><jupyter_text>Data Frame을 다루는데, excel 계산하는 것을 전부 python에서 계산하실 수 있으면 문제없음<jupyter_code>df['one'] = 1
df
df
df.loc[:,['Miles_per_Gallon']]
df2 = pd.read_csv("https://raw.githubusercontent.com/khbae/econometrics/master/01_Data/test_data.txt")
df2
import os
from google.colab import auth
auth.authenticate_user()

from google.colab import drive
drive.mount('/content/gdrive')
os.getcwd()
os.chdir('/content/gdrive/My Drive/Colab Notebooks/2020-11-9/data')<jupyter_output><empty_output><jupyter_text>/content/gdrive/My Drive/Colab Notebooks/2020-11-9/data/manning.csv<jupyter_code>!ls
pd.read_csv('/content/gdrive/My Drive/Colab Notebooks/2020-11-9/data/manning.csv')
df2 = pd.read_csv('manning.csv')
df2.head()
df2['xx'] = df2['y'] ** 2
df2.head()
df2.to_csv('heremyfile.csv')
df2.describe().to_csv('summary_stats.csv')
df2.corr().to_csv('correlation.csv')<jupyter_output><empty_output><jupyter_text>이미지 저장<jupyter_code>i = 10
'here' + str(i) + '.csv'
os.getcwd()
import numpy as np
import matplotlib.pyplot as plt

for i in range(0,10):
  x1 = np.linspace(0.0, 5.0)
  x2 = np.linspace(0.0, 2.0)

  y1 = np.cos(2 * np.pi * x1) * np.exp(-x1)
  y2 = np.cos(2 * np.pi * x2)

  plt.subplot(2, 1, 1)                # nrows=2, ncols=1, index=1
  plt.plot(x1, y1, 'o-')
  plt.title('1st Graph')
  plt.ylabel('Damped oscillation')

  plt.subplot(2, 1, 2)                # nrows=2, ncols=1, index=2
  plt.plot(x2, y2, '.-')
  plt.title('2nd Graph')
  plt.xlabel('time (s)')
  plt.ylabel('Undamped')

  plt.tight_layout()
  # plt.show()
  plt.savefig('./figure/savefig_default' + str(i) + '.png')<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:11: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
  # This is added back by InteractiveShellApp.init_path()
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:16: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
  app.launch_new_instance()
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:11: MatplotlibDeprecationWarning: Adding an axes using the same arg[...]<jupyter_text>Data Import Summary statistics<jupyter_code>df.describe()
df.describe()
df.Miles_per_Gallon.mean()
df.Miles_per_Gallon.std()
df.Miles_per_Gallon.min()
df.Miles_per_Gallon.max()
df.Miles_per_Gallon.quantile(q=0.25)
df.Miles_per_Gallon.quantile(q=0.75)
df
group1 = df[df.Miles_per_Gallon >= 20]
group2 = df[df.Miles_per_Gallon < 20]
group1.describe()
group2.describe()<jupyter_output><empty_output><jupyter_text>Correlation<jupyter_code>df.corr()
group1.corr()
group2.corr()
x = np.random.normal(size=10000)
plt.hist(x,bins=100)
x = np.random.normal(loc=10,scale=5,size=10000)
plt.hist(x,bins=50)
z = (x - 10)/5
plt.hist(z,bins=50)
z.std()
sample_t = np.random.standard_t(df=1000,size=10000)
plt.hist(sample_t,bins=50)
normal_x = np.random.normal(size=10000)
chi_square_one = normal_x**2
plt.hist(chi_square_one,bins=50)
normal_x1 = np.random.normal(size=10000)
normal_x2 = np.random.normal(size=10000)
chi_square_two = (normal_x1)**2 + (normal_x2)**2
plt.hist(chi_square_two,bins=50)
np.random.normal(size=10)
## God knows it. You don't know.
mu = 5 
random_sample = np.random.normal(loc=mu,size=10)
random_sample.mean()
all_est = []
for i in range(0,100):
  random_sample = np.random.normal(loc=mu,size=30)
  mu_est = random_sample.mean()
  all_est.append(mu_est)
  print(i,mu_est)
E_W = np.sum(all_est) / len(all_est)
theta = mu
E_W - theta<jupyter_output><empty_output><jupyter_text>Unbiased estimators<jupyter_code>type(all_est)
all_est2 = np.array(all_est)
(all_est2 - 5).std()<jupyter_output><empty_output><jupyter_text>efficiency<jupyter_code>df
X = df['Cylinders']
Y = df['Acceleration']
X_bar = X.mean()
Y_bar = Y.mean()
((X-X_bar) * (Y-Y_bar)).sum()
((X-X_bar)**2).sum()
beta_1_hat = ((X-X_bar) * (Y-Y_bar)).sum()/((X-X_bar)**2).sum()
beta_1_hat
beta_0_hat = Y_bar - beta_1_hat * X_bar
beta_0_hat
import statsmodels.formula.api as sm
results = sm.ols(formula='Acceleration ~ Cylinders',data=df).fit()
print(results.summary())
SST = ((Y - Y_bar)**2).sum()
SSE = ((beta_0_hat + beta_1_hat * X - Y_bar)**2).sum()
SSR = ((Y - (beta_0_hat + beta_1_hat * X))**2).sum()
SST
SSE
SSR
SSE + SSR
R_sq=SSE/SST

R_sq
print(results.summary())
beta_0_hat, beta_1_hat
df.columns
results = sm.ols(formula='Acceleration ~ Cylinders + Weight_in_lbs',data=df).fit()
df['one'] = 1
X = df.loc[:,['one','Cylinders','Weight_in_lbs']].values
Y = df.loc[:,['Acceleration']].values
np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17460793127559684, 'fraction_numerical': 0.04225352112676056, 'mean_word_length': 5.57386736703874, 'pattern_counts': {'":': 0, '<': 90, '<?xml version=': 0, '>': 87, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '48073', 'license': 'MIT', 'n_tokens_mistral': 4849, 'n_tokens_neox': 4728, 'n_words': 826, 'path': 'stats_exercise.ipynb', 'repo_name': 'khbae/econometrics', 'size': '10011'}