<jupyter_start><jupyter_text>Wind Statistics  Introduction:The data have been modified to contain some missing values, identified by NaN.  Using pandas should make this exerciseeasier, in particular for the bonus question.You should be able to perform all of these operations without usinga for loop or other looping construct.1. The data in 'wind.data' has the following format:<jupyter_code>"""
Yr Mo Dy   RPT   VAL   ROS   KIL   SHA   BIR   DUB   CLA   MUL   CLO   BEL   MAL
61  1  1 15.04 14.96 13.17  9.29   NaN  9.87 13.67 10.25 10.83 12.58 18.50 15.04
61  1  2 14.71   NaN 10.83  6.50 12.62  7.67 11.50 10.04  9.79  9.67 17.54 13.83
61  1  3 18.50 16.88 12.33 10.13 11.17  6.17 11.25   NaN  8.50  7.67 12.75 12.71
"""<jupyter_output><empty_output><jupyter_text>The first three columns are year, month and day.  The   remaining 12 columns are average windspeeds in knots at 12   locations in Ireland on that day.      More information about the dataset go [here](wind.desc).  Step 1. Import the necessary libraries<jupyter_code>import numpy as np
import pandas as pd
pd.set_option('display.max_rows',10)
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Step 2. Import the dataset from this [address](https://raw.githubusercontent.com/guipsamora/pandas_exercises/master/06_Stats/Wind_Stats/wind.data)  Step 3. Assign it to a variable called data and replace the first 3 columns by a proper datetime index.<jupyter_code>#data = np.loadtxt("wind.data", skiprows=2)
data = pd.read_csv("wind.data", sep='\s+')
data.drop(columns=data.columns[data.columns.str.contains("Unnamed:")], inplace=True)
print(data.shape)
data.head(20)
data.dtypes
# Is the year always in one and only one century ?
plt.plot(np.arange(data.shape[0]), data['Yr'])
plt.gcf().patch.set_facecolor('white')
# -> Yes, therefore, let's supose it's the XXth century:
data['Yr'] = data['Yr']+1900
# How is the month column filled ?
fig, axs = plt.subplots(1,2)
axs[0].hist(data['Mo'], bins=range(0,14), width=0.9)
axs[1].plot(range(data.shape[0]), data['Mo'])
axs[1].set(ylim=(0,13))
plt.gcf().patch.set_facecolor('white')
fig.set_size_inches(11,3)
# Let's fill the months

an = 31*[1]+28*[2]+31*[3]+30*[4]+31*[5]+30*[6]+31*[7]+31*[8]+30*[9]+31*[10]+30*[11]+31*[12]
an_b = 31*[1]+29*[2]+31*[3]+30*[4]+31*[5]+30*[6]+31*[7]+31*[8]+30*[9]+31*[10]+30*[11]+31*[12]
biss = [1964, 1968, 1972, 1976, 1980, 1984, 1988, 1992, 1996]
tab_months = []
for y in range(1961,1979):
    if y in biss:
        tab_months += an_b
    else:
        tab_months += an
len(tab_months)
data['Mo'] = tab_months

fig, axs = plt.subplots(1,2)
axs[0].hist(data['Mo'], bins=range(0,14), width=0.9)
axs[1].plot(range(data.shape[0]), data['Mo'].interpolate('linear'))
axs[1].set(ylim=(0,12))
plt.gcf().patch.set_facecolor('white')
fig.set_size_inches(11,3)

# How is the day column filled ?
fig, axs = plt.subplots(1,2)
axs[0].hist(data['Dy'], bins=range(0,32), width=0.9)
axs[1].plot(range(data.shape[0]), data['Dy'])
axs[1].set(ylim=(0,40))
plt.gcf().patch.set_facecolor('white')
fig.set_size_inches(11,3)
# Let's fill the days
m_31 = list(range(1,32))
m_30 = list(range(1,31))
m_29 = list(range(1,30))
m_28 = list(range(1,29))
days = m_31+m_28+m_31+m_30+m_31+m_30+m_31+m_31+m_30+m_31+m_30+m_31
days_b = m_31+m_29+m_31+m_30+m_31+m_30+m_31+m_31+m_30+m_31+m_30+m_31
biss = [1964, 1968, 1972, 1976, 1980, 1984, 1988, 1992, 1996]
tab_days = []
for y in range(1961,1979):
    if y in biss:
        tab_days += days_b
    else:
        tab_days += days
len(tab_days)
data['Dy'] = tab_days
fig, axs = plt.subplots(1,2)
axs[0].hist(data['Dy'], bins=range(0,32), width=0.9)
axs[1].plot(range(data.shape[0]), data['Dy'])
axs[1].set(ylim=(0,40))
plt.gcf().patch.set_facecolor('white')
fig.set_size_inches(11,3)
data.dtypes
data.rename(columns={'Yr': 'year', 'Mo': 'month', 'Dy': 'day'}, inplace=True)
data.index = pd.to_datetime(data[['year', 'month', 'day']])
data.tail(2)<jupyter_output><empty_output><jupyter_text>Step 4. Year 2061? Do we really have data from this year? Create a function to fix it and apply it.<jupyter_code>data['year'].unique()
# I don't seem to have this problem...<jupyter_output><empty_output><jupyter_text>Step 5. Set the right dates as the index. Pay attention at the data type, it should be datetime64[ns].<jupyter_code># Already done<jupyter_output><empty_output><jupyter_text>Step 6. Compute how many values are missing for each location over the entire record.   They should be ignored in all calculations below.<jupyter_code>import missingno as msno
msno.matrix(data)
fig = plt.gcf()
fig.patch.set_facecolor('white')
fig.set_size_inches(12,3)
msno.bar(data)
fig = plt.gcf()
fig.patch.set_facecolor('white')
fig.set_size_inches(12,3)
data.isna().sum()<jupyter_output><empty_output><jupyter_text>Step 7. Compute how many non-missing values there are in total.<jupyter_code>filled = data.iloc[:,3:].notna().sum().sum()
missing = data.iloc[:,3:].isna().sum().sum()
total = data.iloc[:,3:].shape[0]*data.iloc[:,3:].shape[1]
print(filled, missing, total, total==(missing+filled))<jupyter_output>78857 31 78888 True<jupyter_text>Step 8. Calculate the mean windspeeds of the windspeeds over all the locations and all the times. A single number for the entire dataset.<jupyter_code># Solution 1
print(data.iloc[:,3:].agg(np.sum).agg(np.sum)/filled)

# Solution 2
print(data.iloc[:,3:].stack().agg(np.mean))<jupyter_output>10.227883764282167
10.22788376428219<jupyter_text>Step 9. Create a DataFrame called loc_stats and calculate the min, max and mean windspeeds and standard deviations of the windspeeds at each location over all the days  A different set of numbers for each location.<jupyter_code>loc_stats = data.iloc[:,3:].describe().loc[['min', 'max', 'mean', 'std']]
loc_stats<jupyter_output><empty_output><jupyter_text>Step 10. Create a DataFrame called day_stats and calculate the min, max and mean windspeed and standard deviations of the windspeeds across all the locations at each day. A different set of numbers for each day.<jupyter_code># Solution 1
day_stats = data.iloc[:,3:].agg(['min', 'max', 'mean', 'std'], axis=1) # 10.3 s ± 1.04 s
day_stats
# Solution 2
day_stats = data.iloc[:,3:].T.describe().loc[['min', 'max', 'mean', 'std']] # 20.6 s ± 798 ms
day_stats
plt.plot(day_stats.T.index, day_stats.T['mean'])
plt.gcf().set_facecolor('w')
plt.gcf().set_size_inches(16,3)<jupyter_output><empty_output><jupyter_text>Step 11. Find the average windspeed in January for each location.   Treat January 1961 and January 1962 both as January.<jupyter_code>data[data['month']==1].iloc[:,3:].agg(np.mean).to_frame().rename(columns={0:'mean'}).T<jupyter_output><empty_output><jupyter_text>Step 12. Downsample the record to a yearly frequency for each location.<jupyter_code># Solution 1
data.groupby('year').mean().iloc[:,2:].head(2)
# Solution 2
data.reset_index().groupby(pd.Grouper(key='index', freq='Y')).mean().iloc[:,3:].head(2)
# Solution 3
data.resample('A').mean().iloc[:,3:].head(2)
# Solution 4
data.groupby(data.index.to_period('Y')).mean().iloc[:,3:].head(2)<jupyter_output><empty_output><jupyter_text>Step 13. Downsample the record to a monthly frequency for each location.<jupyter_code># Solution 1
data.groupby(['year', 'month']).mean()
# Solution 2
data.resample('M').mean().head(2)<jupyter_output><empty_output><jupyter_text>Step 14. Downsample the record to a weekly frequency for each location.<jupyter_code>data.resample('W').agg(np.mean).iloc[:,3:].head(2)
data.reset_index().groupby(pd.Grouper(key='index', freq='W')).mean().iloc[:,3:].head(2)
# Solution 3
data.groupby(data.index.to_period('W')).mean().iloc[:,3:].head(2)<jupyter_output><empty_output><jupyter_text>Step 15. Calculate the min, max and mean windspeeds and standard deviations of the windspeeds across all locations for each week (assume that the first week starts on January 2 1961) for the first 52 weeks.<jupyter_code>df_W = data.resample('W', loffset='1D').mean().iloc[:,3:].agg([np.min, np.mean, np.max],axis=1)
df_W.T.iloc[:,0:52]
df_W.plot(marker='o', markersize=2, lw=0.5)
plt.gcf().set_facecolor('w')
plt.gcf().set_size_inches(17,3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1694499017681729, 'fraction_numerical': 0.07576129666011788, 'mean_word_length': 5.8734177215189876, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '586420', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3527, 'n_tokens_neox': 3155, 'n_words': 733, 'path': '06_Stats/Wind_Stats/Exercises.ipynb', 'repo_name': 'SmellyArmure/SmellyArmure', 'size': '8144'}