<jupyter_start><jupyter_text>Let's start from what we have seen in the previous notebook...  Let's read some wind data<jupyter_code># first, the imports
import os
import datetime as dt

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(19760812)
%matplotlib inline
# we read data from file 'mast.txt'
ipath = os.path.join('Datos', 'mast.txt')

# Now, we define a function to parse the dates
def dateparse(date, time):
    YY = 2000 + int(date[:2])
    MM = int(date[2:4])
    DD = int(date[4:])
    hh = int(time[:2])
    mm = int(time[2:])
    
    return dt.datetime(YY, MM, DD, hh, mm, 0)
    

cols = ['Date', 'time', 'wspd', 'wspd_max', 'wdir',
        'x1', 'x2', 'x3', 'x4', 'x5', 
        'wspd_std']
wind = pd.read_csv(ipath, sep = "\s*", names = cols, 
                   parse_dates = [[0, 1]], index_col = 0,
                   date_parser = dateparse)<jupyter_output><empty_output><jupyter_text>Basic information (in this case from a  `DataFrame`)<jupyter_code>wind.info()
wind.describe()<jupyter_output><empty_output><jupyter_text>Access the índexes, values, columns (`Series` doesn't have this attribute):<jupyter_code>wind.index
wind.values
wind.values.shape
wind.columns<jupyter_output><empty_output><jupyter_text>Removing/extracting columns There are some columns that are not interesting (named `'x1'`, `'x2'`, `'x3'`, `'x4'` and `'x5'`). We will remove these columns from our `DataFrame`. We have several options to do so:<jupyter_code># We remove a column using 'del' keyword
del wind['x1']
wind.head(3)
# We extract a column using 'pop' method
s = wind.pop('x2')
wind.head(3)
del wind['x3']
del wind['x4']
del wind['x5']
wind.info()<jupyter_output><empty_output><jupyter_text>One of the columns, that extracted using the `pop` method, is referenced using the  `s` variable, and it is a `Series`:<jupyter_code>type(s)
s.head(3)
s.info()<jupyter_output><empty_output><jupyter_text>Is it a TimeSeries?, i.e., Are all the indexes dates?<jupyter_code># s.is_time_series deprecated
s.index.is_all_dates
s.describe()
s.dtype
s.values
s.index
s.columns<jupyter_output><empty_output><jupyter_text>Working with the indexes<jupyter_code># We create a DataFrame
df = pd.DataFrame(np.array([['a','b','c','d','e'], [10,20,30,40,50]]).T,
                  columns = ['col1', 'col2'])
df<jupyter_output><empty_output><jupyter_text>We can re-write the indexes at any time:<jupyter_code>df.index = np.arange(1,6) * 100
df<jupyter_output><empty_output><jupyter_text>We can use a column to define our indexes:<jupyter_code>df.set_index('col1', inplace = True)
df<jupyter_output><empty_output><jupyter_text>We can undo the `set_index` action using:<jupyter_code>df.reset_index(inplace = True)
df<jupyter_output><empty_output><jupyter_text>As with indexes, we can change the name of the columns:<jupyter_code>df.columns = ['column1', 'column2']
df<jupyter_output><empty_output><jupyter_text>The indexes 'column' can have a name (already seen before):<jupyter_code>df.index.name = 'indices'
df<jupyter_output><empty_output><jupyter_text>`pandas` data structures are numpy arrays on steroids Don't forget that behind the scenes we have numpy arrays and `pandas` exposes much of the numpy arrays functionality directly from their data structures.We can see, for instance, what attributes of a numpy array have an equivalent directly in a `Series` (or `DataFrame`):<jupyter_code>numpy_attrs = dir(s.values)
series_attrs = dir(s)
for attr in numpy_attrs:
    if attr not in series_attrs:
        print('NOOOOOOOOOOOOOOOOOOOOOO', attr)
    else:
        print(attr)<jupyter_output><empty_output><jupyter_text>So, a lot of operations we do with a numpy array can be made directly from a `pandas` data structure:<jupyter_code>s.mean()
s.min()
s.max()
s[0:10].tolist()<jupyter_output><empty_output><jupyter_text>... Note:Sometimes could be convenient to use directly the numpy arrays method when performance is an issue.<jupyter_code>%%timeit 
s.mean()
s.min()
s.max()
%%timeit 
s.values.mean()
s.values.min()
s.values.max()<jupyter_output><empty_output><jupyter_text>And where are the steroids? Be patient!!!!!!  'Stuff' that are in a `Series` but not in a numpy array<jupyter_code>numpy_attrs = dir(s.values)
series_attrs = dir(s)
for attr in series_attrs:
    if attr not in numpy_attrs:
        print('NOOOOOOOOOOOOOOOOOOOOOO', attr)
    else:
        print(attr)<jupyter_output><empty_output><jupyter_text>'Stuff' that are in a `DataFrame` but not in a numpy array<jupyter_code>numpy_attrs = dir(s.values)
dataframe_attrs = dir(wind)
for attr in dataframe_attrs:
    if attr not in numpy_attrs:
        print('NOOOOOOOOOOOOOOOOOOOOOO', attr)
    else:
        print(attr)<jupyter_output><empty_output><jupyter_text>Examples of some useful operations.We will see some of this in a more detailed manner and with examples in the next notebooks.<jupyter_code>wind['wspd'].apply(lambda x: str(x) + ' m/s')
wind.corr()
wind.cumsum()
wind.diff()<jupyter_output><empty_output><jupyter_text>Now, we are skimming this. Later we will see it in a more detailed way:Let's do some simple examples:<jupyter_code># Calculate the mean wind speed (column 'wspd'):

# Calculate the median of the wind direction (column 'wdir'):

# Obtain the maximum difference between two time steps
# (column 'wspd_std')<jupyter_output><empty_output><jupyter_text>Other interesting methods are the `pd.rolling_*`:<jupyter_code>pd.rolling_mean(wind, 5, center = True).head(10)<jupyter_output><empty_output><jupyter_text>As you can read in the previous warning message the `rolling_*` functions are deprecated and will not be available in the near future. In the previous text cell I wrote explicitly 'methods' because all the `rolling_*` functions now are grouped in the `rolling` method. How we can do it with the `rolling` method:<jupyter_code>wind.rolling(5, center = True).mean().head(10)<jupyter_output><empty_output><jupyter_text>Other interesting 'stuff' in a `DataFrame` (change `DataFrame` with `Series` or other data structures):<jupyter_code>import inspect
info = inspect.getmembers(wind, predicate=inspect.ismethod)

for stuff in info:
    print(stuff[0])<jupyter_output><empty_output><jupyter_text>Working with missing dataIt is quite usual that our datasets have missing data.<jupyter_code>index = pd.date_range('2000/01/01', freq = '12H', periods = 10)
index = index.append(pd.date_range('2000/01/10', freq = '1D', periods = 3))
df = pd.DataFrame(np.random.randint(1, 100, size = (13, 3)), 
                  index = index, columns = ['col1', 'col2', 'col3'])
df
# Let's fill some values with NaN
df[df > 70] = np.nan
df<jupyter_output><empty_output><jupyter_text>As opposed to what happens with a numpy array, in `pandas`, operations ignore `NaN` values unless we explicitly state the opposite. Let's see this in action:<jupyter_code>df['col1'].sum()
df['col1'].values.sum()
df['col1'].sum(skipna = False)<jupyter_output><empty_output><jupyter_text>We can detect 'null' values (`NaN`) using `isnull`:<jupyter_code>df.isnull()<jupyter_output><empty_output><jupyter_text>Or not null using `notnull`:<jupyter_code>df.notnull()<jupyter_output><empty_output><jupyter_text>We can see that we have `NaN` values. We can fill them using `ffill` or `bfill` (similar to `fillna(method = 'ffill')` and to `fillna(method = 'bfill')`, respectively):<jupyter_code># Let's remember how is our DataFrame
df
df.ffill()
df.bfill()
df.fillna(value = 'Kiko')<jupyter_output><empty_output><jupyter_text>Let's create a new `DataFrame` with indexes with 12H frequency.<jupyter_code>df = pd.DataFrame(np.random.randint(1, 100, size = (15, 3)), 
                  index = pd.date_range('2015/01/01', freq = '12H', periods = 15))
df
df[df > 70] = 'Kiko'
df<jupyter_output><empty_output><jupyter_text>We can remove rows or columns that have a `NaN` value, all `NaN` values,...<jupyter_code>df[df == 'Kiko'] = np.nan
df
# We remove the rows where any value of the row is NaN
# axis = 0 would be equivalent to axis = 'rows' or axis = 'index'
# Later we will see more about the axis keyword...
df.dropna(axis = 'rows') 
# Let's remove the rows where all the values in the row are NaN
df.iloc[2, :] = np.nan
df.dropna(axis = 'rows', how = 'all')
# We can remove columns where any valu in the column is a NaN
df.dropna(axis = 'columns', how = 'any') # axis = 1 is equivalent to axis = 'columns'. More on this later.
                                         # how = 'any' is he default value so we don't need to add it.
# Let's add a column only with not null values and let's repeat the operation
df['col4'] = 9999
df.dropna(axis = 'columns', how = 'any')
# Now let's add a column where all the values are NaN
df['col5'] = np.nan
df.dropna(axis = 'columns', how = 'all')<jupyter_output><empty_output><jupyter_text>We can also fill `NaN` values using `interpolate`:<jupyter_code>df.interpolate()<jupyter_output><empty_output><jupyter_text>But, what is happening here!!! Why null values are not being interpolated? Let's see how are the columns.<jupyter_code>df.info()<jupyter_output><empty_output><jupyter_text>We can see columns `0`, `1` y `2` are of type `object` and this type is not a number. On the other hand, in the column `col4` there isn't any value to interpolate. Last, in column `col5` all the values are `NaN`. Let's convert the first three columns to interpolate:<jupyter_code>df[[0, 1, 2]] = df[[0, 1, 2]].astype(np.float)
df.interpolate()
# Have a look to the docs of the 'interpolate' method to know how to use it<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12767295597484277, 'fraction_numerical': 0.015723270440251572, 'mean_word_length': 5.27284681130835, 'pattern_counts': {'":': 0, '<': 129, '<?xml version=': 0, '>': 131, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '597400', 'license': 'MIT', 'n_tokens_mistral': 3458, 'n_tokens_neox': 3300, 'n_words': 1002, 'path': 'workshops_materials/20160408_1100_Pandas_for_beginners/tutorial/EN - Tutorial 03 - Basic operations with pandas data structures.ipynb', 'repo_name': 'PyDataMadrid2016/Conference-Info', 'size': '9540'}