<jupyter_start><jupyter_text>Advanced pandas<jupyter_code>import numpy as np
import pandas as pd
np.random.seed(12345)
import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
PREVIOUS_MAX_ROWS = pd.options.display.max_rows
pd.options.display.max_rows = 20
np.set_printoptions(precision=4, suppress=True)<jupyter_output><empty_output><jupyter_text>Categorical Data  Background and Motivation* pd.unique(..)* pd.value_counts(..)<jupyter_code>import numpy as np; import pandas as pd
values = pd.Series(['apple', 'orange', 'apple',
                    'apple'] * 2)
values
pd.unique(values)
pd.value_counts(values)<jupyter_output><empty_output><jupyter_text>representing data with repeated values for more efficient storage and computation.* dimension tables containing the distinct values and storing the primary observations as integer keys referencing the dimension table<jupyter_code>values = pd.Series([0, 1, 0, 0] * 2)
values
dim = pd.Series(['apple', 'orange'])
dim
#take() : restore the original Series of strings:
dim.take(values)<jupyter_output><empty_output><jupyter_text>Categorical Type in pandas*  df['fruit'].astype('category') : convert to pandas.Categorical<jupyter_code>fruits = ['apple', 'orange', 'apple', 'apple'] * 2
N = len(fruits)
print(N)
df = pd.DataFrame({'fruit': fruits,
                   'basket_id': np.arange(N),
                   'count': np.random.randint(3, 15, size=N),
                   'weight': np.random.uniform(0, 4, size=N)},
                  columns=['basket_id', 'fruit', 'count', 'weight'])
df
print(type(df['fruit']))
df['fruit']
fruit_cat = df['fruit'].astype('category')
fruit_cat
c = fruit_cat.values
type(c)
c.categories
c.codes
df['fruit'] = df['fruit'].astype('category')
df.fruit
my_categories = pd.Categorical(['foo', 'bar', 'baz', 'foo', 'bar'])
my_categories<jupyter_output><empty_output><jupyter_text>pd.Categorical.from_codes(codes, categories)<jupyter_code># obtained categorical encoded data from another source, you can use the
# alternative from_codes constructor:
categories = ['foo', 'bar', 'baz']
codes = [0, 1, 2, 0, 0, 1]
my_cats_2 = pd.Categorical.from_codes(codes, categories)
my_cats_2
ordered_cat = pd.Categorical.from_codes(codes, categories,
                                        ordered=True)
ordered_cat
my_cats_2.as_ordered()<jupyter_output><empty_output><jupyter_text>Computations with Categoricals<jupyter_code>np.random.seed(12345)
draws = np.random.randn(1000)
draws[:5]
print(draws.min()); print(draws.max())
bins = pd.qcut(draws, 4)#1000 number -> grouping into 4 bins: (not incl, incl]...
bins
bins = pd.qcut(draws, 4, labels=['Q1', 'Q2', 'Q3', 'Q4'])
bins
bins.codes[:10]
bins.categories
# bins.categories[:10]#same as above <- no meaning.
bins = pd.Series(bins, name='quartile')
bins<jupyter_output><empty_output><jupyter_text>labeled bins categorical does not contain information about the bin edges in thedata, so we can use groupby to extract some summary statistics:<jupyter_code>results = (pd.Series(draws)
           .groupby(bins)
           .agg(['count', 'min', 'max'])
           .reset_index()) #newly index 0,1,2,3 setup
#           )
results
results['quartile']<jupyter_output><empty_output><jupyter_text>Better performance with categoricals<jupyter_code>N = 10000000
draws = pd.Series(np.random.randn(N))
labels = pd.Series(['foo', 'bar', 'baz', 'qux'] * (N // 4))
labels
categories = labels.astype('category')
categories
labels.memory_usage()
categories.memory_usage()
%time _ = labels.astype('category') #conversion to category<jupyter_output>CPU times: user 324 ms, sys: 52.3 ms, total: 376 ms
Wall time: 373 ms<jupyter_text>GroupBy operations can be significantly faster with categoricals because the underly‐ing algorithms use the integer-based codes array instead of an array of strings.  Categorical Methods*Series containing categorical data: methods* special attribute **cat** provides access to categorical methods:* set_categories() : extend existing categories<jupyter_code>s = pd.Series(['a', 'b', 'c', 'd'] * 2)
cat_s = s.astype('category')
print("** cat_s: ",type(cat_s))
cat_s
cat_s.cat #.cat: provides access to categorical methods
cat_s.cat.codes
cat_s.cat.categories<jupyter_output><empty_output><jupyter_text>set_categories() : actual set of categories for this data extends beyond thefour values observed in the data<jupyter_code>actual_categories = ['a', 'b', 'c', 'd', 'e']
cat_s2 = cat_s.cat.set_categories(actual_categories)
cat_s2<jupyter_output><empty_output><jupyter_text>While it appears that the data is unchanged, the new categories will be reflected inoperations that use them.<jupyter_code>cat_s.value_counts()
cat_s2.value_counts()
#test
cat_s.isin(['a', 'b'])
cat_s3 = cat_s[cat_s.isin(['a', 'b'])]
cat_s3<jupyter_output><empty_output><jupyter_text>In large datasets, categoricals are often used as a convenient tool for memory savingsand better performance.<jupyter_code>cat_s3.cat.remove_unused_categories() #trim unobserved categories<jupyter_output><empty_output><jupyter_text>Creating dummy variables for modeling<jupyter_code>#test
df3 = pd.DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'b'],
                   'data1': range(6)})
df3
#test
pd.get_dummies(df3['key'], prefix='key') # as for df3['key']'s value 'a' 'b' 'c' -> matched index : 1 ,else :0
cat_s = pd.Series(['a', 'b', 'c', 'd'] * 2, dtype='category')
cat_s
pd.get_dummies(cat_s)<jupyter_output><empty_output><jupyter_text>Advanced GroupBy Use  Group Transforms and "Unwrapped" GroupBys<jupyter_code>df = pd.DataFrame({'key': ['a', 'b', 'c'] * 4,
                   'value': np.arange(12.)})
df
#group means by key
# producing a Series(groupby 'key') of the same shape as df['value']
g = df.groupby('key').value #.SeriesGroupBy object 
g.mean()
#test
df.groupby('key').mean()<jupyter_output><empty_output><jupyter_text>instead of producing a Series of the same shape as df['value']; * with values, replaced by the average grouped by 'key' > pass the functionlambda x: x.mean() to transform :<jupyter_code>g.transform(lambda x: x.mean()) #return must be same size
g.transform('mean')
g.transform(lambda x: x * 2)
# ranks in descending order for each group:
g.transform(lambda x: x.rank(ascending=False)) #rank : first showing (a b c)-> 1, second -> 2 ...
def normalize(x):
    return (x - x.mean()) / x.std()
g.transform(normalize)
g.apply(normalize)
g.transform('mean')
normalized = (df['value'] - g.transform('mean')) / g.transform('std')
normalized<jupyter_output><empty_output><jupyter_text>Grouped Time Resampling* For time series data, the resample() : group operation based on a time intervalization<jupyter_code>N = 15
times = pd.date_range('2017-05-20 00:00', freq='1min', periods=N)
df = pd.DataFrame({'time': times,
                   'value': np.arange(N)})
df
# index by 'time' and then resample:
df.set_index('time').resample('5min').count()<jupyter_output><empty_output><jupyter_text>multiple time series, marked by an additionalgroup key column:<jupyter_code>df2 = pd.DataFrame({'time': times.repeat(3),
                    'key': np.tile(['a', 'b', 'c'], N), #N = 15, [] repeat 15
                    'value': np.arange(N * 3.)})#0 - 44 
print(df2.shape)
df2[:7]
df2.index<jupyter_output><empty_output><jupyter_text>* pd.TimeGrouper() was formally deprecated * pd.TimeGrouper: same resampling for each value of 'key'-> pd.Grouper() is within groupby()<jupyter_code># time_key = pd.TimeGrouper(freq='5min') #deprecated
# resampled = (df2.set_index('time')
#              .groupby(['key', time_key])
#              .sum())
# resampled
# resampled.reset_index()
#test
import pandas as pd

values = [['2017-01-15', 'Tim', 4],
          ['2016-06-17', 'Tim', 3],
          ['2018-04-02', 'Tim', 2],
          ['2017-01-15', 'Sue', 4],
          ['2016-06-17', 'Sue', 3],
          ['2018-04-02', 'Sue', 2],
          ['2017-01-15', 'Eric', 4],
          ['2016-06-17', 'Eric', 3],
          ['2018-04-02', 'Betty', 2],
          ['2017-01-15', 'Betty', 4],
          ['2016-06-17', 'Betty', 3],
          ['2018-01-02', 'Betty', 2]]
df_test1 = pd.DataFrame(values)
df_test1 
df_test1.iloc[:,[0]] #interger location
df_test1.iloc[[1,2],[0,2]]
df_test1.loc[:,[0]] #letter location
df_test1[0] #column 0
# Convert first column to datetime
df_test1[0] = df_test1[0].astype('datetime64[ns]')
# Set first two columns as index
df_test1 = df_test1.set_index([0, 1])
df_test1
# Group by Year
df_test1.groupby(pd.Grouper(level=0, freq='Y')).sum()
# Group by Person
df_test1.groupby(pd.Grouper(level=1)).sum()
# Group by Year, Person
df_test1.groupby(by=[pd.Grouper(level=0, freq='Y'), pd.Grouper(level=1)]).sum()<jupyter_output><empty_output><jupyter_text>Techniques for Method Chaining ```pythondf = load_data()df2 = df[df['col2'] < 0]df2['col1_demeaned'] = df2['col1'] - df2['col1'].mean()result = df2.groupby('key').col1_demeaned.std()``` ```python Usual non-functional waydf2 = df.copy()df2['k'] = v Functional assign waydf2 = df.assign(k=v)``` ```pythonresult = (df2.assign(col1_demeaned=df2.col1 - df2.col2.mean())          .groupby('key')          .col1_demeaned.std())``` ```pythondf = load_data()df2 = df[df['col2'] < 0]``` ```pythondf = (load_data()      [lambda x: x['col2'] < 0])``` ```pythonresult = (load_data()          [lambda x: x.col2 < 0]          .assign(col1_demeaned=lambda x: x.col1 - x.col1.mean())          .groupby('key')          .col1_demeaned.std())```  The pipe Method ```pythona = f(df, arg1=v1)b = g(a, v2, arg3=v3)c = h(b, arg4=v4)``` ```pythonresult = (df.pipe(f, arg1=v1)          .pipe(g, v2, arg3=v3)          .pipe(h, arg4=v4))``` ```pythong = df.groupby(['key1', 'key2'])df['col1'] = df['col1'] - g.transform('mean')``` ```pythondef group_demean(df, by, cols):    result = df.copy()    g = df.groupby(by)    for c in cols:        result[c] = df[c] - g[c].transform('mean')    return result``` ```pythonresult = (df[df.col1 < 0]          .pipe(group_demean, ['key1', 'key2'], ['col1']))```<jupyter_code>pd.options.display.max_rows = PREVIOUS_MAX_ROWS<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16905901116427433, 'fraction_numerical': 0.032994417862838916, 'mean_word_length': 4.7893825735718405, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '647353', 'license': 'MIT', 'n_tokens_mistral': 3821, 'n_tokens_neox': 3561, 'n_words': 833, 'path': 'ch12.ipynb', 'repo_name': 'blockchain99/data_analysis', 'size': '10032'}