<jupyter_start><jupyter_text>Forever Monsterous:  A Kung Faux Pandas Tutorial Kung Faux Pandas (KFP) is a set of tools which extends the [Pandas](https://pandas.pydata.org/ "Pandas Homepage") dataframe to include some tools for generating synthetic data.There are many reasons to create synthetic data, but KFP assumes the reason is to give a statistical analyst "realistic" data, which we term _faux_ data, without exposing the real dataset.   Details on the tools and how to install them are available on the [KFP github repository] (https://github.com/CUD2V/kungfauxpandas).This tutorial assumes you have installed KFP as well as some standard data science tools, such as numpy, scipy, and matplotlib.  Chapter 1:  The Problem Monsters are a tricky lot.   They have widely varying statistics and capabilities and many have spent centuries building up their reputations in order to disuade adventurers from invading their spaces.  The smarter varieties of monsters have retained good lawyers through the ages, and it in most jurisdictions it's actually illegal to publish the "personal" data on any monster.Of course, we would like to study the monsters without violating their privacy rights.   KFP can help this problem!1. Generate a fake data set based on the real stuff2. Work out all steps necessary to clean the data up3. Code the data cleaning steps into a function4. Re-run the data generation using the data cleaner5. Get a clean, synthetic data set to study6. Have a trusted agent (DM?) run your study code on the real data and deliver you the results.<jupyter_code>import sqlite3
import pandas as pd
import numpy as np
import scipy as sp
import scipy.stats as stats
import pylab as plt
from collections import Counter
import datetime
import sys
sys.path.append('../')
sys.path.append('../../../plugins/DataSynthesizer/DataSynthesizer/')
sys.path.append('../../../plugins/DataSynthesizer/')
from importlib import reload
from kungfauxpandas import KungFauxPandas, TrivialPlugin, DataSynthesizerPlugin, KDEPlugin
from fractions import Fraction
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Step 0:  Get the real dataThis scenario models one in which the monsters' private data is unavailable to us as researchers.  We assume there would be a trusted intermediary who could hold the real data and run the processing for us.<jupyter_code># Data from: https://docs.google.com/spreadsheets/d/1FIjaz6S0JXrXaCVhHEDeq-nH7xHzlqAx6inuRbDjhjU/edit#gid=1346812326
# Other fun data: https://docs.google.com/spreadsheets/d/16ajgJpvUI0wYcSU7kjHutw8oB6Zv6eIAo5JtI6PLvu8/edit#gid=776794522
monsterfile = "../../../data/D&D 5e Monster List with Ability Scores.xlsx - Monsters by Ability Score.csv"
df = pd.read_csv(monsterfile)
df.sample(10)<jupyter_output><empty_output><jupyter_text>Step 1: Grab round 1 of the fake data<jupyter_code># Don't need Name for this analysis
df.drop('Name', axis=1, inplace=True)

# Instantiate KFP object
kfpd = KungFauxPandas(plugin=KDEPlugin(),
                      logging='../../../data/kfp_log.db',
                      db_file='../../../data/monsters.db')

# Ensure KFP object has the options we want
kfpd.plugin.capture_covariance = True  # models how the stats interact with each other
kfpd.plugin.determine_factors =  True  # make a judgement about whether a measure is numeric or categorical

# Generate the Fake data
kfpd.plugin.fauxify(df_in=df, refactorize=False)

# Do some renames to improve readability
faux_df = kfpd.plugin.df_out
factor_indices = kfpd.plugin.factor_indices<jupyter_output>Preprocess None
Building KDE Covariate Model<jupyter_text>Now we have fake data! It still needs some work, though. Step 2: EDA and Data Cleanup<jupyter_code>faux_df.sample(5)<jupyter_output><empty_output><jupyter_text>Since we seleccted the KDE plugin, this data was generated with a Gaussian Kernel Density Estimator, which assumes that the samples come from a continuous distribution.   That is, numbers like Hit Points (HP) are assumed to take any decimal value.    Due to our domain expertise, however, we know that these numbers really should all be integers.    One could generate a more complex simulation model or just round off our fake data.<jupyter_code># Get numbers back to integers
faux_df = faux_df.round().astype('int')
faux_df.sample(5)<jupyter_output><empty_output><jupyter_text>These look more like we might expect, however some of our fields were factors (i.e. text fields).  We'd like to restore these to their original labels.<jupyter_code># Back-transform factors to their labels
for factor_name in factor_indices:
    alt = factor_indices[factor_name]
    altlab = np.arange(len(alt))
    lookup = dict(zip(altlab,alt))
    faux_df[factor_name]=[lookup[xx] for xx in faux_df[factor_name]]

faux_df.sample(5)<jupyter_output><empty_output><jupyter_text>This looks much more like what we'd expect.    Lets do some exploration and cleaning.<jupyter_code>distincts = faux_df.Type.unique()
print(distincts)
distincts.sort()
print(distincts)<jupyter_output>['Monstrosity' 'Beast' 'Undead' 'Fiend (Demon)' 'Humanoid' 'Dragon'
 'Giant' 'Construct' 'Aberration' 'Fiend (Devil)' 'Plant' 'Elemental'
 'Fey' 'Fiend' 'Ooze' 'Celestial']
['Aberration' 'Beast' 'Celestial' 'Construct' 'Dragon' 'Elemental' 'Fey'
 'Fiend' 'Fiend (Demon)' 'Fiend (Devil)' 'Giant' 'Humanoid' 'Monstrosity'
 'Ooze' 'Plant' 'Undead']<jupyter_text>+ **Boo!  Why is Giant in there twice? A trailing space!**Lets check the other factors.<jupyter_code>distincts = faux_df.Size.unique()
print(distincts)
distincts.sort()
print(distincts)<jupyter_output>['Large' 'Medium' 'Huge' 'Gargantuan' 'Tiny' 'Small' 'large' 'VARIES'
 'small']
['Gargantuan' 'Huge' 'Large' 'Medium' 'Small' 'Tiny' 'VARIES' 'large'
 'small']<jupyter_text>+ **ooh!  There's "large" and "Large", "small and "Small", and "VARIES" changes capitalization convention.**<jupyter_code>distincts = faux_df.ALIGNMENT.unique()
print(distincts)
distincts.sort()
print(distincts)<jupyter_output>['U' 'CE' 'ANY' 'LE' 'N' 'LG' 'NE' 'LN' 'NOT LAWFUL' 'NG' 'CN' 'NOT GOOD'
 'CG' 'E' 'C' 'ANY EVIL' 'CG OR NE']
['ANY' 'ANY EVIL' 'C' 'CE' 'CG' 'CG OR NE' 'CN' 'E' 'LE' 'LG' 'LN' 'N'
 'NE' 'NG' 'NOT GOOD' 'NOT LAWFUL' 'U']<jupyter_text>* **No problems jump out here**<jupyter_code>distincts = faux_df.CR.unique()
print(distincts)
distincts.sort()
print(distincts)<jupyter_output>['4' '2' '1' '12' '1/4' '6' '9' '8' '16' '5' '24' '3' '11' '1/2' '23'
 '1/8' '13' '14' '7' '0' '10' '19' '20' '21' '30' '15' '17' '26' '18' '22'
 '25']
['0' '1' '1/2' '1/4' '1/8' '10' '11' '12' '13' '14' '15' '16' '17' '18'
 '19' '2' '20' '21' '22' '23' '24' '25' '26' '3' '30' '4' '5' '6' '7' '8'
 '9']<jupyter_text>*  **This has numbers, mostly.  There's a '-' for some reason.   Also, there are fractions which we might want to be numerical.**Lets build a preprocessor that can fix these problems for us.<jupyter_code># Preprocessing instructions
def df_preprocess(df):

    # Get rid of rows with missing data
    df=df.copy().dropna()
    
    # Take care of capitalization problems
    df.loc[:,'Size']=[xx.capitalize().strip() for xx in df.Size]
    df.loc[:,'Type']=[xx.capitalize().strip() for xx in df.Type]
    df.loc[:,'ALIGNMENT'] = [xx.upper().strip() for xx in df.ALIGNMENT]
    
    # Convert any '-' values to a 0
    df.loc[:,'CR'] = [float(Fraction(xx)) if xx is not '-' else 0.0 for xx in df.CR]
    return(df)<jupyter_output><empty_output><jupyter_text>Lets verify that df_preprocess seems to fix our problems on our original data set.<jupyter_code>fixed = df_preprocess(faux_df)
print(fixed.Size.unique(), '\n\n', fixed.Type.unique(),'\n\n',fixed.ALIGNMENT.unique())

print(fixed.CR.unique())<jupyter_output>['Large' 'Medium' 'Huge' 'Gargantuan' 'Tiny' 'Small' 'Varies'] 

 ['Monstrosity' 'Beast' 'Undead' 'Fiend (demon)' 'Humanoid' 'Dragon'
 'Giant' 'Construct' 'Aberration' 'Fiend (devil)' 'Plant' 'Elemental'
 'Fey' 'Fiend' 'Ooze' 'Celestial'] 

 ['U' 'CE' 'ANY' 'LE' 'N' 'LG' 'NE' 'LN' 'NOT LAWFUL' 'NG' 'CN' 'NOT GOOD'
 'CG' 'E' 'C' 'ANY EVIL' 'CG OR NE']
[ 4.     2.     1.    12.     0.25   6.     9.     8.    16.     5.
 24.     3.    11.     0.5   23.     0.125 13.    14.     7.     0.
 10.    19.    20.    21.    30.    15.    17.    26.    18.    22.
 25.   ]<jupyter_text>Generate fake data based on the original _after_ preprocessing.<jupyter_code># Instantiate KFP object
kfpd = KungFauxPandas(plugin=KDEPlugin(),
                      logging='../../../data/kfp_log.db',
                      db_file='../../../data/monsters.db')

# Ensure KFP object has the options we want
kfpd.plugin.capture_covariance = True
kfpd.plugin.determine_factors =  True

# Generate the Fake data
kfpd.plugin.fauxify(df_in=df, preprocess=df_preprocess, refactorize=False)

# Do some renames to improve readability
faux_df = kfpd.plugin.df_out
factor_indices = kfpd.plugin.factor_indices
# Get numbers back to integers
faux_df = faux_df.round().astype('int')
faux_df.sample(10)
factor_indices
# Back-transform factors to their labels
for factor_name in factor_indices:
    alt = factor_indices[factor_name]
    altlab = np.arange(len(alt))
    lookup = dict(zip(altlab,alt))
    faux_df[factor_name]=[lookup[xx] for xx in faux_df[factor_name]]


faux_df.sample(5)<jupyter_output><empty_output><jupyter_text>**We now have fake data which strongly resembles the real stuff!**  Comparing the Faux Data to the OriginalAny analysis you might be interested in can now be run on the faux data and you can expect the results.One of the numbers included in the stats is the so-called "Challenge Rating" (CR).  The meaning is detailed [here](https://www.reddit.com/r/DnD/comments/2egr5y/explaining_5e_cr/), but suffice it to say, it's a popular measure of the monster's dangerousness.One way to test our faux data is to make sure that the generated CR correlates to the generated monsters' stats in the same way that the real CR correlates to the real monsters' stats.  For this, we can use a linear regression.  Faux Data<jupyter_code>import statsmodels.api as sm
from fractions import Fraction
import numpy
#reg = linear_model.LinearRegression()

y = [float(Fraction(xx))  if xx !='-' else 0.0 for xx in faux_df.CR]
#X = faux_df.iloc[:,[xx in ['HP','AC','STR','DEX','INT','CON','WIS','CHA'] for xx in faux_df.columns]]
X = faux_df.loc[:,[xx in ['HP','STR','WIS'] for xx in faux_df.columns]]
#X = sm.add_constant(X)  

M1 = sm.OLS(y,X)
print(M1.fit().summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.929
Model:                            OLS   Adj. R-squared:                  0.929
Method:                 Least Squares   F-statistic:                     2991.
Date:                Wed, 17 Oct 2018   Prob (F-statistic):               0.00
Time:                        22:17:05   Log-Likelihood:                -1524.5
No. Observations:                 690   AIC:                             3055.
Df Residuals:                     687   BIC:                             3069.
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>Real Data<jupyter_code>import statsmodels.api as sm
from fractions import Fraction
import numpy

df = df_preprocess(df)  # Use the same pre-processor used to generate the faux data


y2 = [float(Fraction(xx))  if not np.isnan(xx) else 0 for xx in df.CR]
#X2 = df.iloc[:,[xx in ['HP','AC','STR','DEX','INT','CON','WIS','CHA'] for xx in df.columns]]
X2 = df.loc[:,[xx in ['HP','STR','WIS'] for xx in df.columns]]

#X2 = sm.add_constant(X2)  
X2.columns

M2 = sm.OLS(y2,X2)
print(M2.fit().summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.930
Model:                            OLS   Adj. R-squared:                  0.930
Method:                 Least Squares   F-statistic:                     3056.
Date:                Wed, 17 Oct 2018   Prob (F-statistic):               0.00
Time:                        22:17:05   Log-Likelihood:                -1521.2
No. Observations:                 690   AIC:                             3048.
Df Residuals:                     687   BIC:                             3062.
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>Comparing Histograms<jupyter_code>%matplotlib inline
plt.subplot(1,2,1)
plt.hist(y)
plt.subplot(1,2,2)
plt.hist(y2)
numrows = 2#len(colz)

#colz = ['CR','HP','AC','STR','DEX','INT','CON','WIS','CHA']
colz = ['STR','DEX','CON','INT','WIS','CHA']

for thiscol in enumerate(colz[0:2]):    
    plt.subplot(numrows,2,2*thiscol[0]+1)
    plt.hist(df[thiscol[1]])
    plt.xlabel(thiscol[1]+' Real')
    plt.subplot(numrows,2,2*thiscol[0]+2)
    plt.hist(faux_df[thiscol[1]])
    plt.xlabel(thiscol[1]+' Faux')
    
numrows = 3
for thiscol in enumerate(colz[3:]):
    plt.subplot(numrows,2,2*thiscol[0]+1)
    plt.hist(df[thiscol[1]])
    plt.subplot(numrows,2,2*thiscol[0]+2)
    plt.hist(faux_df[thiscol[1]])
#colz = ['CR','HP','AC','STR','DEX','INT','CON','WIS','CHA']
colz = ['STR','DEX','CON','INT','WIS','CHA']
X3 = df.iloc[:,[xx in colz for xx in df.columns]]
plt.subplot(1,2,1)
f1=plt.imshow(np.corrcoef(X3.transpose()),cmap='copper')
f1.axes.set_xticklabels(colz)
f1.axes.set_xticks(range(len(colz)))
f1.axes.set_yticklabels(colz)
f1.axes.set_yticks(range(len(colz)))
plt.colorbar()
plt.subplot(1,2,2)
X4 = faux_df.loc[:,[xx in colz for xx in df.columns]]
f2=plt.imshow(np.corrcoef(X4.transpose()),cmap='copper')
f2.axes.set_xticklabels(colz)
f2.axes.set_xticks(range(len(colz)))
f2.axes.set_yticklabels(colz)
f2.axes.set_yticks(range(len(colz)))

plt.colorbar()
plt.hist(df.DEX)
plt.hist(df.STR)
plt.hist(df.CON)
plt.hist(df.INT)

plt.hist(df.WIS)
plt.hist(df.CHA)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1496641041623381, 'fraction_numerical': 0.026663896391716877, 'mean_word_length': 3.4444444444444446, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 59, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '339448', 'license': 'Apache-2.0', 'n_tokens_mistral': 5230, 'n_tokens_neox': 4887, 'n_words': 1419, 'path': 'sourcecode/python/notebooks/kfp_dork_test.ipynb', 'repo_name': 'magic-lantern/faker-prototype', 'size': '14439'}