<jupyter_start><jupyter_text>"E is for Exploratory Data Analysis: Categorical Data"> What is Exploratory Data Analysis (EDA), why is it done, and how do we do it in Python?- toc: false - badges: True- comments: true- categories: [E]- hide: False- image: images/e-is-for-eda-text/alphabet-close-up-communication-conceptual-278887.jpg  _What is **Exploratory Data Analysis(EDA)**?_While I answered these questions in the [last post](https://educatorsrlearners.github.io/an-a-z-of-machine-learning/e/2020/06/15/e-is-for-eda.html), since [all learning is repetition](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=224340), I'll do it again :grin:  EDA is an ethos for how we scrutinize data including, but not limited to: - what we look for (i.e. shapes, trends, outliers) - the approaches we employ (i.e. [five-number summary](https://www.statisticshowto.com/how-to-find-a-five-number-summary-in-statistics/), visualizations)- and the decisions we reach{% fn 1 %}   _Why is it done?_Two main reasons: 1. If we collected the data ourselves, we need to know if our data suits our needs or if we need to collect more/different data.   2. If we didn't collect the data ourselves, we need to interrogate the data to answer the "5 W's"- __What__ kind of data do we have (i.e. numeric, categorical)?- __When__ was the data collected? There could be more recent data which we could collect which would better inform our model.- __How__ much data do we have? Also, how was the data collected? - __Why__ was the data collected? The original motivation could highlight potential areas of bias in the data. - __Who__ collected the data? Some of these questions can't necessarily be answered by looking at the data alone which is fine because _[nothing comes from nothing](http://parmenides.me/nothing-comes-from-nothing/)_; someone will know the answers so all we have to do is know where to look and whom to ask.  _How do we do it in Python?_As always, I'll follow the steps outlined in [_Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow_](https://github.com/ageron/handson-ml/blob/master/ml-project-checklist.md) Step 1: Frame the Problem"Given a set of features, can we determine how old someone needs to be to read a book?"  Step 2: Get the DataWe'll be using the same dataset as in the [previous post](https://educatorsrlearners.github.io/an-a-z-of-machine-learning/e/2020/06/15/e-is-for-eda.html).  Step 3: Explore the Data to Gain Insights (i.e. EDA)As always, import the essential libraries, then load the data.<jupyter_code>#hide
import warnings; warnings.simplefilter('ignore')
#For data manipulation
import pandas as pd
import numpy as np

#For visualization
import seaborn as sns
import matplotlib.pyplot as plt
import missingno as msno 

url = 'https://raw.githubusercontent.com/educatorsRlearners/book-maturity/master/csv/book_info_complete.csv'

df = pd.read_csv(url)<jupyter_output><empty_output><jupyter_text>To review,***How much data do we have?***<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>- 23 features- one target- 5,816 observations***What type of data do we have?***<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5816 entries, 0 to 5815
Data columns (total 24 columns):
 #   Column                          Non-Null Count  Dtype  
---  ------                          --------------  -----  
 0   title                           5816 non-null   object 
 1   description                     5816 non-null   object 
 2   plot                            5816 non-null   object 
 3   csm_review                      5816 non-null   object 
 4   need_to_know                    5816 non-null   object 
 5   par_rating                      2495 non-null   float64
 6   kids_rating                     3026 non-null   float64
 7   csm_rating                      5816 non-null   int64  
 8   Author                          5468 non-null   object 
 9   Genre                           5816 non-null   object 
 10  Topics                          3868 non-null   object 
 11  Book type                       5816 non-null   object 
 12  Publisher                       56[...]<jupyter_text>Looks like mostly categorical with some numeric. Lets take a closer look.<jupyter_code>df.head().T<jupyter_output><empty_output><jupyter_text>Again, I collected the data so I know the target is `csm_rating` which is the minimum age Common Sense Media (CSM) says a reader should be for the given book.Also, we have essentially three types of features:- Numeric     - `par_rating` : Ratings of the book by parents    - `kids_rating` : Ratings of the book by children      - :dart:`csm_rating` : Ratings of the books by Common Sense Media    - `Number of pages` : Length of the book     - `Publisher's recommended age(s)`: Self explanatory    - Date    - `Publication date` : When the book was published    - `Last updated`: When the book's information was updated on the websitewith the rest of the features being categorical and text; these features will be our focus for today.   Step 3.1 Housekeeping  Clean the feature names to make inspection easier. {% fn 3 %}<jupyter_code>df.columns
df.columns = df.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
df.columns<jupyter_output><empty_output><jupyter_text>Much better. Now lets subset the data frame so we only have the features of interest. Given there are twice as many text features compared to non-text features, and the fact that I'm  ~~lazy~~ efficient, I'll create a list of the features I ***don't*** want<jupyter_code>numeric = ['par_rating', 'kids_rating', 'csm_rating', 'number_of_pages', "publisher's_recommended_ages", "publication_date", "last_updated"]<jupyter_output><empty_output><jupyter_text>and use it to keep the features I ***do*** want.<jupyter_code>df_strings = df.drop(df[numeric], axis=1)<jupyter_output><empty_output><jupyter_text>_Voila!_<jupyter_code>df_strings.head().T<jupyter_output><empty_output><jupyter_text>Clearly, the non-numeric data falls into two groups:- text     - `description`    - `plot`    - `csm_review`    - `need_to_know`- categories     - `author`/`authors`    - `genre`    - `award`/`awards`    - etc.    Looking at the output above, so many questions come to mind: 1. How many missing values do we have?2. How long are the descriptions? 3. What's the difference between `csm_review` and `need_to_know`?4. Similarly, what the difference between `description` and `plot`?5. How many different authors do we have in the dataset?6. How many types of books do we have? and I'm sure more will arise once we start. Where to start? Lets answer the easiest questions first :grin: Categories ***How many missing values do we have?***  A cursory glance at the output above indicates there are potentially a ton of missing values; lets inspect this hunch visually.<jupyter_code>msno.bar(df_strings, sort='descending');<jupyter_output><empty_output><jupyter_text>Hunch confirmed: 10 the 17 columns are missing values with some being practically empty. To get a precise count, we can use `sidetable`.{% fn 2 %}<jupyter_code>import sidetable

df_strings.stb.missing(clip_0=True, style=True)<jupyter_output><empty_output><jupyter_text>OK, we have lots of missing values and several columns which appear to be measuring similar features (i.e., authors, illustrators, publishers, awards) so lets inspect these features in pairs.    `author` and `authors` Every book has an author, even if the author is "[Anonymous](https://bookshop.org/a/9791/9781538718469)," so  then why do we essentially have two columns for the same thing? :thinking: `author` is for books with a single writer whereas `authors` is for books with multiple authors like  [_Good Omens_](https://bookshop.org/a/9791/9780060853983).Let's test that theory.<jupyter_code>msno.matrix(df_strings.loc[:, ['author', 'authors']]);<jupyter_output><empty_output><jupyter_text>*Bazinga!* We have a perfect correlation between missing data for `author` and `authors` but lets' have a look just in case.<jupyter_code>df_strings.loc[df_strings['author'].isna() & df_strings["authors"].notna(), ['title', 'author', 'authors']].head()
df_strings.loc[df_strings['author'].notna() & df_strings["authors"].isna(), ['title', 'author', 'authors']].head()
df_strings.loc[df_strings['author'].notna() & df_strings["authors"].notna(), ['title', 'author', 'authors']].head()<jupyter_output><empty_output><jupyter_text>My curiosity is satiated.Now the question is how to successfully merge the two columns? We could replace the `NaN` in `author` with the:- values in `authors`- word `multiple`- first author in `authors`- more/most popular of the authors in `authors`and I'm sure I could come up with even more if I thought about/Googled it but the key is to understand that no matter what we choose, it will have consequences when we build our model{% fn 3 %}. Next question which comes to mind is::thinking: ***How many different authors are there?***<jupyter_code>df_strings.loc[:, 'author'].nunique()<jupyter_output><empty_output><jupyter_text>Wow! Nearly half our our observations contain a unique name meaning this feature has [high cardinality](https://www.kdnuggets.com/2016/08/include-high-cardinality-attributes-predictive-model.html).:thinking: ***Which authors are most represented in the data set?***  Lets create a [frequency table](https://www.mathsteacher.com.au/year8/ch17_stat/03_freq/freq.htm) to find out.<jupyter_code>author_counts = df_strings.loc[:, ["title", 'author']].groupby('author').count().reset_index()
author_counts.sort_values('title', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Given that I've scraped the data from a website focusing on children, teens, and young adults, the results above only make sense; authors like [<NAME>](https://bookshop.org/contributors/dr-seuss), [<NAME>](https://bookshop.org/contributors/eoin-colfer-20dba4fd-138e-477e-bca5-75b9fa9bfe2f), and [<NAME>](https://bookshop.org/books?keywords=lemony+snicket) are famous children's authors whereas [<NAME>](https://bookshop.org/books?keywords=percy+jackson), [<NAME>](https://bookshop.org/books?keywords=Walter+Dean+Myers) occupy the teen/young adult space and [<NAME>](https://bookshop.org/contributors/neil-gaiman) writes across ages. :thinking: ***How many authors are only represented once?*** That's easy to check.<jupyter_code>from matplotlib.ticker import FuncFormatter

ax = author_counts['title'].value_counts(normalize=True).nlargest(5).plot.barh()
ax.invert_yaxis();

#Set the x-axis to a percentage
ax.xaxis.set_major_formatter(FuncFormatter(lambda x, _: '{:.0%}'.format(x)))<jupyter_output><empty_output><jupyter_text>Wow! So approximately 60% of the authors have one title in our data set.  **Why does that matter?**When it comes time to build our model we'll need to either [label encode](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html), [one-hot encode](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html), or [hash](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html) this feature and whichever we decide to do will end up effecting the model profoundly due to the high [cardinality](https://pkghosh.wordpress.com/2017/10/09/combating-high-cardinality-features-in-supervised-machine-learning/) of this feature; however, we'll deal with all this another time :grin:.  `illustrator` and `illustrators`Missing values can be quite informative. :thinking: What types of books typically have illustrators?  :bulb: Children's books! Therefore, if a book's entries for both `illustrator` and `illustrators` is blank, that *probably* means that book doesn't have illustrations which would mean it is *more likely* to be for older children.Let's test this theory in the simplest way I can think of :smile:<jupyter_code>#Has an illustrator
df.loc[df['illustrator'].notna() | df['illustrators'].notna(), ['csm_rating']].hist();
#Doesn't have an illustrator
df.loc[df['illustrators'].isna() & df["illustrator"].isna(), ['csm_rating']].hist();<jupyter_output><empty_output><jupyter_text>:bulb: *Who* the illustrator is doesn't matter as much as *whether* there is an illustrator.Looks like when I do some feature engineering I'll need to create a `has_illustrator` feature.  `book_type` and `genre`These two features should be relatively straightforward but we'll have a quick look anyway. `book_type` should be easy because, after a cursory inspection using `head` above, I'd expect to only see 'fiction' or 'non-fiction' but I'll double check.<jupyter_code>ax_book_type = df_strings['book_type'].value_counts().plot.barh();
ax_book_type.invert_yaxis()<jupyter_output><empty_output><jupyter_text>Good! The only values I have are the ones I expected but the ratio is highly skewed. :thinking: What impact will this have on our model?`genre` (e.g. fantasy, romance, sci-fi) is a *far* broader topic than `booktype` but how many different genres are represented in the data set?<jupyter_code>df_strings['genre'].nunique()<jupyter_output><empty_output><jupyter_text>:roll_eyes: GreatWhat's the breakdown?<jupyter_code>ax_genre = df_strings['genre'].value_counts().plot.barh();
ax_genre.invert_yaxis()<jupyter_output><empty_output><jupyter_text>That's not super useful but what if I took 10 most common genres?<jupyter_code>ax_genre_10 = df_strings['genre'].value_counts(normalize=True).nlargest(10).plot.barh();
ax_genre_10.invert_yaxis()

#Set the x axis to percentage
ax_genre_10.xaxis.set_major_formatter(FuncFormatter(lambda x, _: '{:.0%}'.format(x)))<jupyter_output><empty_output><jupyter_text>Hmmm. Looks like approximately half the books fall into one of three genres. :bulb: To reduce dimensionality, recode any genre outside of the top 10 as 'other'.Will save that idea for the feature engineering stage.   `award` and `awards`Since certain awards (e.g. [The Caldecott Medal](https://cloviscenter.libguides.com/children/caldecott:~:text=The%20Medal%20shall%20be%20awarded,the%20illustrations%20be%20original%20work.)) are only awarded to children's books whereas others, namely [The RITA Award](https://en.wikipedia.org/wiki/RITA_AwardWinners) is only for "mature" readers.   :thinking: Will knowing if a work is an award winner provide insight?  :thinking: Which awards are represented?<jupyter_code>award_ax = df_strings['award'].value_counts().plot.barh()
award_ax.invert_yaxis();
awards_ax = df_strings['awards'].str.split(",").explode().str.strip().value_counts().plot.barh()
awards_ax.invert_yaxis()<jupyter_output><empty_output><jupyter_text>Hmmmmm. The Caldecott Medal is for picture books so that should mean the target readers are very young; however, we've already seen that "picture books" is the second most common value in `genre` so being a Caldecott Medal winner won't add much. Also, to be eligible for the other awards, a book needs to be aimed a t children 14 or below so that doesn't really tell us much either. Conclusion: drop this feature. While I could keep going and analyze `publisher`, `publishers`, and `available_on`, I'd be using the exact same techniques as above so, instead, time to move on to... Text `description`, `plot`, `csm_review`, `need_to_know`Now for some REALLY fun stuff! :thinking: How long are each of these observations? Trying to be as efficient as possible, I'll: - make a list of the features I want<jupyter_code>variables = ['description', 'plot', 'csm_review', 'need_to_know']<jupyter_output><empty_output><jupyter_text>- write a function to:    - convert the text to lowercase     - tokenize the text and remove [stop words](https://en.wikipedia.org/wiki/Stop_words)    - identify the length of each feature<jupyter_code>from nltk import word_tokenize
from nltk.corpus import stopwords

stop = stopwords.words('english')

def text_process(df, feature): 
    df.loc[:, feature+'_tokens'] = df.loc[:, feature].apply(str.lower)
    df.loc[:, feature+'_tokens'] = df.loc[:, feature+'_tokens'].apply(lambda x: [item for item in x.split() if item not in stop])
    df.loc[:, feature+'_len'] = df.loc[:, feature+'_tokens'].apply(len) 
    return df<jupyter_output><empty_output><jupyter_text>- loop through the list of variables saving it to the data frame<jupyter_code>for var in variables: 
    df_text = text_process(df_strings, var)
df_text.iloc[:, -8:].head()<jupyter_output><empty_output><jupyter_text>:thinking: `description` seems to be significantly shorter than the other three.  Let's plot them to investigate.<jupyter_code>len_columns  = df_text.columns.str.endswith('len')
df_text.loc[:,len_columns].hist();
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Yep - `description` is significantly shorter but how do the other three compare.<jupyter_code>columns = ['plot_len', 'need_to_know_len', 'csm_review_len']
df_text[columns].plot.box()
plt.xticks(rotation='vertical');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11200658862286017, 'fraction_numerical': 0.014589093476086829, 'mean_word_length': 4.8279053822420295, 'pattern_counts': {'":': 0, '<': 111, '<?xml version=': 0, '>': 112, 'https://': 23, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '11484', 'license': 'Apache-2.0', 'n_tokens_mistral': 5421, 'n_tokens_neox': 5074, 'n_words': 1743, 'path': '_notebooks/2020-07-06-e-is-for-eda.ipynb', 'repo_name': 'educatorsRlearners/an-a-z-of-machine-learning', 'size': '17037'}