<jupyter_start><jupyter_text>The Sower Requirements:- "data/interim/sower.csv"  EDAThe dataset is basically a times series. The data comes from different channels (Amazon, Apple, Barnes and Noble) and it is not normalized.<jupyter_code>import pandas as pd

import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns;
sns.set(rc={'figure.figsize':(10,8)})
sns.set(style="whitegrid")

from bokeh.io import show, output_notebook
from bokeh.palettes import Spectral

output_notebook()
df = pd.read_csv("../data/interim/sower.csv")
df.head()
df["todate"] = pd.to_datetime(df["todate"])<jupyter_output><empty_output><jupyter_text>Let's take a closer on what we've got,  Channels It looks like we have equivalent variables for different sellers/channels , e.g "soldunits", "amazonsoldunits" and "applesoldunits",<jupyter_code>CHANNELS = [
    "amazon",
    "apple",
    "barnesandnoble",
]<jupyter_output><empty_output><jupyter_text>Let'do totals, ignoring for now the CHANNELS. I manually check some examples, and it looks reasonable. If can always can back.<jupyter_code>df_plot = df.loc[:, ~df.columns.str.contains("|".join(CHANNELS), regex=True)] 

# variable with a channel
#df_channels = df.filter(regex="|".join(CHANNELS), axis=1)
#df_channels.head()

#import re
#normalize = lambda x: re.sub(r"|".join(CHANNELS), "", x)<jupyter_output><empty_output><jupyter_text>Let's unpivot<jupyter_code>df_plot = df_plot.melt(id_vars=["entity", "formattype", "todate"], var_name="variable", value_name="value")<jupyter_output><empty_output><jupyter_text>Plot Let's plot all data, so we will have a better picture,<jupyter_code>from bokeh.plotting import figure
from bokeh.layouts import column, row
from bokeh.models import ColumnDataSource, Select, TextInput, HoverTool

BOOKS = sorted(df_plot['entity'].unique())
TYPES = sorted(df_plot['formattype'].unique())
VARIABLES = sorted(df_plot['variable'].unique())
    
def update_data(*args, **kwargs):
    query = ' & '.join([f'{k}=="{v}"' for k, v in kwargs.items()])
    qs = df_plot.query(query)
    return dict(x=qs["todate"].values, y=qs["value"].values)
    
def modify_doc(doc):
    
    p = figure(title="Sower: Interactive Plot",
                x_axis_label="Date",
                y_axis_label="Value",
                x_axis_type='datetime', 
                width=1000)
    
    p.add_tools(HoverTool(
        tooltips=[
            ('(x, y)', "(@x{%F}, @y)"),],
        formatters={
            '@x': 'datetime'}))
    
    # data
    source = ColumnDataSource(
                data=update_data(entity="The Walking Dead", formattype="print", variable="soldunits"))
        
    p.scatter(x='x', y='y', source=source, color="purple", alpha=0.5)
    
    # filters
    select = Select(title="Book", value="The Walking Dead", options=BOOKS)
    select2 = Select(title="Type", value="print", options=TYPES)
    select3 = Select(title="Variable", value="soldunits", options=VARIABLES)

    def update_points(attrname, old, new):
        source.data = update_data(entity=select.value, formattype=select2.value, variable=select3.value)
    select.on_change('value', update_points)
    select2.on_change('value', update_points)
    select3.on_change('value', update_points)

    layout = column(row(select, select2, select3, width=800), row(p))

    doc.add_root(layout)

show(modify_doc)<jupyter_output><empty_output><jupyter_text>NotesThat is a starting point. Let's put some notes down,- **We should validate:**    - For instance, filtering `The Walking Dead > print > accruedpreordersoldunits`, we clearly see a growth in the number of preorders ahead of the data of publication (2017-08-29). That is a good indication that our aggregations hold true.        - However, we should validate the data quality and our assumptions with stakeholders    - **Most fields *are not* telling us a story:**         - We could not extract any information from most of the fields, either because sparsity or because the number does not describe what we are after.          - *Most fields *are not* independent:*    Just think of `soldunits` and `salesrevenue`. Both differ basically by a factor and they are describing the same underlying quantity.    Let's check,  The Walking Dead<jupyter_code>df = df[df["entity"] == "The Walking Dead"]
COLUMNS = [
    "soldunits",
    "salesrevenue",
    "amazonsoldunits",
    "totalunits",
    
]

wdf = df.set_index(["entity", "formattype", "todate"])[COLUMNS]
wdf.corr()
sns.pairplot(wdf)<jupyter_output><empty_output><jupyter_text>All those variables are highly correlated. Actually, they represent the *same information*<jupyter_code>all(df["soldunits"] == df["amazonsoldunits"] + df["applesoldunits"] + df["barnesandnoblesoldunits"])<jupyter_output><empty_output><jupyter_text>Note that we added `amazonsoldunits` in the comparison. In matter of fact, Amazon represents more than 90% of sales,<jupyter_code>f'{100*(df["amazonsoldunits"]/df["soldunits"]).mean():.02f}% of sold units from Amazon'<jupyter_output><empty_output><jupyter_text>So that brings us down to 2 variables basically,- `soldunits`- `subscriptionsunit`<jupyter_code>all(df["totalunits"] == df["soldunits"] + df["subscriptionunits"])<jupyter_output><empty_output><jupyter_text>Let's the distribution for those variables,<jupyter_code>sns.boxplot(x="formattype", y="soldunits", data=df, showfliers=False)<jupyter_output><empty_output><jupyter_text>Looking at the distributions, a bad day for print still has a strong position compared to audio and ebook,<jupyter_code>df[["formattype", "soldunits"]].describe(include="all").T
sns.boxplot(x="formattype", y="subscriptionunits", data=df, showfliers=False)<jupyter_output><empty_output><jupyter_text>All subscriptions exclusively for audio,   Let's take a look at different formats,<jupyter_code>wdf = df.pivot_table("totalunits", ["todate"], "formattype")
p = figure(title="Sower: Which format is the most sold? (The Walking Dead)", 
            x_axis_label="Date",
            y_axis_label="Units",
            x_axis_type='datetime')

p.varea_stack(stackers=list(wdf.columns), x="todate", color=Spectral[3], legend_label=list(wdf.columns), source=wdf.reset_index())

# reverse the legend entries to match the stacked order
p.legend.items.reverse()

show(p)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13473149030823006, 'fraction_numerical': 0.0060374960279631395, 'mean_word_length': 5.208086785009862, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '330561', 'license': 'MIT', 'n_tokens_mistral': 2113, 'n_tokens_neox': 2024, 'n_words': 505, 'path': 'notebooks/02-eda.ipynb', 'repo_name': 'g4brielvs/sower', 'size': '6294'}