<jupyter_start><jupyter_text>Solving the Mystery of Chai Time Data Science    A Data Science podcast series by <NAME>------    Mr. RsTaK, Where are we?    Hello my dear Kagglers. As you all know I love Kaggle and its community. I spend most of my time surfing my Kaggle feed, scrolling over discussion forms, appreciating the efforts put on by various Kagglers via their unique / interesting way of Storytelling. So, this morning when i was following my usual routine in Kaggle, I came across this Dataset named [Chai Time Data Science | CTDS.Show](https://www.kaggle.com/rohanrao/chai-time-data-science) provided by Mr. [Vopani](https://www.kaggle.com/rohanrao) and Mr. [<NAME>](https://www.kaggle.com/init27). At first glance, I was like what's this? How they know I'm having a tea break? Oh no buddy! I was wrong. It's [CTDS.Show](https://chaitimedatascience.com) :)---    Chai Time Data Science (CTDS.Show)    Chai Time Data Science show is a [Podcast](https://anchor.fm/chaitimedatascience) + [Video](https://www.youtube.com/playlist?list=PLLvvXm0q8zUbiNdoIazGzlENMXvZ9bd3x) + [Blog](https://sanyambhutani.com/tag/chaitimedatascience/) based show for interviews with Practitioners, Kagglers & Researchers and all things Data Science.[CTDS.Show](https://chaitimedatascience.com), driven by the community under the supervision of Mr. [<NAME>](https://www.kaggle.com/init27) gonna  complete its 1 year anniversary on **21st June, 2020** and to celebrate this achievement they decided to run a **Kaggle contest** around the dataset with all of the **75+ ML Hero interviews** on the series.According to our Host, The competition is aimed at articulating insights from the Interviews with ML Heroes. Provided a dataset consists of detailed Stats, Transcripts of CTDS.Show, the goal is to use these and come up with interesting insights or stories based on the 100+ interviews with ML Heroes.We have our Dataset containing :* **Description.csv** : This file consists of the **descriptions texts** from YouTube and Audio* **Episodes.csv** : This file contains the **statistics of all the Episodes** of the Chai Time Data Science show.* **YouTube Thumbnail Types.csv** : This file consists of the **description of Anchor/Audio thumbnail of the episodes*** **Anchor Thumbnail Types.csv** : This file contains the **statistics of the Anchor/Audio thumbnail*** **Raw Subtitles** : Directory containing **74 text files** having raw subtitles of all the episodes* **Cleaned Subtitles** : Directory containing cleaned subtitles (in CSV format)      Hmm.. Seems we have some stories to talk about..Congratulating [CTDS.Show](https://chaitimedatascience.com) for their **1 year anniversary**, Let's get it started :)---**Btw This gonna be a long kernel. So, Hey! Looking for a guide :) ?**&emsp;&emsp;0. Importing Necessary Libraries&emsp;&emsp;1. A Closer look to our Dataset&emsp;&emsp;&emsp;&emsp;1.1. Exploring YouTube Thumbnail Types.csv&emsp;&emsp;&emsp;&emsp;1.2. Exploring Anchor Thumbnail Types.csv&emsp;&emsp;&emsp;&emsp;1.3. Exploring Description.csv&emsp;&emsp;&emsp;&emsp;1.4. Exploring Episodes.csv&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.1. Missing Values ?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.2. M0-M8 Episodes&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.3. Solving the Mystery of Missing Values&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.4. Is it a Gender Biased Show?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.5. Time for a Chai Break&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.6. How to get More Audience?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.7. Youtube Favors CTDS?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.8. Do Thumbnails really matter?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.9. How much Viewers wanna watch?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.10. Performance on Other Platforms&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.11. Distribution of Heores by Country and Nationality&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.12. Any Relation between Release Date of Epiosdes?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.4.13. Do I know about Release of anniversary interview episode?&emsp;&emsp;&emsp;&emsp;1.5. Exploring Raw / Cleaned Substitles&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.5.1. A Small Shoutout to <NAME>&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.5.2. Intro is Bad for CTDS ?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.5.3. Who Speaks More ?&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.5.4. Frequency of Questions Per Episode &emsp;&emsp;&emsp;&emsp;&emsp;&emsp;1.5.5. Favourite Text ? &emsp;&emsp;2. End Notes&emsp;&emsp;3. Credits   Note :  Sometimes, Plotly Graphs fails to render with the Kernel. Please restart the page in that case Importing Necessary LibrariesGo back to our Guide<jupyter_code>import os

import warnings
warnings.simplefilter("ignore")
warnings.filterwarnings("ignore", category=DeprecationWarning) 

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import missingno as msno

import plotly.express as px

import plotly.graph_objects as go
from plotly.subplots import make_subplots

!pip install pywaffle
from pywaffle import Waffle

from bokeh.layouts import column, row
from bokeh.models.tools import HoverTool
from bokeh.models import ColumnDataSource, Whisker
from bokeh.plotting import figure, output_notebook, show

output_notebook()

from IPython.display import IFrame

pd.set_option('display.max_columns', None)<jupyter_output>Collecting pywaffle
  Downloading pywaffle-0.5.1-py2.py3-none-any.whl (525 kB)
[K     |████████████████████████████████| 525 kB 5.1 MB/s 
[?25hRequirement already satisfied: matplotlib in /opt/conda/lib/python3.7/site-packages (from pywaffle) (3.2.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->pywaffle) (2.4.7)
Requirement already satisfied: numpy>=1.11 in /opt/conda/lib/python3.7/site-packages (from matplotlib->pywaffle) (1.18.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib->pywaffle) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->pywaffle) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->pywaffle) (1.2.0)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from cyc[...]<jupyter_text>A Closer look to our DirectoriesGo back to our GuideLet's dive into each and every aspect of our Dataset step by step in order to get every inches out of it... Exploring YouTube Thumbnail Types.csvGo back to our GuideAs per our knowledge, This file consists of the description of Anchor/Audio thumbnail of the episodes. Let's explore more about it...<jupyter_code>YouTube_df=pd.read_csv("../input/chai-time-data-science/YouTube Thumbnail Types.csv")
print("No of Datapoints : {}\nNo of Features : {}".format(YouTube_df.shape[0], YouTube_df.shape[1]))
YouTube_df.head()<jupyter_output>No of Datapoints : 4
No of Features : 6<jupyter_text>So, Basically [CTDS](https://chaitimedatascience.com) uses **4 types of Thumbnails** in their Youtube videos. Its 2020 and people still uses YouTube default image as thumbnail ! Hmm... a Smart decision or just a blind arrow, We'll figure it out in our futher analysis ... Exploring Anchor Thumbnail Types.csvGo back to our GuideSo, This file contains the statistics of the Anchor/Audio thumbnail<jupyter_code>Anchor_df=pd.read_csv("../input/chai-time-data-science/Anchor Thumbnail Types.csv")
print("No of Datapoints : {}\nNo of Features : {}".format(Anchor_df.shape[0], Anchor_df.shape[1]))
Anchor_df.head()<jupyter_output>No of Datapoints : 4
No of Features : 6<jupyter_text>It's just similar to Youtube Thumbnail Types. If you are wondering What's anchor then it's a free platform for podcast creation<jupyter_code>IFrame('https://anchor.fm/chaitimedatascience', width=800, height=450)<jupyter_output><empty_output><jupyter_text>Exploring Description.csvGo back to our GuideThis file consists of the descriptions texts from YouTube and Audio<jupyter_code>Des_df=pd.read_csv("../input/chai-time-data-science/Description.csv")
print("No of Datapoints : {}\nNo of Features : {}".format(Des_df.shape[0], Des_df.shape[1]))
Des_df.head()<jupyter_output>No of Datapoints : 85
No of Features : 2<jupyter_text>So, We have description for every episode. Let's have a close look what we have here<jupyter_code>def show_description(specific_id=None, top_des=None):
    
    if specific_id is not None:
        print(Des_df[Des_df.episode_id==specific_id].description.tolist()[0])
        
    if top_des is not None:
        for each_des in range(top_des):  
            print(Des_df.description.tolist()[each_des])
            print("-"*100)<jupyter_output><empty_output><jupyter_text>⚒️ About the Function : In order to explore our Descriptions, I just wrote a small script. It has two options:* Either You provide specific episode id(specific_id) to have a look at that particular description* Or you can provide a number(top_des) and this script will display description for top "x" numbers that you provided in top_des<jupyter_code>show_description("E1")
show_description(top_des=3)<jupyter_output>Interview with ML Hero Series: https://medium.com/p/bfaaf38df219

http://chaitimedatascience.com/
A show for Interviews with Practitioners, Kagglers & Researchers and all things Data Science hosted by <NAME>. 

You can expect weekly episodes every Sunday, Thursday available as Video, Podcast, and blogpost.

Intro track: 
Flow by LiQWYD https://soundcloud.com/liqwyd
----------------------------------------------------------------------------------------------------
In the first Episode, Sanyam Bhutani interviews Kaggle Triple Grandmaster: <NAME>. They talk about Abhishek's journey into Data Science and Kaggle; his Kaggle Experience and current projects. 

Interview with Machine Learning Hero Series: https://medium.com/dsnet/interviews-with-machine-learning-heroes-ad9358385278 

Follow:
<NAME>: https://www.kaggle.com/abhishek
https://www.linkedin.com/in/abhisvnit/
https://twitter.com/abhi1thakur

<NAME>: https://twitter.com/bhutanisanyam1

About:
http://[...]<jupyter_text>Advice : Feel free to play with the function "show_description()" to have a look over various descriptions provided in a go🧠 My Cessation:    * Although I went through some description and realized it just contains **URL**, **Necessary links** for social media sites with a **little description** of the current show and some **announcements regarding future releases*** I'm **not gonna put stress** in this area because **I don't think there's much to scrap in them**. Right now, let's move ahead.      Exploring Episodes.csvGo back to our GuideThis file contains the statistics of all the Episodes of the Chai Time Data Science show.Okay ! So, it's the big boy itself ..<jupyter_code>Episode_df=pd.read_csv("../input/chai-time-data-science/Episodes.csv")
print("No of Datapoints : {}\nNo of Features : {}".format(Episode_df.shape[0], Episode_df.shape[1]))
Episode_df.head()<jupyter_output>No of Datapoints : 85
No of Features : 36<jupyter_text>Wew ! That's a lot of features. I'm sure We'll gonna explore some interesting insights from this Metadata If you reached till here, then please bare me for couple of minutes more.. Now, We'll gonna have a big sip of our "Chai" :)  Missing ValuesBefore diving into our analysis, Let's have check  for Missing Values in our CSV..For this purpose, I'm gonna use this library named [missingno](https://github.com/ResidentMario/missingno).Just use this line :>import missingno as msno[missingno](https://github.com/ResidentMario/missingno) helps us to deal with missing values in a dataset with the help of visualisations. With over 2k stars on github, this library is already very popular.<jupyter_code>msno.matrix(Episode_df)<jupyter_output><empty_output><jupyter_text>Aah shit ! Here We go again..📌 Observations :* We can clearly see that **heroes_kaggle_username**, **heroes_twitter_handle** have **lots of missing values*** We can observe **bunch of data missing** from column name **heroes to heroes_twitter_handle** in a **continous way(that big block region)** which shows a **specific reason** of data missing at those points* **Few datapoints** are too **missing** in **anchor**, **spotify** and **apple** section i.e. missing data in **podcasts**There is also a chart on the right side of plot.It summarizes the general shape of the data completeness and points out the rows with the maximum and minimum nullity in the dataset.Well, Before giving any False Proclaim Let's explore more about it..<jupyter_code>temp=Episode_df.isnull().sum().reset_index().rename(columns={"index": "Name", 0: "Count"})
temp=temp[temp.Count!=0]

Source=ColumnDataSource(temp)

tooltips = [
    
    ("Feature Name", "@Name"),
    ("No of Missing entites", "@Count")
]

fig1 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400,tooltips=tooltips, x_range = temp["Name"].values, title = "Count of Missing Values")
fig1.vbar("Name", top = "Count", source = Source, width = 0.4, color = "#76b4bd", alpha=.8)

fig1.xaxis.major_label_orientation = np.pi / 8
fig1.xaxis.axis_label = "Features"
fig1.yaxis.axis_label = "Count"

fig1.grid.grid_line_color="#feffff"


show(fig1)<jupyter_output><empty_output><jupyter_text>📌 Observations :* Columns from **heroes** to **heroes_nationality** has same about of missing data. Seems We can find a **reasonable relation** between them* About **45.88% (85-39)** and **22.35% (85-19)** of Data missing in column name **heroes_kaggle_username** and **heroes_twitter_handle** respectively* We have just **1 missing value** in **anchor** and **spotify**, **2 missing values** in **apple section** that is quite easy to handle🧠 My Cessation: * Come-on. I don't understand. Chai Time Data Science show is about interviews with our Heroes. So, How do we have **11 missing values in Feature "heroes"**Let's find out..<jupyter_code>Episode_df[Episode_df.heroes.isnull()]<jupyter_output><empty_output><jupyter_text>💭 Interesting..* episode_id "E0" was all about Chai Time Data Science Launch Announcement* episode_id "E69" was Birthday SpecialIt make sense why there's no hero for the following episodesBut What are these M0-M8 episodes .. ? M0-M8 Episodes* Looking around for a while, I realized M0-M8 was a small mini-series based on fast.ai summaries and the Things Jeremy says to do that were released on same date.🧠 My Cessation: * Well for the sake of our analysis, I'll treat them as outlier for the current CSV and will analyise them seperately. So, I'm gonna remove them from this CSV, storing seperately for later analysis<jupyter_code>temp=[id for id in Episode_df.episode_id if id.startswith('M')]
fastai_df=Episode_df[Episode_df.episode_id.isin(temp)]
Episode_df=Episode_df[~Episode_df.episode_id.isin(temp)]<jupyter_output><empty_output><jupyter_text>Also, ignoring "E0" and "E69" for right now ...<jupyter_code>dummy_df=Episode_df[(Episode_df.episode_id!="E0") & (Episode_df.episode_id!="E69")]
msno.matrix(dummy_df)
temp=dummy_df.isnull().sum().reset_index().rename(columns={"index": "Name", 0: "Count"})
temp=temp[temp.Count!=0]

Source=ColumnDataSource(temp)

tooltips = [
    ("Feature Name", "@Name"),
    ("No of Missing entites", "@Count")
]

fig1 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400,tooltips=tooltips, x_range = temp["Name"].values, title = "Count of Missing Values")
fig1.vbar("Name", top = "Count", source = Source, width = 0.4, color = "#76b4bd", alpha=.8)

fig1.xaxis.major_label_orientation = np.pi / 4
fig1.xaxis.axis_label = "Features"
fig1.yaxis.axis_label = "Count"

fig1.grid.grid_line_color="#feffff"

show(fig1)<jupyter_output><empty_output><jupyter_text>Now, That's much better.. But We still have a lots of Missing Values Solving the Mystery of Missing Values<jupyter_code>parent=[]
names =[]
values=[]
temp=dummy_df.groupby(["category"]).heroes_gender.value_counts()
for k in temp.index:
    parent.append(k[0])
    names.append(k[1])
    values.append(temp.loc[k])

df1 = pd.DataFrame(
    dict(names=names, parents=parent,values=values))


parent=[]
names =[]
values=[]
temp=dummy_df.groupby(["category","heroes_gender"]).heroes_kaggle_username.count()
for k in temp.index:
    parent.append(k[0])
    names.append(k[1])
    values.append(temp.loc[k])

df2 = pd.DataFrame(
    dict(names=names, parents=parent,values=values))


fig = px.sunburst(df1, path=['names', 'parents'], values='values', color='parents',hover_data=["names"], title="Heroes associated with Categories")
fig.update_traces( 
                 textinfo='percent entry+label',
                 hovertemplate = "Industry:%{label}: <br>Count: %{value}"
                )
fig.show()


fig = px.sunburst(df2, path=['names', 'parents'], values='values', color='parents', title="Heroes associated with Categories having Kaggle Account")
fig.update_traces( 
                 textinfo='percent entry+label',
                 hovertemplate = "Industry:%{label}: <br>Count: %{value}"
                )
fig.show()<jupyter_output><empty_output><jupyter_text>📌 Observations :    * Heores associated with "Category" **Kaggle** are expected to have a **Kaggle account*** Ignoring the counts from "Category" Kaggle (74-31=43), **Out of 43 only 15 Heroes have Kaggle account**.* This **explains all Missing 28 Values** from our CSV* Similarly We have **8 Heroes who don't have twitter handle**. It's okay. Even I don't have a twitter handle :DWanna know a fun fact ?> Because of this Kaggle platform, Now I've aprox **42%** chance of becoming a CTDS Hero :) ...     Ahem Ahem... Focus [RsTaK](https://www.kaggle.com/rahulgulia) Focus.. Let's get back to our work.    Wait? Guess I missed something.. What's that gender ratio? Is it a Gender Biased Show?<jupyter_code>gender = Episode_df.heroes_gender.value_counts()

fig = plt.figure(
    FigureClass=Waffle, 
    rows=5,
    columns=12,
    values=gender,
    colors = ('#20639B', '#ED553B'),
    title={'label': 'Gender Distribution', 'loc': 'left'},
    labels=["{}({})".format(a, b) for a, b in zip(gender.index, gender) ],
    legend={'loc': 'lower left', 'bbox_to_anchor': (0, -0.4), 'ncol': len(Episode_df), 'framealpha': 0},
    font_size=30, 
    icons = 'child',
    figsize=(12, 5),  
    icon_legend=True
)<jupyter_output><empty_output><jupyter_text>Jokes apart, We can't give any strong statement over this. But yea, I'm hoping for more Female Heroes :D🧠 My Cessation: I won't talk much about relation of gender with other features because :* Gender feature is highly biased towards one categorySo, We can not conclude any relation with other features. * If we had a good gender ratio, then we could have talked about impact of gender  Even if we somehow observe any positive conclusion for Female gender then I would say it will be just a coincidence. There are other factors apart from gender that may have resulted in positive conclusion for Female gender.With such a biased and small sample size for Female, We can not comment any strong statement on that<jupyter_code>dummy_df[dummy_df.apple_listeners.isnull()]<jupyter_output><empty_output><jupyter_text>📌 Observations :    Following our analysis, We realized :* CTDS had an episode with **Tuatini Godard, episode_id : "E12" exclusively for Youtube**. Although it was an **Audio Only video**(if it makes sense :D ) released on Youtube* If it was an Audio Only Version, then **Why it wasn't released on other platforms** ? Hmmm... interesting. Well, I think Mr. [<NAME>](https://www.kaggle.com/init27) can answer this well. * Similarly, CTDS had an **episode with Rachel Thomas released at every platform expect for Apple**    With this, We have solved all the mysteries related to the Missing Data. Now we can finally explore other aspects of this CSV  But before that.. Time for a Chai Break While having a sip of my Chai (Tea), I'm just curious **Why this show is named "Chai Time Data Science"?** Well, I **don't have a solid answer** for this but maybe its just **because our Host loves Chai?** Hmmm.. So You wanna say our Host is more Hardcore Chai Lover than me?Hey ! Hold my Chai..<jupyter_code>fig = go.Figure([go.Pie(labels=Episode_df.flavour_of_tea.value_counts().index.to_list(),values=Episode_df.flavour_of_tea.value_counts().values,hovertemplate = '<br>Type: %{label}</br>Count: %{value}<br>Popularity: %{percent}</br>', name = '')])
fig.update_layout(title_text="What Host drinks everytime ?", template="plotly_white", title_x=0.45, title_y = 1)
fig.data[0].marker.line.color = 'rgb(255, 255, 255)'
fig.data[0].marker.line.width = 2
fig.update_traces(hole=.4,)
fig.show()<jupyter_output><empty_output><jupyter_text>📌 Observations :* **Masala Chai (count=16)** and **Ginger Chai (count=16)** seems to be **favourite Chai** of our Host **followed by Herbal Tea (count=11) and Sulemani Chai (count=11)*** Also, Our Host seems to be **quite experimental with Chai**. He has varities of flavour in his bellyOh Man ! This time you win. You're a real Chai loverNow, One Question arises..❓So, Host drinking any specific Chai at specific time have any relation with other factors or success of CTDS?🧠 My Cessation: * Thinking practically, **I don't think** drinking Chai at any specific time can have **any real impact** for the show.* Believing on such things is an **example of superstition**. * No doubt, as per the data it **may have some relation** with other factors. But to support any statement here, I would like to quote a **famous sentence used in statistics** i.e.       Correlation does not imply Causation    How to get More Audience?    Well, rewarding for your victory in that Chai Lover Challenge, I'll try to assist CTDS on how to get more Audience 😄 * Ofcourse CTDS.Shows are a kind of gem, fully informative, covering interviews with some successfull people* But talking Statistically here, We'll gonna define Success of an Episode by amount of Audience it gathered<jupyter_code>temp=dummy_df.isnull().sum().reset_index().rename(columns={"index": "Name", 0: "Count"})
temp=temp[temp.Count!=0]

Source=ColumnDataSource(temp)

tooltips = [
    ("Feature Name", "@Name"),
    ("No of Missing entites", "@Count")
]

fig1 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400,tooltips=tooltips, x_range = temp["Name"].values, title = "Count of Missing Values")
fig1.vbar("Name", top = "Count", source = Source, width = 0.4, color = "#76b4bd", alpha=.8)

fig1.xaxis.major_label_orientation = np.pi / 4
fig1.xaxis.axis_label = "Features"
fig1.yaxis.axis_label = "Count"

fig1.grid.grid_line_color="#feffff"

show(fig1)
Episode_df.release_date = pd.to_datetime(Episode_df.release_date)
Source = ColumnDataSource(Episode_df)
fastai_df.release_date = pd.to_datetime(fastai_df.release_date)
Source2 = ColumnDataSource(fastai_df)

tooltips = [
    ("Episode Id", "@episode_id"),
    ("Episode Title", "@episode_name"),
    ("Hero Present", "@heroes"),
    ("CTR", "@youtube_ctr"),
    ("Category", "@category"),
    ("Date", "@release_date{%F}"),
    ]

tooltips2 = [
    ("Episode Id", "@episode_id"),
    ("Episode Title", "@episode_name"),
    ("Hero Present", "@heroes"),
    ("Subscriber Gain", "@youtube_subscribers"),
    ("Category", "@category"),
    ("Date", "@release_date{%F}"),
    ]


fig1 = figure(background_fill_color="#ebf4f6",plot_width = 600, plot_height = 400, x_axis_type = "datetime", title = "CTR Per Episode")
fig1.line("release_date", "youtube_ctr", source = Source, color = "#03c2fc", alpha = 0.8, legend_label="youtube_ctr")
fig1.varea(source=Source, x="release_date", y1=0, y2="youtube_ctr", alpha=0.2, fill_color='#55FF88', legend_label="youtube_ctr")
fig1.line("release_date", Episode_df.youtube_ctr.mean(), source = Source, color = "#f2a652", alpha = 0.8,line_dash="dashed", legend_label="Youtube CTR Mean : {:.3f}".format(Episode_df.youtube_ctr.mean()))
fig1.circle(x="release_date", y="youtube_ctr", source = Source2, color = "#5bab37", alpha = 0.8, legend_label="M0-M8 Series")

fig1.add_tools(HoverTool(tooltips=tooltips,formatters={"@release_date": "datetime"}))
fig1.xaxis.axis_label = "Release Date"
fig1.yaxis.axis_label = "Click Per Impression"

fig1.grid.grid_line_color="#feffff"

fig2 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400, x_axis_type = "datetime", title = "Subscriber Gain Per Episode")
fig2.line("release_date", "youtube_subscribers", source = Source, color = "#03c2fc", alpha = 0.8, legend_label="Subscribers")
fig2.varea(source=Source, x="release_date", y1=0, y2="youtube_subscribers", alpha=0.2, fill_color='#55FF88', legend_label="Subscribers")
fig2.circle(x="release_date", y="youtube_subscribers", source = Source2, color = "#5bab37", alpha = 0.8, legend_label="M0-M8 Series")


fig2.add_tools(HoverTool(tooltips=tooltips2,formatters={"@release_date": "datetime"}))
fig2.xaxis.axis_label = "Release Date"
fig2.yaxis.axis_label = "Subscriber Count"

fig2.grid.grid_line_color="#feffff"

show(column(fig1, fig2))<jupyter_output><empty_output><jupyter_text>📌 Observations :From the Graphs, We can see :* On an average, CTDS episodes had **2.702 CTR*** **38 out of 76 (50% exact)** have CTRs above the average* Episode **E1, E27 and E49 seems to be lucky** for CTDS in terms of **Subscriber Count*** Episode E19 had the **best CTR (8.460)** which is self explanatory from the Episode Title. Everyone loves to know about MOOC(s) and ML Interviews in Industries 🤖 M0-M8 Series :* Despite related to Fast.ai, M0-M7 **doesn't perform** that well as compared to other vides related to fast.ai* **M0 and M1 though received a good amount of CTR** but other M Series quite below the average CTR* **M0 and M1 has better impact on subscriber gain** as compared to other M series but overall series doesn't perform well on Subscriber gain* All M0-M8 series were released on the **same day**, which can be the reason for this incident. M0-M1 despite having good CTR **fails to hold the viewers interest on M series**  💭 Interestingly.., * Episode **E19 despite of having best CTR** till now (8.460), **didn't contributed much** in Subscriber Count (only 7 Subscriber Gained) But Why ❓* CTR **doen't mean that person likes the content**, or **he/she will be watching that video** for long or will be **subscribing to the channel*** Maybe that video was **recommended on his newsfeed and he/she clicked on it** just to check the video* Maybe he/she **doesn't liked the content** * Maybe he/she **accidently clicked** on the videoThere's an huge possibility of such cases. But in conclusion, We can say high CTR reflect cases like :* People clicked on the video maybe because of the **Thumbnail, or Title was soothing**. Maybe he/she clicked because of the **hero mentioned in Title/Thumbnail**📃 I don't know how Youtube algorithm works. But for the sake of answering exceptional case of E19, My hypothetical answer will be:* Title contains the word **"MOOC"**. Since now a days everyone wanna **break into DataScience**, Youtube algorithm may have **suggested **that video to people looking for MOOCs* Most of other episodes have similar kind of Titles stating **"Interview with"** or have some terms that **are't that begineer friendly**. Resulting in **low CTR** * Supporting my hypothesis, observe **E27** (having fast.ai in Title that is a famous MOOC), **E38**(Title with Youngest Grandmaster may have attracted people to click),**E49** (Getting started in Datascience), **E60**(Terms like NLP and Open Source Projects) and **E75**(again fast.ai)* You can **argue for E12** which has the word "Freelancing" in the Title. Well **exceptions** will be thereOkay What's about organic reach of channel or reach via Heroes?<jupyter_code>Source = ColumnDataSource(Episode_df)
Source2 = ColumnDataSource(fastai_df)

tooltips = [
    ("Episode Id", "@episode_id"),
    ("Hero Present", "@heroes"),
    ("Impression Views", "@youtube_impression_views"),
    ("Non Impression Views", "@youtube_nonimpression_views"),
    ("Category", "@category"),
    ("Date", "@release_date{%F}"),
    ]

tooltips2 = [
    ("Episode Id", "@episode_id"),
    ("Hero Present", "@heroes"),
    ("Subscriber Gain", "@youtube_subscribers"),
    ("Category", "@category"),
    ("Date", "@release_date{%F}"),
    ]


fig1 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400, x_axis_type = "datetime", title = "Impression-Non Impression Views Per Episode")
fig1.line("release_date", "youtube_impression_views", source = Source, color = "#03c2fc", alpha = 0.8, legend_label="Impression Views")
fig1.line("release_date", "youtube_nonimpression_views", source = Source, color = "#f2a652", alpha = 0.8, legend_label="Non Impression Views")
fig1.varea(source=Source, x="release_date", y1=0, y2="youtube_impression_views", alpha=0.2, fill_color='#55FF88', legend_label="Impression Views")
fig1.varea(source=Source, x="release_date", y1=0, y2="youtube_nonimpression_views", alpha=0.2, fill_color='#e09d53', legend_label="Non Impression Views")
fig1.circle(x="release_date", y="youtube_impression_views", source = Source2, color = "#5bab37", alpha = 0.8, legend_label="M0-M8 Series Impression Views")
fig1.circle(x="release_date", y="youtube_nonimpression_views", source = Source2, color = "#2d3328", alpha = 0.8, legend_label="M0-M8 Series Non Impression Views")



fig1.add_tools(HoverTool(tooltips=tooltips,formatters={"@release_date": "datetime"}))
fig1.xaxis.axis_label = "Release Date"
fig1.yaxis.axis_label = "Total Views"

fig2 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400, x_axis_type = "datetime", title = "Subscriber Gain Per Episode")
fig2.line("release_date", "youtube_subscribers", source = Source, color = "#03c2fc", alpha = 0.8, legend_label="Subscribers")
fig2.varea(source=Source, x="release_date", y1=0, y2="youtube_subscribers", alpha=0.2, fill_color='#55FF88', legend_label="Subscribers")
fig2.circle(x="release_date", y="youtube_subscribers", source = Source2, color = "#5bab37", alpha = 0.8, legend_label="M0-M8 Series")


fig2.add_tools(HoverTool(tooltips=tooltips2,formatters={"@release_date": "datetime"}))
fig2.xaxis.axis_label = "Release Date"
fig2.yaxis.axis_label = "Subscriber Count"


show(column(fig1, fig2))<jupyter_output><empty_output><jupyter_text>📌 Observations :* Mostly Non Impression Views are **greater than Impression views**. CTDS seems to have a **loyal fan base** that's sharing his videos, producing more Non Impression Views* In some cases, there's sharp increase in views and **big difference** between Impression and Non Impression Views. * People **love to see specific Heroes**. Hero choice do matter* Total Views(especially Non Impression Views) definately plays role in Subscriber Gain* Though M series doesn't have good performance, but if you watch carefully, you'll realise **M series has better Impression Views than Non Impression Views** Youtube Favors CTDS?<jupyter_code>data1={
      "Youtube Impressions":Episode_df.youtube_impressions.sum(), 
      "Youtube Impression Views": Episode_df.youtube_impression_views.sum(), 
      "Youtube NonImpression Views" : Episode_df.youtube_nonimpression_views.sum()
     }

text=("Youtube Impressions","Youtube Impression Views","Youtube NonImpression Views")

fig = go.Figure(go.Funnelarea(
    textinfo= "text+value",
    text =list(data1.keys()),
    values = list(data1.values()),
    title = {"position": "top center", "text": "Youtube and Views"},
      name = '', showlegend=False,customdata=['Video Thumbnail shown to Someone', 'Views From Youtube Impressions', 'Views without Youtube Impressions'], hovertemplate = '%{customdata} <br>Count: %{value}</br>'
  ))
fig.show()<jupyter_output><empty_output><jupyter_text>📌 Observations :Few things to note here :    * Well, I havn't cracked Youtube Algorithm, but it seems Youtube has its blessing over CTDS* CTDS Episodes is only able to convert **2.84%** of **Youtube Impressions into Viewers*** **65.12%** of CTDS views are **Non Impression views**    Seems It's clear Youtube Thumbnail, Video Title are the important factor for deciding whether a person will click on the video or not. Wait, You want some figures? Do Thumbnails really matter ?<jupyter_code>colors = ["red", "olive", "darkred", "goldenrod"]

index={
    0:"YouTube default image",
    1:"YouTube default image with custom annotation",
    2:"Mini Series: Custom Image with annotations",
    3:"Custom image with CTDS branding, Title and Tags"
}

p = figure(background_fill_color="#ebf4f6", plot_width=600, plot_height=300, title="Thumbnail Type VS CTR")

base, lower, upper = [], [], []

for each_thumbnail_ref in index:
    if each_thumbnail_ref==2:
        temp = fastai_df[fastai_df.youtube_thumbnail_type==each_thumbnail_ref].youtube_ctr 
    else:
        temp = Episode_df[Episode_df.youtube_thumbnail_type==each_thumbnail_ref].youtube_ctr
    mpgs_mean = temp.mean()
    mpgs_std = temp.std()
    lower.append(mpgs_mean - mpgs_std)
    upper.append(mpgs_mean + mpgs_std)
    base.append(each_thumbnail_ref)

    source_error = ColumnDataSource(data=dict(base=base, lower=lower, upper=upper))
    p.add_layout(
        Whisker(source=source_error, base="base", lower="lower", upper="upper")
    )

    tooltips = [
        ("Episode Id", "@episode_id"),
        ("Hero Present", "@heroes"),
        ]

    color = colors[each_thumbnail_ref % len(colors)]
    p.circle(y=temp, x=each_thumbnail_ref, color=color, legend_label=index[each_thumbnail_ref])
    print("Mean CTR for Thumbnail Type {} : {:.3f} ".format(index[each_thumbnail_ref], temp.mean()))
show(p)<jupyter_output>Mean CTR for Thumbnail Type YouTube default image : 2.565 
Mean CTR for Thumbnail Type YouTube default image with custom annotation : 2.725 
Mean CTR for Thumbnail Type Mini Series: Custom Image with annotations : 1.969 
Mean CTR for Thumbnail Type Custom image with CTDS branding, Title and Tags : 3.115<jupyter_text>📌 Observations :From above Box-Plot : * It seems Type of Thumbnail do have **some impact on CTR*** Despite of using **YouTube default image for maximum of time**, it's average CTR is **lowest** as compared to CTR from other Youtube Thumbnail* Since **Count of other YouTube thumbnails are less**, We can't say What's the best Thumbnail* CTR depends on other factors too like Title, Hero featured in the Episode etc. Still we can **confidently** say that **Thumbnails other than YouTube default image attracts** more Users to click on the Video * As We talked about M series, M0-M1 failed to keep interest of people in the series. * Although their Mean CTR is lowest yet we can observe M0-M1 has a **better CTR as compared to majority of Episodes with Youtube default thumbnails**. In short, Don't use **Default Youtube Image for Thumbnail** How much Viewers wanna watch?Episodes have different duration.In order to get a significant insight, I'll calculate the **percentage of each Episode watched**..<jupyter_code>a=Episode_df[["episode_id", "episode_duration", "youtube_avg_watch_duration"]]
a["percentage"]=(a.youtube_avg_watch_duration/a.episode_duration)*100

b=fastai_df[["episode_id", "episode_duration", "youtube_avg_watch_duration"]]
b["percentage"]=(b.youtube_avg_watch_duration/b.episode_duration)*100

temp=a.append(b).reset_index().drop(["index"], axis=1)

Source = ColumnDataSource(temp)

tooltips = [
    ("Episode Id", "@episode_id"),
    ("Episode Duration", "@episode_duration"),
    ("Youtube Avg Watch_duration Views", "@youtube_avg_watch_duration"),
    ("Percentage of video watched", "@percentage"),
    ]


fig1 = figure(background_fill_color="#ebf4f6", plot_width = 1000, plot_height = 400, x_range  = temp["episode_id"].values, title = "Percentage of Episode Watched")
fig1.line("episode_id", "percentage", source = Source, color = "#03c2fc", alpha = 0.8)
fig1.line("episode_id", temp.percentage.mean(), source = Source, color = "#f2a652", alpha = 0.8,line_dash="dashed", legend_label="Mean : {:.3f}".format(temp.percentage.mean()))

fig1.add_tools(HoverTool(tooltips=tooltips))
fig1.xaxis.axis_label = "Episode Id"
fig1.yaxis.axis_label = "Percentage"

fig1.xaxis.major_label_orientation = np.pi / 3
show(column(fig1))<jupyter_output><empty_output><jupyter_text>📌 Observations :* On an average, **13.065%** of Episode is watched by Viewers* But **most Episode** haave watched percentage **less** than this threshold.  How does it make sense ❓* That's because we have **some outliers** like E0 and M series that has watched percentage over **20%**.  But Why such outliers occured ❓* That's because they have **low Episode Duration**In this fast moving world, Humans get **bored** of things very **easily**. E0 and M Series having **low Episode Duration** made viewers to **watch more**.If they'll subscribe to the channel or not that's a different thing. That depends on the content.In order to give more to Viewers and Community, Short lengthed Episodes can be a big step  Performance on Other Platforms<jupyter_code>colors = ["red", "olive", "darkred", "goldenrod"]

index={
    0:"YouTube default playlist image",
    1:"CTDS Branding",
    2:"Mini Series: Custom Image with annotations",
    3:"Custom image with CTDS branding, Title and Tags"
}

p = figure(background_fill_color="#ebf4f6", plot_width=600, plot_height=300, title="Thumbnail Type VS Anchor Plays")

base, lower, upper = [], [], []

for each_thumbnail_ref in index:
    if each_thumbnail_ref==2:
        temp = fastai_df[fastai_df.youtube_thumbnail_type==each_thumbnail_ref].anchor_plays 
    else:
        temp = Episode_df[Episode_df.youtube_thumbnail_type==each_thumbnail_ref].anchor_plays
    mpgs_mean = temp.mean()
    mpgs_std = temp.std()
    lower.append(mpgs_mean - mpgs_std)
    upper.append(mpgs_mean + mpgs_std)
    base.append(each_thumbnail_ref)

    source_error = ColumnDataSource(data=dict(base=base, lower=lower, upper=upper))
    p.add_layout(
        Whisker(source=source_error, base="base", lower="lower", upper="upper")
    )

    tooltips = [
        ("Episode Id", "@episode_id"),
        ("Hero Present", "@heroes"),
        ]

    color = colors[each_thumbnail_ref % len(colors)]
    p.circle(y=temp, x=each_thumbnail_ref, color=color, legend_label=index[each_thumbnail_ref])
    print("Mean Anchor Plays for Thumbnail Type {} : {:.3f} ".format(index[each_thumbnail_ref], temp.mean()))
show(p)<jupyter_output>Mean Anchor Plays for Thumbnail Type YouTube default playlist image : 620.939 
Mean Anchor Plays for Thumbnail Type CTDS Branding : 534.500 
Mean Anchor Plays for Thumbnail Type Mini Series: Custom Image with annotations : 309.000 
Mean Anchor Plays for Thumbnail Type Custom image with CTDS branding, Title and Tags : 387.375<jupyter_text>📌 Observations :    * **55.40%** of the Anchor Thumbnail have **CTDS Branding*** But on an Average, **Podcasts with YouTube default playlist** image performs **better** in terms of Anchor Plays<jupyter_code>Episode_df.release_date = pd.to_datetime(Episode_df.release_date)
Source = ColumnDataSource(Episode_df)

tooltips = [
    ("Episode Id", "@episode_id"),
    ("Episode Title", "@episode_name"),
    ("Hero Present", "@heroes"),
    ("Anchor Plays", "@anchor_plays"),
    ("Category", "@category"),
    ("Date", "@release_date{%F}"),
    ]

tooltips2 = [
    ("Episode Id", "@episode_id"),
    ("Episode Title", "@episode_name"),
    ("Hero Present", "@heroes"),
    ("Spotify Starts Plays", "@spotify_starts"),
    ("Spotify Streams", "@spotify_streams"),
    ("Spotify Listeners", "@spotify_listeners"),
    ("Category", "@category"),
    ("Date", "@release_date{%F}"),
    ]


fig1 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400, x_axis_type = "datetime", title = "Anchor Plays Per Episode")
fig1.line("release_date", "anchor_plays", source = Source, color = "#03c2fc", alpha = 0.8, legend_label="Anchor Plays")
fig1.line("release_date", Episode_df.anchor_plays.mean(), source = Source, color = "#f2a652", alpha = 0.8, line_dash="dashed", legend_label="Anchor Plays Mean : {:.3f}".format(Episode_df.youtube_ctr.mean()))


fig1.add_tools(HoverTool(tooltips=tooltips,formatters={"@release_date": "datetime"}))
fig1.xaxis.axis_label = "Release Date"
fig1.yaxis.axis_label = "Anchor Plays"

fig2 = figure(background_fill_color="#ebf4f6", plot_width = 600, plot_height = 400, x_axis_type = "datetime", title = "Performance on Spotify Per Episode")
fig2.line("release_date", "spotify_starts", source = Source, color = "#03c2fc", alpha = 0.8, legend_label="Spotify Starts Plays")
fig2.line("release_date", "spotify_streams", source = Source, color = "#f2a652", alpha = 0.8, legend_label="Spotify Streams")
fig2.line("release_date", "spotify_listeners", source = Source, color = "#03fc5a", alpha = 0.8, legend_label="Spotify Listeners")


fig2.add_tools(HoverTool(tooltips=tooltips2,formatters={"@release_date": "datetime"}))
fig2.xaxis.axis_label = "Release Date"
fig2.yaxis.axis_label = "Total Plays"


show(column(fig1,fig2))<jupyter_output><empty_output><jupyter_text>* It's 2020 and Seems now-a-days people aren't not much into podcasts and Audios Distribution of Heores by Country and Nationality<jupyter_code>temp=Episode_df.groupby(["heroes_location", "heroes"])["heroes_nationality"].value_counts()

parent=[]
names =[]
values=[]
heroes=[]
for k in temp.index:
    parent.append(k[0])
    heroes.append(k[1])
    names.append(k[2])
    values.append(temp.loc[k])

df = pd.DataFrame(
    dict(names=names, parents=parent,values=values, heroes=heroes))
df["World"] = "World"

fig = px.treemap(
    df,
    path=['World', 'parents','names','heroes'], values='values',color='parents')

fig.update_layout( 
    width=1000,
    height=700,
    title_text="Distribution of Heores by Country and Nationality")
fig.show()<jupyter_output><empty_output><jupyter_text>* Most of our Heroes lives in USA but There's quite range of Diversity in Heroes Nationality within a country which is good to know  Any Relation between Release Date of Epiosdes?<jupyter_code>a=Episode_df.release_date
b=(a-a.shift(periods=1, fill_value='2019-07-21')).astype('timedelta64[D]')
d = {'episode_id':Episode_df.episode_id, 'heroes':Episode_df.heroes, 'release_date': Episode_df.release_date, 'day_difference': b}
temp = pd.DataFrame(d)

Source = ColumnDataSource(temp)

tooltips = [
    ("Episode Id", "@episode_id"),
    ("Hero Present", "@heroes"),
    ("Day Difference", "@day_difference"),
    ("Date", "@release_date{%F}"),
    ]

fig1 = figure(background_fill_color="#ebf4f6", plot_width = 1000, plot_height = 400, x_axis_type  = "datetime", title = "Day difference between Each Release Date")
fig1.line("release_date", "day_difference", source = Source, color = "#03c2fc", alpha = 0.8)

fig1.add_tools(HoverTool(tooltips=tooltips,formatters={"@release_date": "datetime"}))
fig1.xaxis.axis_label = "Date"
fig1.yaxis.axis_label = "No of Days"

fig1.xaxis.major_label_orientation = np.pi / 3
show(column(fig1))<jupyter_output><empty_output><jupyter_text>📌 Observations :* Seems 2020 made Sanyam a bit consistant on his Release Date having a difference of 3 or 4 days between each release till 18th July Do I know about Release of anniversary interview episode?Because of time shortage, I didn't scraped new data myself.Though I visited his Youtube Channel and manually examined his Release Patterns| Episode Id    |      Release      |    Day Difference   ||----------|:-------------:|:-------------:|| E75 |  2020-06-18 |  4 || E76 |  2020-06-21 |  3 || E77 |  2020-06-28 |  7 || E78 |  2020-07-02 |  4 || E79 |  2020-07-09 |  7 || E80 |  2020-07-12 |  3 |Maybe He's experimenting with a new pattern Can We pin-point when 1 year anniversary interview episode❓**Actually No!**Though a small pattern can be observed in Release dates, He has bit **odd recording pattern** :* Who knows He may have 3-4 videos already recorded and ready to be released. * Even if He records anniversary interview episode today, We can not say when He'll gonna release that EpisodeAs per his Release pattern, He's been releasing his Episodes **after 3 or 4 days**. * Considering **E77 and E79 as exception**, He'll more probably release **E81** on **2020-07-16** or **2020-07-15*** If He's experimenting with a new pattern (**7 day difference after one video**), then E81 will be released on **2020-07-19** followed by **E82** on **2020-07-22** or **2020-07-23**If I'm correct then Mr. [<NAME>](https://www.kaggle.com/init27), Please don't forget to give a small shoutout to me 😄  Exploring Raw / Cleaned SubstitlesGo back to our GuideSo, We have 2 directories here :* Raw Subtitles : Tanscript in Text format* Cleaned Subtitles : Tanscript in CSV format with Timestamp<jupyter_code>def show_script(id):
    return pd.read_csv("../input/chai-time-data-science/Cleaned Subtitles/{}.csv".format(id))
df = show_script("E1")
df<jupyter_output><empty_output><jupyter_text>A Small Shoutout to Ramshankar YadhunathI would like to give a small shoutout to [Ramshankar Yadhunath](https://www.kaggle.com/thedatabeast) for providing a feature engineering script in his [Kernel](https://www.kaggle.com/thedatabeast/making-perfect-chai-and-other-tales). Hey Guys, If you followed me till here, then dont forget to check out his [Kernel](https://www.kaggle.com/thedatabeast/making-perfect-chai-and-other-tales) too.<jupyter_code># feature engineer the transcript features
def conv_to_sec(x):
    """ Time to seconds """

    t_list = x.split(":")
    if len(t_list) == 2:
        m = t_list[0]
        s = t_list[1]
        time = int(m) * 60 + int(s)
    else:
        h = t_list[0]
        m = t_list[1]
        s = t_list[2]
        time = int(h) * 60 * 60 + int(m) * 60 + int(s)
    return time


def get_durations(nums, size):
    """ Get durations i.e the time for which each speaker spoke continuously """

    diffs = []
    for i in range(size - 1):
        diffs.append(nums[i + 1] - nums[i])
    diffs.append(30)  # standard value for all end of the episode CFA by Sanyam
    return diffs


def transform_transcript(sub, episode_id):
    """ Transform the transcript of the given episode """

    # create the time second feature that converts the time into the unified qty. of seconds
    sub["Time_sec"] = sub["Time"].apply(conv_to_sec)

    # get durations
    sub["Duration"] = get_durations(sub["Time_sec"], sub.shape[0])

    # providing an identity to each transcript
    sub["Episode_ID"] = episode_id
    sub = sub[["Episode_ID", "Time", "Time_sec", "Duration", "Speaker", "Text"]]

    return sub


def combine_transcripts(sub_dir):
    """ Combine all the 75 transcripts of the ML Heroes Interviews together as one dataframe """

    episodes = []
    for i in range(1, 76):
        file = "E" + str(i) + ".csv"
        try:
            sub_epi = pd.read_csv(os.path.join(sub_dir, file))
            sub_epi = transform_transcript(sub_epi, ("E" + str(i)))
            episodes.append(sub_epi)
        except:
            continue
    return pd.concat(episodes, ignore_index=True)


# create the combined transcript dataset
sub_dir = "../input/chai-time-data-science/Cleaned Subtitles"
transcripts = combine_transcripts(sub_dir)
transcripts.head()<jupyter_output><empty_output><jupyter_text>Now, We have some Data to work with. Thanking [<NAME>](https://www.kaggle.com/thedatabeast) once again, Let's get it started ..  Note : Transcript for E0 and E4 is missing Intro is Bad for CTDS ?From our previous analysis, We realised Majority of the Episodes have quite less watch time i.e. less than 13.065% of the total Duration. In such case, How much CTDS intro hurts itself in terms of intro duration.Let's find out...<jupyter_code>temp = Episode_df[["episode_id","youtube_avg_watch_duration"]]
temp=temp[(temp.episode_id!="E0") & (temp.episode_id!="E4")]

intro=[]

for i in transcripts.Episode_ID.unique():
    intro.append(transcripts[transcripts.Episode_ID==i].iloc[0].Duration)
temp["Intro_Duration"]=intro
temp["diff"]=temp.youtube_avg_watch_duration-temp.Intro_Duration

Source = ColumnDataSource(temp)

tooltips = [
    ("Episode Id", "@episode_id"),
    ("Youtube Avg Watch_duration Views", "@youtube_avg_watch_duration"),
    ("Intro Duration", "@Intro_Duration"),
    ("Avg Duration of Content Watched", "@diff"),
    ]


fig1 = figure(background_fill_color="#ebf4f6", plot_width = 1000, plot_height = 600, x_range  = temp["episode_id"].values, title = "Impact of Intro Durations")
fig1.line("episode_id", "youtube_avg_watch_duration", source = Source, color = "#03c2fc", alpha = 0.8, legend_label="Youtube Avg Watch_duration Views")
fig1.line("episode_id", "Intro_Duration", source = Source, color = "#f2a652", alpha = 0.8, legend_label="Intro Duration")
fig1.line("episode_id", "diff", source = Source, color = "#03fc5a", alpha = 0.8, legend_label="Avg Duration of Content Watched")


fig1.add_tools(HoverTool(tooltips=tooltips))
fig1.xaxis.axis_label = "Episode Id"
fig1.yaxis.axis_label = "Percentage"

fig1.xaxis.major_label_orientation = np.pi / 3
show(column(fig1))
print("{:.2f} % of Episodes have Avg Duration of Content Watched less than 5 minutes".format(len(temp[temp["diff"]<300])/len(temp)*100))
print("{:.2f} % of Episodes have Avg Duration of Content Watched less than 4 minutes".format(len(temp[temp["diff"]<240])/len(temp)*100))
print("{:.2f} % of Episodes have Avg Duration of Content Watched less than 3 minutes".format(len(temp[temp["diff"]<180])/len(temp)*100))
print("{:.2f} % of Episodes have Avg Duration of Content Watched less than 2 minutes".format(len(temp[temp["diff"]<120])/len(temp)*100))
print("In {} case, Viewer left in the Intro Duration".format(len(temp[temp["diff"]<0])))<jupyter_output>81.08 % of Episodes have Avg Duration of Content Watched less than 5 minutes
72.97 % of Episodes have Avg Duration of Content Watched less than 4 minutes
45.95 % of Episodes have Avg Duration of Content Watched less than 3 minutes
22.97 % of Episodes have Avg Duration of Content Watched less than 2 minutes
In 1 case, Viewer left in the Intro Duration<jupyter_text>🧠 My Cessation: * Observing the graph and stats, it's clear it's a high time* We don't have Transcript of M Series where the Percentage of Episode Watched i.e. Episode had small Duration.* Concluding from analysis We can now strongly comment, Episode with shorter length will definitely helpThere's lots of things to improve.* Shorter Duration Videos can be delivered highlighting the important aspects of the Shows* Short Summaries can be provided in the description. Maybe after reading them, Viewers could devote for a longer Show (depending on the interest on the topic reflected in summery)* Full length Show can be provided as Podcast in Apple, Spotify, Anchor. If Viewer after shorter duration videos and summeries wishes to have a full show, they can have it from thereWith 45.95% of Episodes having Avg Duration of Content Watched less than 3 minutes, We can hardly gain any useful insight or can comment on quality of Content delivered.But Okay! We can have some fun though 😄 Who Speaks More ?<jupyter_code>host_text = []
hero_text = []
for i in transcripts.Episode_ID.unique():
    host_text.append([i, transcripts[(transcripts.Episode_ID==i) & (transcripts.Speaker=="<NAME>")].Text])
    hero_text.append([i, transcripts[(transcripts.Episode_ID==i) & (transcripts.Speaker!="<NAME>")].Text])

temp_host={}
temp_hero={}
for i in range(len(transcripts.Episode_ID.unique())):
    host_text_count = len(host_text[i][1])
    hero_text_count = len(hero_text[i][1])
    temp_host[hero_text[i][0]]=host_text_count
    temp_hero[hero_text[i][0]]=hero_text_count
    
def getkey(dict): 
    list = [] 
    for key in dict.keys(): 
        list.append(key)          
    return list

def getvalue(dict): 
    list = [] 
    for key in dict.values(): 
        list.append(key)          
    return list
Source = ColumnDataSource(data=dict(
    x=getkey(temp_host),
    y=getvalue(temp_host),
    a=getkey(temp_hero),
    b=getvalue(temp_hero),
))

tooltips = [
    ("Episode Id", "@x"),
    ("No of Times Host Speaks", "@y"),
    ("No of Times Hero Speaks", "@b"),
]

fig1 = figure(background_fill_color="#ebf4f6",plot_width = 1000, tooltips=tooltips,plot_height = 400, x_range = getkey(temp_host), title = "Who Speaks More ?")
fig1.vbar("x", top = "y", source = Source, width = 0.4, color = "#76b4bd", alpha=.8, legend_label="No of Times Host Speaks")
fig1.vbar("a", top = "b", source = Source, width = 0.4, color = "#e7f207", alpha=.8, legend_label="No of Times Hero Speaks")

fig1.xaxis.axis_label = "Episode"
fig1.yaxis.axis_label = "Count"

fig1.grid.grid_line_color="#feffff"
fig1.xaxis.major_label_orientation = np.pi / 4

show(fig1)<jupyter_output><empty_output><jupyter_text>* Excluding Few Episodes, Ratio between No of Times One Speaks is quite mantained* E69 was AMA episode. That's why there is no Hero Frequency of Questions Per Episode<jupyter_code>ques=0
total_ques={}
for episode in range(len(transcripts.Episode_ID.unique())):
    for each_text in range(len(host_text[episode][1])):
        ques += host_text[episode][1].reset_index().iloc[each_text].Text.count("?")
    total_ques[hero_text[episode][0]]= ques
    ques=0
from statistics import mean 
Source = ColumnDataSource(data=dict(
    x=getkey(total_ques),
    y=getvalue(total_ques),
))

tooltips = [
    ("Episode Id", "@x"),
    ("No of Questions", "@y"),
]

fig1 = figure(background_fill_color="#ebf4f6",plot_width = 1000, plot_height = 400,tooltips=tooltips, x_range = getkey(temp_host), title = "Questions asked Per Episode")
fig1.vbar("x", top = "y", source = Source, width = 0.4, color = "#76b4bd", alpha=.8, legend_label="No of Questions asked Per Episode")
fig1.line("x", mean(getvalue(total_ques)), source = Source, color = "#f2a652", alpha = 0.8,line_dash="dashed", legend_label="Average Questions : {:.3f}".format(mean(getvalue(total_ques))))

fig1.xaxis.axis_label = "Episode"
fig1.yaxis.axis_label = "No of Questions"

fig1.legend.location = "top_left"

fig1.grid.grid_line_color="#feffff"
fig1.xaxis.major_label_orientation = np.pi / 4

show(fig1)<jupyter_output><empty_output><jupyter_text>* On an Average, around 30 Questions are asked by Host* E69 being AMA Episode justifies the reason of having such high no of Questions Favourite Text ?⚒️ About the Function : Well, I'm gonna write a small function. You can pass a Hero Name and it will create a graph about 7 most common words spoken by that person But before that I would like to give a small Shoutout to [<NAME>](https://www.kaggle.com/parulpandey) for providing a text cleaning script in her [Kernel](https://www.kaggle.com/parulpandey/how-to-explore-the-ctds-show-data).<jupyter_code>import re
import nltk
from statistics import mean 
from collections import Counter
import string

def clean_text(text):
    '''Make text lowercase, remove text in square brackets,remove links,remove punctuation
    and remove words containing numbers.'''
    text = text.lower()
    text = re.sub('\[.*?\]', '', text)
    text = re.sub('https?://\S+|www\.\S+', '', text)
    text = re.sub('<.*?>+', '', text)
    text = re.sub('[%s]' % re.escape(string.punctuation), '', text)
    text = re.sub('\n', '', text)
    text = re.sub('\w*\d\w*', '', text)
    return text


def text_preprocessing(text):
    """
    Cleaning and parsing the text.

    """
    tokenizer = nltk.tokenize.RegexpTokenizer(r'\w+')
    nopunc = clean_text(text)
    tokenized_text = tokenizer.tokenize(nopunc)
    #remove_stopwords = [w for w in tokenized_text if w not in stopwords.words('english')]
    combined_text = ' '.join(tokenized_text)
    return combined_text
transcripts['Text'] = transcripts['Text'].apply(str).apply(lambda x: text_preprocessing(x))
def get_data(speakername=None):
    label=[]
    value=[]

    text_data=transcripts[(transcripts.Speaker==speakername)].Text.tolist()
    temp=list(filter(lambda x: x.count(" ")<10 , text_data)) 

    freq=nltk.FreqDist(temp).most_common(7)
    for each in freq:
        label.append(each[0])
        value.append(each[1])
        
        
    Source = ColumnDataSource(data=dict(
        x=label,
        y=value,
    ))

    tooltips = [
        ("Favourite Text", "@x"),
        ("Frequency", "@y"),
    ]

    fig1 = figure(background_fill_color="#ebf4f6",plot_width = 600, tooltips=tooltips, plot_height = 400, x_range = label, title = "Favourite Text")
    fig1.vbar("x", top = "y", source = Source, width = 0.4, color = "#76b4bd", alpha=.8)

    fig1.xaxis.axis_label = "Text"
    fig1.yaxis.axis_label = "Frequency"


    fig1.grid.grid_line_color="#feffff"
    fig1.xaxis.major_label_orientation = np.pi / 4

    show(fig1)

get_data(speakername="<NAME>")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13123975102710136, 'fraction_numerical': 0.022217128347997814, 'mean_word_length': 5.05142979086641, 'pattern_counts': {'":': 15, '<': 150, '<?xml version=': 0, '>': 156, 'https://': 30, 'lorem ipsum': 0, 'www.': 17, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 131, 'profanity_word_list.json': 1, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '8297', 'license': 'MIT', 'n_tokens_mistral': 19424, 'n_tokens_neox': 18150, 'n_words': 5677, 'path': 'src/what-s-this-chai-and-datascience.ipynb', 'repo_name': 'RsTaK/CTDS.SHOW', 'size': '56845'}