<jupyter_start><jupyter_text>AuthorsHere I am going to do some analysis of the Beatles songs authorship. Only official authorship data is analyzed, i.e., all non-cover songs belong to Lennon-McCartney, Harrison, or Starkey. Sure, the lion share of the Beatles songs is attributed to Lennon-McCartney. But perhaps, there is some interesting information behind it to reveal. Let's look at the share of songs by authors first. What is the share of covers?<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

# this allows plots to appear directly in the notebook
%matplotlib inline

# load data
df = pd.read_json('data.json')

# configure graphics 
plt.style.use('ggplot')
plt.figure()
color = dict(boxes='DarkGreen', whiskers='DarkOrange', medians='DarkBlue', caps='Gray')
pd.value_counts(df['cover'], sort=True).plot.pie(title='Cover songs', autopct='%.2f', fontsize=12)<jupyter_output><empty_output><jupyter_text>About 88% are original songs. Let's analyze them.<jupyter_code>dfnc = df[df.cover==False]

pd.value_counts(dfnc['formal_writers'], sort=True).plot.pie(autopct='%.2f', fontsize=12)<jupyter_output><empty_output><jupyter_text>87% by Lennon-McCartney, about 12% by Harrison. Poor Ringo has only 1%. But that's rather obvious. What's the time dynamics?<jupyter_code>fig, axs = plt.subplots(figsize=(12, 6), nrows=1, ncols=3)

for ax, singer in zip(axs.flatten(), ('Lennon-McCartney', 'Harrison', 'Starkey')):

    gr = dfnc[dfnc['formal_writers']==singer].groupby('year')
    writers_list = []

    for key in gr.groups.keys():
        year = gr.get_group(key)['year'].values[0]
        
        songs_by_author = gr.get_group(key)['formal_writers'].value_counts().sum()
        songs_by_all = df[df['year']==gr.get_group(key)['year'].values[0]]['formal_writers'].count() * 100
        
        writers = songs_by_author / songs_by_all
        writers_list.append({'year': year, 'writers': writers})

    writers_df = pd.DataFrame(index=[k['year'] for k in writers_list], data={'writers': [k['writers'] for k in writers_list]})
    writers_df.sort_index()['writers'].plot(kind='bar', ax=ax, title=singer, rot=45)<jupyter_output><empty_output><jupyter_text>The share of Lennon-McCartney's songs is gradually rising to the maximum of about 90% in 1967, and then is falling down a bit. Harrison's share is different from year to year. Ringo contributed in two years only.But we include covers in the stats. But what if we don't?<jupyter_code>fig, axs = plt.subplots(figsize=(12, 6), nrows=1, ncols=3)

for ax, singer in zip(axs.flatten(), ('Lennon-McCartney', 'Harrison', 'Starkey')):

    gr = dfnc[dfnc['formal_writers']==singer].groupby('year')
    writers_list = []

    for key in gr.groups.keys():
        year = gr.get_group(key)['year'].values[0]
        
        songs_by_author = gr.get_group(key)['formal_writers'].value_counts().sum()
        songs_by_all = dfnc[dfnc['year']==gr.get_group(key)['year'].values[0]]['formal_writers'].count() * 100
        
        writers =  songs_by_author / songs_by_all
        writers_list.append({'year': year, 'writers': writers})

    writers_df = pd.DataFrame(index=[k['year'] for k in writers_list], data={'writers': [k['writers'] for k in writers_list]})
    writers_df.sort_index()['writers'].plot(kind='bar', ax=ax, title=singer, rot=45)<jupyter_output><empty_output><jupyter_text>We see that the share of Lennon-McCartney's songs is actually slightly falling with time, while Harrison's one is generaly rising.Let's look at authorship against vocals, length, number of words in lyrics and number of words in titles.So, how many of songs sung by each musician were written by them?<jupyter_code># number of songs sung by Lennon / number of songs by Lennon-McCartney
dfnc[dfnc.vocals=='Lennon'].count()[0] / dfnc[dfnc.formal_writers=='Lennon-McCartney'].count()[0] * 100
# number of songs sung by McCartney / number of songs by Lennon-McCartney 
dfnc[dfnc.vocals=='McCartney'].count()[0] / dfnc[dfnc.formal_writers=='Lennon-McCartney'].count()[0] * 100<jupyter_output><empty_output><jupyter_text>So Lennon and McCartney as solo singers sang about the same amount of the songs they wrote, about a third each. Other songs they wrote were sung by other Beatles members or in cooperation with them. But what about Harrison?<jupyter_code># number of songs sung by Harrison / number of songs by Harrison
dfnc[dfnc.vocals=='Harrison'].count()[0] / dfnc[dfnc.formal_writers=='Harrison'].count()[0] * 100<jupyter_output><empty_output><jupyter_text>Harrison sings more than 100% of songs he wrote! How can it be? Actually, Harrison sang several songs by Lennon-McCartney (while they didn't sing songs by Harrison or Starr). Which are they?<jupyter_code># original songs sung by Harrison but not written by him
dfnc[(dfnc.vocals=='Harrison') & (dfnc.formal_writers!='Harrison')][['title', 'formal_writers']]<jupyter_output><empty_output><jupyter_text>So there are two songs by Lennon-McCartney sung by Harrison alone. And what about Ringo?<jupyter_code># original songs sung by Harrison but not written by him
dfnc[(dfnc['vocals']=='Starr') & (dfnc['formal_writers']!='Starkey')][['title', 'formal_writers']]<jupyter_output><empty_output><jupyter_text>Oh well, Ringo sang four songs written for him by Lennon & Mccartney.Let's look at lengths then.<jupyter_code># length medians by writers
dfnc.groupby('formal_writers')['length'].median()
dfnc.boxplot(column='length', by='formal_writers', figsize=(8, 8))<jupyter_output>/home/hrutr/.virtualenvs/tbsa/lib/python3.5/site-packages/numpy/core/fromnumeric.py:57: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
  return getattr(obj, method)(*args, **kwds)<jupyter_text>We see that the median lengths are a little different; songs by Starr are the longest, by Lennon-McCartney are the shortest, and by Harrison are in between. Lennon-McCartney songs have the biggest number of outliers and much higher range.<jupyter_code># number of words in title medians by writers
dfnc.groupby('formal_writers')['num_words_in_title'].median()
dfnc.boxplot(column='num_words_in_title', by='formal_writers', figsize=(8, 8))<jupyter_output>/home/hrutr/.virtualenvs/tbsa/lib/python3.5/site-packages/numpy/core/fromnumeric.py:57: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
  return getattr(obj, method)(*args, **kwds)<jupyter_text>So, songs of all authors have the same median number of words in titles: it's three. But all of them, except Starkey, have many songs that have more.<jupyter_code># number of words in lyrics medians by writers
dfnc.groupby('formal_writers')['num_words_in_lyrics'].median()
dfnc.boxplot(column='num_words_in_lyrics', by='formal_writers', figsize=(8, 8))<jupyter_output>/home/hrutr/.virtualenvs/tbsa/lib/python3.5/site-packages/numpy/core/fromnumeric.py:57: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
  return getattr(obj, method)(*args, **kwds)<jupyter_text>Harrison has the shortest lyrics, Lennon and McCartney go second, and the longest lyrics belong to Ringo's songs. Still the outliers from both sides belong to Lennon-Mccartney.Finally, let's plot the data in the coordinates of number of words in lyrics and length, just to see with own eyes in what kind Lennon-McCartney's songs are different to Harrison's and Starkey's. It's clearly seen that songs by Harrison and Starr are grouped mostly in the center.<jupyter_code>ax = dfnc[dfnc['formal_writers']=='Lennon-McCartney'].plot.scatter(x='num_words_in_lyrics', y='length', 
                                                                figsize=(12, 4), color='Blue', 
                                                                label='Lennon-McCartney')
dfnc[dfnc['formal_writers']=='Harrison'].plot.scatter(x='num_words_in_lyrics', y='length', color='Green', 
                                                   label='Harrison', ax=ax)
dfnc[dfnc['formal_writers']=='Starkey'].plot.scatter(x='num_words_in_lyrics', y='length', color='Red', 
                                                  label='Starkey', ax=ax)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1266634406000484, 'fraction_numerical': 0.009920154851197677, 'mean_word_length': 4.921919770773639, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '608570', 'license': 'MIT', 'n_tokens_mistral': 2778, 'n_tokens_neox': 2612, 'n_words': 746, 'path': 'TBSA - Step 4.ipynb', 'repo_name': 'dpol2000/tbsa', 'size': '8266'}