<jupyter_start><jupyter_text>setup importing stuff and such<jupyter_code>import spotipy
import matplotlib
import numpy as np
%matplotlib notebook
from matplotlib import pylab as plt
from matplotlib import mlab
sp = spotipy.Spotify()<jupyter_output><empty_output><jupyter_text>fetch all the playlist details that have 'punk' in the name(note that this doesn't get the track lists, we'll do that a bit later)<jupyter_code>results = sp.search(type='playlist', q='punk', limit=50)['playlists']
print "gathering details about", results['total'], "playlists"
punk_playlists = results['items']
while results['next']:
    results = sp.next(results)['playlists']
    punk_playlists += results['items']<jupyter_output>gathering details about 5351 playlists<jupyter_text>basic statsto get a feel for the dataset, let's to do some basic stats before we plow ahead with the track analysis title lengthwe expect a peak a 4 characters (the minimal 'Punk'), what else happens?<jupyter_code>print "number of results:", len(punk_playlists)

print 
title_lengths = filter(lambda c:c<100, map(lambda pl:len(pl['name']), punk_playlists))
n, bins, patches = plt.hist(title_lengths, 50, normed=1, facecolor='green', alpha=0.75)
mu = np.mean(title_lengths)
sigma = np.std(title_lengths)
# add a 'best fit' line
y = mlab.normpdf( bins, mu, sigma)
l = plt.plot(bins, y, 'r--', linewidth=1)

plt.xlabel('Number of Characters')
plt.ylabel('Probability')
plt.title(r'$\mathrm{Histogram\ of\ Punk playlist title lengths:}\ \mu='+str(mu)+',\ \sigma='+str(sigma)+'$')
# plt.axis([40, 160, 0, 0.03])
plt.grid(True)<jupyter_output>number of results: 5351<jupyter_text>ok, so _a_ peak where expected, but the vast majority are longer, mean is just over 16 characters.--- word counts in the titlesSo picking that apart a little more, let's take a look at some lightly cleaned word counts across all the titles<jupyter_code>from collections import Counter
from string import punctuation
stopwords = "and of the or in".split()
print "top words in titles"
word_count = Counter()
for pl in punk_playlists:
    word_count.update([w.strip(punctuation) for w in pl['name'].lower().split() if w not in stopwords and len(w) > 2])
word_count.most_common(10)<jupyter_output>top words in titles<jupyter_text>remember friends: Daft Punk may be playing in your house, your house, but it's a pretty good guess that when 'punk' is proceeded by 'daft' its probably not actually a punk playlist...the other results here are basically the expected neighbouring genres (e.g. 'pop', 'rock', 'metal') and of course some self labelling ('playlist')small aside, this seems to indicate that some of the playlists don't mention 'punk' in the name. Is that a problem? (this makes me wonder how the search algorithm works...). Let's see how many there are and what they look like. That's not punk.<jupyter_code>print len([pl['name'] for pl in punk_playlists if "punk" not in pl['name'].lower()]), "of the search results don't say punk.\n here they are:"
print '\n'.join([pl['name'] for pl in punk_playlists if "punk" not in pl['name'].lower()])<jupyter_output>87 of the search results don't say punk.
 here they are:
Pünk
Workout Shout!
French Touch
Post-Hardcore Crash Course
💰🔪☠
Post Garage Wave Revival
.01
New Rock
🌚 cool songs 🌝
Is It New Wave?
moosic
Locos x los 2010
Teenage Dirtbag
Skunk Rock
Crossfit Hutto Rock
math rock !
Dance Anthems - Ministry of Sound
Hits 2013
From The Garage
Cheap Beer & Dirty Basements
Proper Naughty
Canadians Rock!
Everything Rock
random
THE DEFINITIVE 70s
current songs
Arctic Monkeys: Origens
Emo/Emoish/Indierock
Rock 2
70's ROCK - The Ultimate Playlist
PØP PÛNK
Main Playlist
God tier 2.0
TOPSIFY 80s HITS
for when its cold outside
All Things Post
NOW 85
Die 257ers Party Playlist
Bare tunes
poleng's
Sweat Through This
Tank
Highschool Hits
Vidar
Quote Songs🖊💘
Best of '80s Indie • Alternative eighties
GRUNGE FOR LIFE
Main Playlist
De Boa na Praia
Mi playlist 
Today's Best Boy Bands Piano
Mixed Bag
Idk.
Sunday Breakfast
French Touch
Good. 
Originals vs Covers + Sampled
EMO
cool tunes 
Hard. Heavy. Loud.
Lounge & W[...]
{'fraction_non_alphanumeric': 0.09778102901731285, 'fraction_numerical': 0.015362106803218726, 'mean_word_length': 5.369565217391305, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '10269', 'license': 'MIT', 'n_tokens_mistral': 1439, 'n_tokens_neox': 1323, 'n_words': 535, 'path': 'genre by playlist.ipynb', 'repo_name': 'gearmonkey/genre_playlist_analysis', 'size': '4101'}