<jupyter_start><jupyter_text>The Goal ![Final product preview](https://github.com/warwickdatascience/spotify-workshop/raw/master/notebooks/wall_of_music.png)  Part One  Fundamentals  Running Code You can create a code cell in Colab by selecting an existing cell and selecting the `+ Code` button on the top bar.You can then enter code into a cell. For example, use `print("Hello, World!")` to say hello.Finally, you can run code by pressing the play button next to the cell (or use `ctrl+enter`/`cmd+enter` (OSX)). If you want to step code before it completes, press the stop button.Note: Python generally doesn't care about spaces or new lines, but identation does matter.Lines beginning with `` are not run. These are called comments.You will find `TODO`s followed by `"_______"`s. These are for you to fill in during the breakout sessions.There are also a few `CKPT`s. These are optional checkpoint tasks that don't appear in the final notebook. Complete these to help understand what your data looks like at any point.  Imports _Live explanation_ [PyPI](https://pypi.org/)  Variables Variables are a way of given Python memory. The format for assigning a value to a variable is:```pythonvariable_name = value```Note, the variable names can only contain letters, numbers, underscores, and hyphens (no spaces!). Once a variable has been assigned a value, whenever Python sees it in the future, it will replace it with the value it was last set to.<jupyter_code>rpm = 45
print(rpm)
rpm = 78
print(rpm)<jupyter_output>78<jupyter_text>Be careful! Google Colab lets you run cells out of order. It is advised to run things from top to bottom only to avoid confusion. To run multiple cells, look at the `Runtime` dropdown on the top bar. High-level parameters that do not change during the script are often denoted with all-caps.  Variable Types If you want to pass text data to Python, you need to explicitly use single or double quotes (they mean the same) to let Python know that you intended to input text and not a variable.<jupyter_code>greeting = 'Hello'
print(greeting)
greeting = "Hello"
print(greeting)
greeting = Hello<jupyter_output><empty_output><jupyter_text>Functions You use functions to get Python to do something. They have a common syntax:```pythonfunction_name(input1, input2, ...)```Some functions _return_ a value, in which case we can assign this to a variable.```pythonvariable = function_name(input1, input2, ...)```Some inputs have names, in which case we use this syntax.```pythonfunction_name(input1, named_input=input2)```Functions that belong to a certain package/module can be accessed using dot notation.<jupyter_code># Import the package `numpy` using the name `np`
import numpy as np
# Create a 2x2 identity matrix
np.identity(2)
np.random.uniform(size=(2, 3))<jupyter_output><empty_output><jupyter_text>We have written several helper functions to make your life easier and code cleaner. You don't need to do anything to these apart from run the cell that contains them. A description of what they do is given at the top of their definitions.  Data Structures There are two key data structures we will be using.The first is the list. This is used to store a sequence of ordered elements. We use square brackets to create them.<jupyter_code>past_olympics_hosts = ['Rio', 'London', 'Beijing', 'Athens']<jupyter_output><empty_output><jupyter_text>We can then access the elements using non-negative integers (think of these as 'house numbers' for data but starting at zero) and square brackets.<jupyter_code>past_olympics_hosts[1]<jupyter_output><empty_output><jupyter_text>Another useful structure is the dictionary. This is used for mapping a key to a value. We create them using curly brackets containing comma-separated key value pairs.<jupyter_code>capitals = {
    'spain': 'madrid',
    'france': 'paris',
    'portugal': 'lisbon'
}<jupyter_output><empty_output><jupyter_text>We access these in the same way as lists, but using a key rather than an index.<jupyter_code>capitals['spain']<jupyter_output><empty_output><jupyter_text>For Loops Loops allow us to run a block of code multiple times. The most common type is a for loop. This iterates over a data structure, using a temporary variable to keep track of where we are. The notation is as follows.<jupyter_code>for city in past_olympics_hosts:
    print("-", city)<jupyter_output>- Rio
- London
- Beijing
- Athens<jupyter_text>Sometimes we wish to loop over a sequence of numbers, in which case we use a `range` object. This will iterate from the first number up to (but not including) the second.<jupyter_code>for n in range(2, 5):
    print(n)<jupyter_output>2
3
4<jupyter_text>The Spotify API  APIs _Live explanation_Example: `https://www.googleapis.com/books/v1/volumes?q=isbn:9781491912058`[List of public APIs](https://github.com/public-apis/public-apis)  The Spotify API Anyone with a Spotify account (free or premium) can use Spotify API at no cost. First, visit [developer.spotify.com/](https://developer.spotify.com/), sign-in, and select 'Dashboard' form the top bar.From here, click 'Create an App' and fill in the details. These can be anything reasonable. For example, "WDSS Spotify Workshop", "An app for learning about the Spotify API" would do.Each app has a client ID and secret. There's no real harm in sharing either but it is best practice to keep both or at the very least the secret hidden. You will need to use these when creating a connection to the API in the template. With these credentials you can make queries related to **publically accessible data**, such as users' public playlists, track/artist/album information and audio features/analysis. It is also possible to use this API to create applications that read a user's personal Spotify data and make changes but this requires a second level of authentication that we will not look at (you can read about this [here](https://developer.spotify.com/documentation/general/guides/authorization-guide/) and [here](https://spotipy.readthedocs.io/en/2.15.0/authorization-code-flow) though)  SpotipyWe could access the Spotify API directly using web requests, but this is added work. Instead, rather than reinventing the wheel, we use `spotipy`, a Python wrapper for the API. We create a client `sp` using our authentication keys and then access its methods to make queries. For example, `sp.featured_playlists()` will get a list of Spotify's featured playlists. Note, certain `spotipy` features won't work without setting up OAuth.[Spotipy documentation](https://spotipy.readthedocs.io/en/2.15.0/)  Usernames You can find your Spotify username by visiting [this link](https://www.spotify.com/uk/account/overview/).If you cannot find your own username, just use ours (`vr1q8on6ji60ceo0kregujkkx`).  Data Scraping  Pagination The Spotify API returns results using pagination. That is, only a certain number of items are returned at a time and you have to request 'page 2', etc. This is all taken care of using the `depaginate` helper, which flattens these results into one list (a generator to be precise).  Audio Features The Spotify API enables you to collect audio features about any tracks including tempo, key, instrumentalness, and danceability.We will use these as the foundation for laying out our wall of music.[List of available audio features](https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/) > **Breakout Session One**>> Complete the setup and data scraping steps of the template  Part Two  Pre-processing  Dataframes Dataframes are a more powerful data structure not included in base Python. Instead, we gain access to these by importing `pandas` (under the alias `pd`). You can print out a dataframe by typing its name in a cell of its own. If you only want the first few rows use `df_name.head()` and `df_name.tail()` for the bottom.  Sampling Our end goal is to create a grid of cover arts. This dimensions of this grid will be determined by the number of tracks in our playlist. If we have 100 tracks then we get a perfect 10-by-10 square. If we add one more to make 101, we now have a prime number of tracks so the best we can do is 1-by-101. This would look ridiculous and so we use the parameter `MIN_RATIO` to set the minimum ratio between side lengths that we are willing to accept. Then, we take our initial number of tracks and reduce it until we can form a rectangle with sides meeting this constraint. This is handled by the `find_valid_sample_count(...)` helper.We then randomly sample this many rows from the table.  Feature Engineering Most of our audio features are ready for use, but the `key` column is certainly not. The twelve western keys are encoded using the numbers 0 to 11. This is inappropriate as it puts the keys C and B at a distance of 11 even though they come right next to each other. | Key | Encoding ||:---:|:--------:||  C  |     0    ||  Db |     1    ||  D  |     2    ||  Eb |     3    ||  E  |     4    ||  F  |     5    ||  F |     6    ||  G  |     7    ||  Ab |     8    ||  A  |     9    ||  Bb |    10    ||  B  |    11    | What we need is a cyclic system. To do this we space the keys around a unit circle and record their x and y coordinates. Now, the key is represented by two variables instead of one but the distance between keys is now correct.<jupyter_code>import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6, 6))
x_circ = np.cos(np.linspace(0, 2 * np.pi, 100))
y_circ = np.sin(np.linspace(0, 2 * np.pi, 100))
ax.plot(x_circ, y_circ, c='k')
x_key = np.cos(np.linspace(0, 11/6 * np.pi, 12))
y_key = np.sin(np.linspace(0, 11/6 * np.pi, 12))
s_key = ['C', 'Db', 'D', 'Eb', 'E', 'F', 'F#', 'G', 'Ab', 'A', 'Bb', 'B']
ax.scatter(x_key, y_key, c='r')
for x, y, s in zip(x_key, y_key, s_key):
    ax.text(x * 1.1, y * 1.1, s, ha='center', va='center')
ax.set_xlim(-1.2, 1.2)
ax.set_ylim(-1.2, 1.2)
plt.show()<jupyter_output><empty_output><jupyter_text>There's also this thing called the circle of fifths. If you don't know music theory, this is not worth sweating about. The bottom line is, there are two ways of measuring the distance between keys: melodically and harmonically. Both of these are useful for us and so we use the helper `mel_to_harm` to convert the keys to an encoding that respresents harmonic distance and repeat the above process.  ![Circle of Fifths](https://hellomusictheory.com/wp-content/uploads/2020/07/circle-00-768x767.png)  Standardisation Our variables exists on wildly different scales (compare `tempo` to `key`, or even `duration_ms`). We standardise these by centering them around zero and scaling them so they have the same unit variance.<jupyter_code>fig, axs = plt.subplots(3, 3, figsize=(12, 8))
mus = (-1, 2, 7)
sds = (1, 4, 2)
col_names = ('Original', 'Centered', 'Scaled')
row_names = ('Variable 1', 'Variable 2', 'Variable 3')
colours = plt.rcParams['axes.prop_cycle'].by_key()['color']
for j in range(3):
    for i, (mu, sd) in enumerate(zip(mus, sds)):
        if j > 0: mu = 0
        if j > 1: sd = 1
        x = np.linspace(-15, 15, 1000)
        y = np.exp(-.5 * ((x - mu) / sd)**2) / (sd * np.sqrt(2 * np.pi))
        ax = axs[i][j]
        ax.axvline(0, c='k', linestyle='--')
        ax.plot(x, y, c=colours[i])
        ax.set_xlim(-15, 15)
        if i == 2: ax.set_xlabel(col_names[j])
        if j == 0: ax.set_ylabel(row_names[i])
        ax.set_yticklabels([])
        ax.set_xticklabels([])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0910757946210269, 'fraction_numerical': 0.015019210618232623, 'mean_word_length': 4.27059364933272, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 47, 'https://': 11, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '561151', 'license': 'MIT', 'n_tokens_mistral': 3508, 'n_tokens_neox': 3325, 'n_words': 1562, 'path': 'notebooks/help-guide.ipynb', 'repo_name': 'DavidMcK9509/spotify-workshop', 'size': '11452'}