<jupyter_start><jupyter_text>Last.fm Downloader  Setup and Installation* Clone credentials-sample.json as credentials.json* Create a [Last.fm API Account](https://www.last.fm/api/account/create)* Copy your key and user id and save to relevant fields in credentials.jsonOnce finished run the cells below to generate CSV exports of your Last.fm music history. **NOTE: Depending on size of your last.fm listening history, this process may take some time. Please be patient.**  Acknowledgements and Helpful Resources   - ["Analyzing Last.fm Listening History"](http://geoffboeing.com/2016/05/analyzing-lastfm-history/) and [Code](https://github.com/gboeing/data-visualization/tree/master/lastfm-listening-history) by <NAME>  - Last.fm API documentation: http://www.last.fm/api  - For anything more complicated, you might use this Python wrapper for the API: https://github.com/pylast/pylast -----  Credentials and Authentification<jupyter_code>import json

with open("credentials.json", "r") as file:
    credentials = json.load(file)
    last_fm_cr = credentials['last_fm']
    key = last_fm_cr['KEY']
    username = last_fm_cr['USERNAME']
# how long to pause between consecutive API requests
pause_duration = 0.2<jupyter_output><empty_output><jupyter_text>Dependencies<jupyter_code>import requests
import json
import time
import pandas as pd<jupyter_output><empty_output><jupyter_text>First get your all-time most played tracks, artists, and albums<jupyter_code>url = 'https://ws.audioscrobbler.com/2.0/?method=user.get{}&user={}&api_key={}&limit={}&extended={}&page={}&format=json'
limit = 200 #api lets you retrieve up to 200 records per call
extended = 0 #api lets you retrieve extended data for each track, 0=no, 1=yes
page = 1 #page of results to start retrieving at
method = 'toptracks'
request_url = url.format(method, username, key, limit, extended, page)
artist_names = []
track_names = []
play_counts = []
response = requests.get(request_url).json()
for item in response[method]['track']:
    artist_names.append(item['artist']['name'])
    track_names.append(item['name'])
    play_counts.append(item['playcount'])

top_tracks = pd.DataFrame()
top_tracks['artist'] = artist_names
top_tracks['track'] = track_names
top_tracks['play_count'] = play_counts
top_tracks.to_csv('data/lastfm_top_tracks.csv', index=None, encoding='utf-8')
top_tracks.head()
method = 'topartists'
request_url = url.format(method, username, key, limit, extended, page)
artist_names = []
play_counts = []
response = requests.get(request_url).json()
for item in response[method]['artist']:
    artist_names.append(item['name'])
    play_counts.append(item['playcount'])

top_artists = pd.DataFrame()
top_artists['artist'] = artist_names
top_artists['play_count'] = play_counts
top_artists.to_csv('data/lastfm_top_artists.csv', index=None, encoding='utf-8')
top_artists.head()
method = 'topalbums'
request_url = url.format(method, username, key, limit, extended, page)
artist_names = []
album_names = []
play_counts = []
response = requests.get(request_url).json()
for item in response[method]['album']:
    artist_names.append(item['artist']['name'])
    album_names.append(item['name'])
    play_counts.append(item['playcount'])

top_albums = pd.DataFrame()
top_albums['artist'] = artist_names
top_albums['album'] = album_names
top_albums['play_count'] = play_counts
top_albums.to_csv('data/lastfm_top_albums.csv', index=None, encoding='utf-8')
top_albums.head()<jupyter_output><empty_output><jupyter_text>Now get all your scrobbles Last.fm provides this 'recenttracks' API method to get 'all' scrobbles. However, it seems to be pretty spotty for data from circa 2007. The best way to determine top tracks, artists, albums is with the cells above. However, the code below retrieves time series data of all scrobbles (but with the caveat of spotty data from 2007 and earlier).Sample URL: https://ws.audioscrobbler.com/2.0/?method=user.getrecenttracks&user=gboeing&api_key={}&limit=1&extended=0&page=1&format=json<jupyter_code>def get_scrobbles(method='recenttracks', username=username, key=key, limit=200, extended=0, page=1, pages=0):
    '''
    method: api method
    username/key: api credentials
    limit: api lets you retrieve up to 200 records per call
    extended: api lets you retrieve extended data for each track, 0=no, 1=yes
    page: page of results to start retrieving at
    pages: how many pages of results to retrieve. if 0, get as many as api can return.
    '''
    # initialize url and lists to contain response fields
    url = 'https://ws.audioscrobbler.com/2.0/?method=user.get{}&user={}&api_key={}&limit={}&extended={}&page={}&format=json'
    responses = []
    artist_names = []
    artist_mbids = []
    album_names = []
    album_mbids = []
    track_names = []
    track_mbids = []
    timestamps = []
    
    # make first request, just to get the total number of pages
    request_url = url.format(method, username, key, limit, extended, page)
    response = requests.get(request_url).json()
    total_pages = int(response[method]['@attr']['totalPages'])
    if pages > 0:
        total_pages = min([total_pages, pages])
        
    print('{} total pages to retrieve'.format(total_pages))
    
    # request each page of data one at a time
    for page in range(1, int(total_pages) + 1, 1):
        if page % 10 == 0: print(page, end=' ')
        time.sleep(pause_duration)
        request_url = url.format(method, username, key, limit, extended, page)
        responses.append(requests.get(request_url))
    
    # parse the fields out of each scrobble in each page (aka response) of scrobbles
    for response in responses:
        scrobbles = response.json()
        for scrobble in scrobbles[method]['track']:
            # only retain completed scrobbles (aka, with timestamp and not 'now playing')
            if 'date' in scrobble.keys():
                artist_names.append(scrobble['artist']['#text'])
                artist_mbids.append(scrobble['artist']['mbid'])
                album_names.append(scrobble['album']['#text'])
                album_mbids.append(scrobble['album']['mbid'])
                track_names.append(scrobble['name'])
                track_mbids.append(scrobble['mbid'])
                timestamps.append(scrobble['date']['uts'])
                
    # create and populate a dataframe to contain the data
    df = pd.DataFrame()
    df['artist'] = artist_names
    df['artist_mbid'] = artist_mbids
    df['album'] = album_names
    df['album_mbid'] = album_mbids
    df['track'] = track_names
    df['track_mbid'] = track_mbids
    df['timestamp'] = timestamps
    df['datetime'] = pd.to_datetime(df['timestamp'].astype(int), unit='s')
    
    return df
# get all scrobbled tracks ever, in order of recency (pages=0 to get all)
scrobbles = get_scrobbles(pages=0)
# save the dataset
scrobbles.to_csv('data/lastfm_scrobbles.csv', index=None, encoding='utf-8')
print('{:,} total rows'.format(len(scrobbles)))
scrobbles.head()<jupyter_output>10,684 total rows
{'fraction_non_alphanumeric': 0.1284796573875803, 'fraction_numerical': 0.009279086366880799, 'mean_word_length': 4.4734375, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '121830', 'license': 'MIT', 'n_tokens_mistral': 2271, 'n_tokens_neox': 2201, 'n_words': 574, 'path': 'last_fm/lastfm_downloader.ipynb', 'repo_name': 'Zackhardtoname/qs_ledger', 'size': '7011'}