<jupyter_start><jupyter_text>1. What is an API?**API** stands for **Application Programming Interface**, it defines interactions between multiple software components.An API simplifies programming by abstracting the underlying implementation by only exposing functions a developer might actually need. It can thus also hide informations from developers.On one hand it can hide functions a outside developer shall have no access to, on the other hand it can hide multiple complicated functions inside one simple API call. *There are multiple kinds of APIs*- **Libraries**:    - Libraries we already got to know are basically APIs, since they abstract and simplify the underlying implementation making it really simple to use them.        - Examples: `pandas`, `matplotlib`, `sklearn`        - E.g. You can find the `sklearn` API Reference at: https://scikit-learn.org/stable/modules/classes.html- **Interface between programs**:    - An API might also act as a middleman between two programs.    - E.g. you want to use a library in Python which was implemented for R. An API might translate the Python commands into R to make the library run.- **In Operating Systems**    - Abstract the underlying implementation of the Operating System to make sure the outcome is generic.        - E.g. POSIX is an API standard utilized (mostly) by UNIX systems such as Linux, BSD, macOS        - It makes sure POSIX-conform API calls work the same way on each Operating System        - E.g. the `echo` command shall work the same way independent of the Operating System you use    - Backwards Compatibility:        - E.g. translating old system calls into those utilized in newer OS versions.        - Examples: Win32API (allowing old Windows Programs to be run in `Compatibility Mode` under new versions), being able to play original XBOX/Playstation games on the current Console iteration- **Web API**    - Allow to access functions through Web Protocols, such as HTTP (Hypertext Transfer Protocol)    - API defines endpoints (e.g. `api.twitter.com/2/users/by/username/`) to fetch or push data, or even trigger code execution    - Responses are usually defined in `JSON` (JavaScript Object Notation) or `XML` (Extensible Markup Language) (somewhat similar to Python dictionaries)  2. Example: Getting to know the Twitter Web APITwitter offers an API allowing developers to easily extract and push data from/to Twitter.To get access you need to register as a developer at https://developer.twitter.com/ and apply for API acess._Note: Not needed for this lecture as we only want to show an example of an API in the real world. You can find the data we will extract in `../data/twitter.p`.__Although we absolutely recommend to register for any API access in the wild to play around. Good alternatives to Twitter would be any major site that you are well familiar with. E.g. Google, YouTube, Spotify, Facebook_You can look up the possible API commands at https://developer.twitter.com/en/docs/twitter-api  Requesting Barack Obamas Twitter Profile:You can retrieve basic information about Twitter Users using the following API endpoint: `https://api.twitter.com/2/users/by/username/`>```python>import pandas as pd>import matplotlib.pyplot as plt>import requests>> the Twitter API endpoint>twitter = "https://api.twitter.com/2/"> Include your API token into the HTTP Header>headers = {"Authorization": "Bearer "}> Send a HTTP-GET Request to retrieve the user "BarackObama">resp = requests.get(twitter + "users/by/username/BarackObama", headers=headers)>print(resp.json())>``` Output>```python>{'data': {'id': '813286', 'name': 'Barack Obama', 'username': 'BarackObama'}}```<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import requests

# the Twitter API endpoint
twitter = "https://api.twitter.com/2/"
# Include your API token into the HTTP Header
headers = {"Authorization": "Bearer AAAAAAAAAAAAAAAAAAAAAGFXMwEAAAAAsIEoxSy%2B%2BkvkOY2q6%2Fi2KFPLm7Q%3DtZ5Ul8yDXkDOpVhLoR5Iw1AjP7Wr8sc3jEVhrzuvnZaCLB4tm8"}
# Send a HTTP-GET Request to retrieve the user "BarackObama"
resp = requests.get(twitter + "users/by/username/BarackObama", headers=headers)
print(resp.json())<jupyter_output>{'data': {'id': '813286', 'name': 'Barack Obama', 'username': 'BarackObama'}}<jupyter_text>Requesting the Top Ten most followed Twitter usersBesides the generic retrieval per user, the API also allows to pass queries for lists of users.`https://api.twitter.com/2/users/by?usernames=,,`<jupyter_code>resp = requests.get(twitter + "users/by?usernames=BarackObama,justinbieber,katyperry", headers=headers)
print(resp.json())<jupyter_output>{'data': [{'id': '813286', 'name': 'Barack Obama', 'username': 'BarackObama'}, {'id': '27260086', 'name': '<NAME>', 'username': 'justinbieber'}, {'id': '21447363', 'name': 'KATY PERRY', 'username': 'katyperry'}]}<jupyter_text>It also allows to retrieve more than the three default fields (`id, name, username`) by requesting a key value pair by adding `&key=value` at the end of the request.Examples:| key | value | returned fields || --- | --- | --- || `user.fields` | `created_at` | `user.created_at` || `expansions` | `pinned_tweet_id` | `tweet.id`, `tweet.text` || `tweet.fields` | `created_at` | `includes.users.created_at` |Thus requesting `https://api.twitter.com/2/users/by?usernames=BarackObama&user.fields=created_at&expansions=pinned_tweet_id` will additionally return the data Obamas Account has been created at and the id of his currently pinned Tweet.<jupyter_code>resp = requests.get(twitter + "users/by?usernames=BarackObama,justinbieber,katyperry&user.fields=created_at&expansions=pinned_tweet_id", headers=headers)
print(resp.json())<jupyter_output>{'data': [{'username': 'BarackObama', 'created_at': '2007-03-05T22:08:25.000Z', 'name': 'Barack Obama', 'id': '813286'}, {'username': 'justinbieber', 'created_at': '2009-03-28T16:41:22.000Z', 'name': '<NAME>', 'pinned_tweet_id': '1344890442576490496', 'id': '27260086'}, {'username': 'katyperry', 'created_at': '2009-02-20T23:45:56.000Z', 'name': '<NAME>', 'pinned_tweet_id': '1299195688719249409', 'id': '21447363'}], 'includes': {'tweets': [{'id': '1344890442576490496', 'text': '#ANYONE out now\nhttps://t.co/Uh4E6vUzZv https://t.co/UjiStYfwrz'}, {'id': '1299195688719249409', 'text': 'IT’S HERE! IT’S REALLY HERE! 🙃 I finally got back my smile! Hope this record puts one on your face 🙂 #SMILE 🙂 IS OUT EVERYWHERE NOW! LOVE YOU GUYS SO MUCH ENJOY 🤡♥️ (sent from my hospital bed lol) https://t.co/BImXF3kEcw https://t.co/2UmVajDoyn'}]}}<jupyter_text>Lets try it out for the top 10 most followed Twitter users:>```python    most_followed_users = [                "BarackObama",                "justinbieber",                "katyperry",                "rihanna",                "Cristiano",                "taylorswift13",                "ladygaga",                "ArianaGrande",                "TheEllenShow",                "YouTube"                ]    most_followed_users_str = ",".join(most_followed_users)    resp = requests.get(twitter + f"users/by?usernames={most_followed_users_string}", headers=headers)>```<jupyter_code>most_followed_users = [
            "BarackObama",
            "justinbieber",
            "katyperry",
            "rihanna",
            "Cristiano",
            "taylorswift13",
            "ladygaga",
            "ArianaGrande",
            "TheEllenShow",
            "YouTube"
            ]
most_followed_users_str = ",".join(most_followed_users)
resp = requests.get(twitter + f"users/by?usernames={most_followed_users_str}&user.fields=created_at&expansions=pinned_tweet_id", headers=headers)
resp.json()<jupyter_output><empty_output><jupyter_text>Output>```python{'data': [      {'id': '813286',       'username': 'BarackObama',       'name': '<NAME>',       'created_at': '2007-03-05T22:08:25.000Z'}       ,      {'pinned_tweet_id': '1344890442576490496',       'id': '27260086',       'username': 'justinbieber',       'name': '<NAME>',       'created_at': '2009-03-28T16:41:22.000Z'}       ,      {'pinned_tweet_id': '1299195688719249409',       'id': '21447363',       'username': 'katyperry',       'name': '<NAME>',       'created_at': '2009-02-20T23:45:56.000Z'}       ,      {'id': '79293791',       'username': 'rihanna',       'name': 'Rihanna',       'created_at': '2009-10-02T21:37:33.000Z'}       ,      {'id': '155659213',       'username': 'Cristiano',       'name': '<NAME>',       'created_at': '2010-06-14T19:09:20.000Z'}       ,      {'pinned_tweet_id': '1360093736932429828',       'id': '17919972',       'username': 'taylorswift13',       'name': '<NAME>',       'created_at': '2008-12-06T10:10:54.000Z'}       ,      {'pinned_tweet_id': '1266218931028549632',       'id': '14230524',       'username': 'ladygaga',       'name': '<NAME>',       'created_at': '2008-03-26T22:37:48.000Z'}       ,      {'pinned_tweet_id': '1322025587368742912',       'id': '34507480',       'username': 'ArianaGrande',       'name': '<NAME>',       'created_at': '2009-04-23T02:56:31.000Z'}       ,      {'pinned_tweet_id': '1260696876149506048',       'id': '15846407',       'username': 'TheEllenShow',       'name': '<NAME>',       'created_at': '2008-08-14T03:50:42.000Z'}       ,      {'id': '10228272',       'username': 'YouTube',       'name': 'YouTube',       'created_at': '2007-11-13T21:43:46.000Z'}]       ,     'includes': {'tweets': [       {'id': '1344890442576490496',        'text': 'ANYONE out now\nhttps://t.co/Uh4E6vUzZv https://t.co/UjiStYfwrz'},       {'id': '1299195688719249409',        'text': 'IT’S HERE! IT’S REALLY HERE! 🙃 I finally got back my smile! Hope this record puts one on your face 🙂 SMILE 🙂 IS OUT EVERYWHERE NOW! LOVE YOU GUYS SO MUCH ENJOY 🤡♥️ (sent from my hospital bed lol) https://t.co/BImXF3kEcw https://t.co/2UmVajDoyn'},       {'id': '1360093736932429828',        'text': 'My new version of Love Story (Taylor’s Version) is out now  💛💛 Get it instantly when you pre-order Fearless (Taylor’s Version)  https://t.co/NqBDS6cGFl https://t.co/KdHdZXnWbP'},       {'id': '1266218931028549632',        'text': 'Now dance motherf💕ckers!!!!!!! Chromatica https://t.co/GjJUC3PRWz'},       {'id': '1322025587368742912',        'text': '🤍 positions (the album) is out now 🤍 https://t.co/FpkiHYLFqt https://t.co/J33o6KMTmo'}]},     'errors': [       {'detail': 'Could not find tweet with pinned_tweet_id: [1260696876149506048].',       'title': 'Not Found Error',       'resource_type': 'tweet',       'parameter': 'pinned_tweet_id',       'value': '1260696876149506048',       'type': 'https://api.twitter.com/2/problems/resource-not-found'}]}```<jupyter_code>a = 1
f"Hello {a}"<jupyter_output><empty_output><jupyter_text>3. Analyzing the dataWe extracted for you the user- and most recent tweet- (last 7 days) data. For both we only used the `created_at` and `public_metrics` queries. >```pythontweet_dict = {}for user in most_followed_users:    resp = requests.get(twitter + f"tweets/search/recent?query=from:{user}&tweet.fields=public_metrics,created_at", headers=headers, proxies=proxies)    tweet_dict[user] = resp.json()>```>```pythonuser_dict = {}for user in most_followed_users:    resp = requests.get(twitter + f"users/by?usernames={user}&user.fields=public_metrics,created_at", headers=headers, proxies=proxies)    user_dict[user] = resp.json()>```>```python> this is the data made available to youtwitter_data = {    'user_info': user_dict,    'tweets': tweet_dict}>```Lets see how much we can find out about their Twitter behavior just from requesting these two keys through the API!<jupyter_code>tweet_dict = {}
for user in most_followed_users:
    resp = requests.get(twitter + f"tweets/search/recent?query=from:{user}&tweet.fields=public_metrics,created_at", headers=headers)
    tweet_dict[user] = resp.json()

user_dict = {}
for user in most_followed_users:
    resp = requests.get(twitter + f"users/by?usernames={user}&user.fields=public_metrics,created_at", headers=headers)
    user_dict[user] = resp.json()

# this is the data made available to you
twitter_data = {
    'user_info': user_dict,
    'tweets': tweet_dict
}
# load default libraries
import numpy as np
import pandas as pd
import pickle

# load the data
twitter_data = pickle.load( open( "../data/twitter.p", "rb" ) )<jupyter_output><empty_output><jupyter_text>3.1 Inspect the dictionary<jupyter_code>twitter_data['user_info'].keys()
twitter_data['user_info']['BarackObama'].keys()
#twitter_data['user_info']['BarackObama']['data'][0]<jupyter_output><empty_output><jupyter_text>The dictionary consists of two parts:- `user_info` containing the metadata (num followers, etc.) of each of the top ten users- `tweets` containing tweets of the corresponding users and its metadata (num likes, etc.)Each subdictionary contains for each user key a list of dictionaries hidden behind the `data` key  3.2 Building a DataFrameTo better access individual values we want to turn the dictionary into a DataFrame.<jupyter_code>twitter_data['user_info'].keys()
df = pd.DataFrame()
for user in twitter_data['user_info'].keys():
    # user data
    data = twitter_data['user_info'][user]['data'][0]
    # public metrics is a dictionary inside user data
    # with dict.pop(key) we can remove a key from a dictionary
    # with dict.update(other_dict) we can merge two dictionaries together
    data.update(data.pop('public_metrics'))
    # when adding a row from a dictionary into a pandas DataFrame you need to pass 'ignore_index=True'
    # Note: pd.DataFrame.append() doesn't work inplace like the list.append()! 
    # Thus we need to overwrite our DataFrame with the appended one
    df = df.append(data, ignore_index = True)    
                
df
# Set the index to a meaningful unique identifier
df = df.set_index('username')
df<jupyter_output><empty_output><jupyter_text>3.3 Analyze TweetsWe got the user metadata into a DataFrame, but we're still left with the tweets part of the dictionary untouched.Lets do something with it ;-) The tweet subdictionary contains all(*) Tweets a user has posted in the last 7 days (Fetched at 18.02.2021).(*) _Note: Twitter only allows to fetch 100 Tweets at once. Which is why the number of Tweets the "YouTube" Account has posted is incorrect_  3.3.1. Number of Tweets posted> **Task:** Insert the number of Tweets posted by each user in the last 7 days into our DataFrame.<jupyter_code>## your code here 
# %load ../src/_solutions/number_of_tweets.py
# Build a dictionary containing ("User": "number_of_tweets") pairs
n_recent_tweets = {}

for user in twitter_data['tweets'].keys():
    if 'data' in twitter_data['tweets'][user].keys():
        n_recent_tweets[user] = len(twitter_data['tweets'][user]['data'])
    else:
        # <NAME> seems to be inactive for at least the last 7 days
        n_recent_tweets[user] = 0

# our dictionary makes it simple to put our fresh extracted data into our DataFrame
# Since the DataFrame Index and the dictionary keys fit
df['n_recent_tweets'] = pd.Series(n_recent_tweets)
df<jupyter_output><empty_output><jupyter_text>3.3.2 Most likes on a tweet> **Task:** Insert the number of most liked Tweets each user in the last 7 days into our DataFrame.<jupyter_code>## your code here 
# %load ../src/_solutions/max_number_of_tweets.py
# Build a dictionary containing ("User": "maximum likes on tweets") pairs
most_likes = {}

for user in twitter_data['tweets'].keys():
    maxx = 0
    if 'data' in twitter_data['tweets'][user]:
        for i in range(len(twitter_data['tweets'][user]['data'])):
            maxx = max(maxx, twitter_data['tweets'][user]['data'][i]['public_metrics']['like_count'])
    most_likes[user] = maxx
    
# our dictionary makes it simple to put our fresh extracted data into our DataFrame
# Since the DataFrame Index and the dictionary keys fit
df['most_likes'] = pd.Series(most_likes)
df<jupyter_output><empty_output><jupyter_text>At this point we should notice a pattern in the way our code finds out the maximum value for some kind of metric (keyword: code refactoring).> **Task:** Define a function which does the following:    > **INPUT**: metric, function (min, max, std, ...)  > **OUTPUT**: dictionary where for each user the function was applied to all values related to the metric<jupyter_code>## your code here 
# %load ../src/_solutions/tweets_metric_func.py
def tweets_metric_eval(metric, func):
    dict_metric = {}
    for user in twitter_data['tweets'].keys():
        val_list = []
        if 'data' in twitter_data['tweets'][user]:
            for tweet in twitter_data['tweets'][user]['data']:
                val_list.append(tweet['public_metrics'][metric])
        try:
            dict_metric[user] = func(val_list)
        except Exception as e:
            dict_metric[user] = np.nan
    return dict_metric<jupyter_output><empty_output><jupyter_text>Now, using the function, we can easily calculate the minimum/maximum/... values with respect to a metric. Try it out!> **Task:**  > - Insert for each user the maximum number of replys on a Tweet into our DataFrame.  > - Insert for each user the maximum number of retweets on a Tweet into our DataFrame. > - Insert for each user the minimal number of like_count on a Tweet into our DataFrame.  > - Insert for each user the avarage number of likes on a Tweet into our Dataframe.<jupyter_code>## your code here 
# %load ../src/_solutions/metric_tests.py
df['reply_count_max'] = pd.Series(tweets_metric_eval('reply_count', np.max))
df['like_count_min'] = pd.Series(tweets_metric_eval('like_count', np.min))
df['retweets_max'] = pd.Series(tweets_metric_eval('retweet_count', np.max))
df['like_count_std'] = pd.Series(tweets_metric_eval('like_count', np.std))
df<jupyter_output><empty_output><jupyter_text>3.4 Analyze DataFrameNow let's look at our built DataFrame and read the minimum or maximum values from a column.> **Task:**  > **Q1)** Who tweeted the most in the last 7 days?  > **Q2)** Who has the most likes on a tweet?  > **Q3)** Who has the most retweets on a tweet?<jupyter_code>## your code here 
# %load ../src/_solutions/max_tweeted.py
# %load ../src/_solutions/max_likes.py
# %load ../src/_solutions/max_retweets.py<jupyter_output><empty_output><jupyter_text>> **Task**: Compute the daily mean of posted tweets since the creation of the account for a single user.> _Hint: You can use `pd.datetime.now()` to retrieve the current date. The columns of interest are: `created_at` and `tweet_count`._<jupyter_code>df['created_at'] = pd.to_datetime(df['created_at']).dt.tz_convert(tz=None)
## your code here
# %load ../src/_solutions/daily_mean_posted_tweets.py<jupyter_output><empty_output><jupyter_text>4. Building our own API  FastAPIFastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.- Fast: Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic). One of the fastest Python frameworks available.- Fast to code: Increase the speed to develop features by about 200% to 300%. *- Fewer bugs: Reduce about 40% of human (developer) induced errors. *- Intuitive: Great editor support. Completion everywhere. Less time debugging.- Easy: Designed to be easy to use and learn. Less time reading docs.- Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.- Robust: Get production-ready code. With automatic interactive documentation.- Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as Swagger) and JSON Schema.Source: https://fastapi.tiangolo.com/  **Very simple example of an Web-API** **Underlying implementation**The underlying implementation consists of seperated functions.```pythonmodel = sklearn.linear_model.LinearRegression()xs, y, valid_xs, valid_y = train_test_split(dataset)def fit_model(model, xs, y):    return model.fit(xs,y)def predict(trained_model, xs):    return trained_model.predict(xs)def eval_prediction(y_orig, y_predicted):    return sklearn.metrics.mean_squared_error(y_orig, y_predicted)def something_secret():     do secret stuff``` **API**The Web-API allows by accessing a website at `/train_and_eval` to chain all of the above functions together to train and evaluate a model.It doesn't expose the `something_secret()` function to the outside world though!```python@api.get("/train_and_eval")def train_and_eval():    model_trained = fit_model(xs, y)    valid_y_pred = predict(model_trained, valid_xs)    rmse = eval_prediction(valid_y, valid_y_pred)        return {"rmse": rmse, "model_params": model_trained.get_params()}```  Lets try it out!We have implemented a very rudimentary API, which you can find in `../src/api_01.py`.The only thing it does is returning "Hello World" when accessing it.<jupyter_code># %load ../src/api_01.py
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def read_root():
    return {"message": "Hello World"}<jupyter_output><empty_output><jupyter_text>You can launch it the following way:- Open your Terminal (Linux, macOS) or Anaconda Prompt (Windows)- Activate class14 in `conda`- Head over to the `src` directory of todays lesson (use `cd`, `ls` and `pwd` to orientate yourself)- Run `uvicorn api_01:app --debug`- _Further Info_    - `uvicorn` is a  minimal low-level server/application interface allowing asynchronous executions    - Similarly to `jupyter lab` it hosts some kind of website on our host    - Syntax: `uvicorn :`    - The `--debug` flag makes sure the server restarts each time we change the source file. >After starting the server head over to https://127.0.0.1:8000/.>You should see the "Hello World" message>_Note: `127.0.0.1` is the same as `localhost` which is the IP-Address/Hostname of your own PC_ FastAPI also offers a Docpage at http://127.0.0.1:8000/docs which:- Shows you all available API commands- Allows you to experiment with them- Prints you the `curl` command you should use from the terminal to receive the same result as in the browser    - _Note: We limit ourselves to the HTTP-GET Method, which is used to retrieve data. (Your browser does per default) If we wanted to send something to the server we should use the HTTP-POST Method, which is easier to do with `curl` than manually through your browser._  Try something a bit more complex>**Goal**: Make a model accessible through an API  The Iris DatasetThe Iris Dataset is one of the most well known Classification datasets.Based on the length and width of the sepal and petal we want to classify iris flowers correctly.![_img/iris.png](_img/iris.png)Taken from [Machine Learning for Beginners](https://www.datacamp.com/community/tutorials/machine-learning-in-r)The sepal is the small/lower blossom, petal the upper/main blossom<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
iris = datasets.load_iris()
print(iris['DESCR'])
X = iris['data']
y = iris['target']
y
iris['target_names']
# Split dataset into training and test data
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)<jupyter_output><empty_output><jupyter_text>Lets train a Random Forest Classifier to classify our data.Reminder: Random Forests consist of multiple Decision Trees.<jupyter_code># n_jobs = -1 allows to use all our available computing power
rf = RandomForestClassifier(n_jobs=-1, random_state=1)
# train the Random Forest
rf.fit(x_train, y_train)
print(rf)
# Evaluate performance
accuracy_score(y_test, rf.predict(x_test))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12046511627906976, 'fraction_numerical': 0.03463002114164905, 'mean_word_length': 4.457083525611444, 'pattern_counts': {'":': 7, '<': 78, '<?xml version=': 0, '>': 123, 'https://': 25, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '282307', 'license': 'MIT', 'n_tokens_mistral': 7980, 'n_tokens_neox': 7305, 'n_words': 2411, 'path': '13-2021-xx-xx/notebooks/14-01-FastAPI.ipynb', 'repo_name': 'eotp/python-FU-class', 'size': '23724'}