<jupyter_start><jupyter_text>Session 3 Homework Solution======================== Exercise 1 - Linear FittingUse scikitlearn to perform multiple linear regression for solubility using the dataset `delaney-processed.csv`. Unfortunately the dataset doesn't contain all of the molecular descriptors as described in the original paper. Use the available descriptors as independent variables in the linear fit. - Use your model to predict solubilities for the dataset.- Compute the $R^2$ statistic for the fit<jupyter_code>import pandas as pd
from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Loading and Transforming the Data<jupyter_code>df = pd.read_csv("data/delaney-processed.csv")
df.head()
# Get X and Y values as NumPy arrays
X = df.iloc[:, 2:8]
X.head()
Y = df["measured log solubility in mols per litre"]
Y.head()
X = X.to_numpy()
Y = Y.to_numpy()<jupyter_output><empty_output><jupyter_text>Fitting the linear model<jupyter_code>regression = LinearRegression().fit(X, Y)
regression.coef_<jupyter_output><empty_output><jupyter_text>Using the model to predict values<jupyter_code>predicted = regression.predict(X)
df["multiple_regression"] = predicted
df.head()<jupyter_output><empty_output><jupyter_text>Computing $r^2$ score<jupyter_code>from sklearn.metrics import r2_score
r2_score(Y, predicted)
help(r2_score)<jupyter_output>Help on function r2_score in module sklearn.metrics._regression:

r2_score(y_true, y_pred, *, sample_weight=None, multioutput='uniform_average')
    R^2 (coefficient of determination) regression score function.
    
    Best possible score is 1.0 and it can be negative (because the
    model can be arbitrarily worse). A constant model that always
    predicts the expected value of y, disregarding the input features,
    would get a R^2 score of 0.0.
    
    Read more in the :ref:`User Guide <r2_score>`.
    
    Parameters
    ----------
    y_true : array-like of shape (n_samples,) or (n_samples, n_outputs)
        Ground truth (correct) target values.
    
    y_pred : array-like of shape (n_samples,) or (n_samples, n_outputs)
        Estimated target values.
    
    sample_weight : array-like of shape (n_samples,), default=None
        Sample weights.
    
    multioutput : {'raw_values', 'uniform_average', 'variance_weighted'},             array-like of shape (n_outputs,) or None[...]<jupyter_text>Exercise 2 - Regular ExpressionsParse the file `SBE-b-CD-data.sdf` in your `data` folder using regular expressions. The SDF file contains information for 220 molecules. For each molecule, there is a section which looks like this:```>  (-)__Sulpiride>  293>  35```  Loading the dataFor this homework  thwe recommend you read in the file using `open` function. We recommend this because the file is irregular, it is not just a table. Additionally, the assignment is to use regular expressions which work on strings. When you use `with open` then use `.read` the entire file contents will be in one string. Do note that if your data is tabular and contains text, you can still [use regular expressions in pandas dataframes](https://pandas.pydata.org/docs/user_guide/text.html). The question of strategies for opening a file and the question of using regular expressions are not related. You should choose the best option to open a file based on the file structure, and the best option for text processing based on what you're looking for.Notably, there are other options you might use with `open`. You could also use `readlines` (will put each line into a list, or `readline` to read a single line at a time.<jupyter_code>import re
with open("data/SBE-b-CD-data.sdf") as f:
    data = f.read()
type(data)
# Preview the first 500 characters
print(data[:500])<jupyter_output>(-)__Sulpiride
  Marvin  01071110562D
 
 23 24  0  0  0  0  0  0  0  0999 V2000
    0.3200    0.2956    0.0000 C   0  0  3  0  0  0  0  0  0  0  0  0
    1.1450    0.2956    0.0000 C   0  0  3  0  0  0  0  0  0  0  0  0
    1.5575    1.0101    0.0000 C   0  0  3  0  0  0  0  0  0  0  0  0
    2.3824    1.0101    0.0000 C   0  0  0  0  0  0  0  0  0  0  0  0
    2.7949    0.2956    0.0000 C   0  0  0  0  0  0  0  0  0  0  0  0
    2.3824   -0.4189    0.0000 C   0  0  3  0  0  0  0  0  0  0  0  0<jupyter_text>Composing our regular expressionNow we will write our regular expression. To do a proof of concept on this, I would recommend constructing your regular expression using part of the file and [pythex](https://pythex.org/) as a proof of concept.Let's break down the rules for what we're looking for then translate that into regular expression characters.```>  (-)__Sulpiride>  293>  35```- Notice that these all follow the pattern of ``, a line break and then the value. Let's write a regular expression to get this kind of group. For `PROPERTY`, what we are looking for is mostly words, along with a _ for `Temperature_K`.Let's first write an expression for getting ``. We want characters and `_` to be between `` We could write `[A-Za-z]` to indicate any letter A-Z of any case. You can add a `_` to also have this be a valid character.``This pattern will not match anything because as written it will only look for a single letter between the the brackets. To say that there should be more than one character, add `+` to the end. Note that you could have also used `.` instead of `[A-Za-z_]` to indicate that you would accept *any* character. For our case, this will have an effect on our results. If we had numbers (like ``) the first pattern would not find the key, while the second would.<jupyter_code>pattern = re.compile('<[A-Za-z_]+>')
found = pattern.findall(data)

# Show first 20 found
found[:20]<jupyter_output><empty_output><jupyter_text>Next, we'll add in to look for a newline. On the following line, we would like to get the value. We'll specify this should be any character up to the next line (`.`). Recall that the period `.` represents any character except a line break. Use the plust modifier to indicate more than one character.<jupyter_code>pattern = re.compile('<[A-Za-z_]+>\n.+')
found = pattern.findall(data)
found[:20]<jupyter_output><empty_output><jupyter_text>Finally, we'll add parenthesis to group the parts we would like to keep. Put parenthesis around `[A-Za-z_]+` and `.+` to get the property name and value in different groups.<jupyter_code>pattern = re.compile('<([A-Za-z_]+)>\n(.+)')
found = pattern.findall(data)
found[:20]<jupyter_output><empty_output><jupyter_text>Exercise 3 - BonusThis exercise directs you how to retrieve papers from ChemRxiv (chemistry preprint server) using their REST API. Your task is then to do some processing in pandas to retrieve article abstracts, then to look for phrases in the abstract using regular expressions. This homework requires learning some extra material, so it is a bonus.First, you can use the Python `requests` module (part of the Python Standard Library) to query the rest API for ChemRxiv. This is the URL we will go to. If you visit this url in your browser, you will get a list of the 100 most recent papers uploaded to ChemRxiv.`https://api.figshare.com/v2/articles?institution=259&page_size=100`To retrieve this information using Python, do```pythonimport requestsresults = requests.get('https://api.figshare.com/v2/articles?institution=259&page_size=100')```The 'payload' of this is stored in `results.json()`. This is where the information which you see in your browser. You can convert what you've retrieved into a dataframe by doing `df = pd.DataFrame(results.json())`.You can retrieve the abstracts by calling `requests.get` on the `url` for each paper. Save this in a column called `detail`. This step will take a while to execute.After retrieving the details, you must get the json and retreive the "description" field. You can write a custom function which does this and apply it to the `detail` column.Next, use [pandas str contains](https://pandas.pydata.org/docs/reference/api/pandas.Series.str.contains.html) to search the abstract for phrases of interest. We suggest trying out machine/deep learning. Your results will vary since it will always grab the 100 most recently uploaded papers!<jupyter_code>import requests
results = requests.get('https://api.figshare.com/v2/articles?institution=259&page_size=100')
df = pd.DataFrame(results.json())
df.head()
df["detail"] = df["url"].apply(requests.get)
def get_abstract(paper_detail):
    abstract = paper_detail.json()["description"]
    return abstract
df["abstract"] = df["detail"].apply(get_abstract)
df["abstract"]
# use str.contains to find strings in df which contain phrase. Regular expressions work with this function.
ml_papers = df[df["abstract"].str.contains("(machine|deep) learning")]
ml_papers.info()
ml_papers["abstract"]
# Read more from one paper. Notice that the row name is preserved from the original dataframe.
# note that yours will be different because this code returns the 100 most recently uploaded papers to chemrxiv!
ml_papers["abstract"][37]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08862301982940068, 'fraction_numerical': 0.03002104796720948, 'mean_word_length': 4.236658932714617, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 49, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '605468', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2832, 'n_tokens_neox': 2639, 'n_words': 1158, 'path': 'book/3-homework-solution.ipynb', 'repo_name': 'janash/python-analysis', 'size': '9027'}