<jupyter_start><jupyter_text>Interacting with the ESCALATE REST API This is a tutorail for using the ESCALATE REST API. It presumes only knowledge of basic python. It also presumes you have a local instance of ESCALATE running at http://localhost:8000<jupyter_code>import json
from pprint import pprint

import requests  # requests library will send and receive data from the escalate server
from requests.api import post

import pandas as pd<jupyter_output><empty_output><jupyter_text>Quick intro to REST APIsAll you need to know about REST is that:* It is a protocol for exchanging data between a client (e.g. me) and a server (e.g. ESCALATE)* Data formats are human *and* machine readable (XML or JSON)* There are two main HTTP 'verbs' or functions:   1. GET data from the server  2. POST data to the server       (there are others, e.g. PUT and PATCH, but we won't use these in this tutorial)* The python `requests` library implements the HTTP verbs for interacting with servers, including with REST APIs  Example: GET from the PubChem API* [Pubchem](https://pubchem.ncbi.nlm.nih.gov/) is a great way to get chemical informatics data from a vast repository   - e.g. just search for any compound above   - Has a graphical web interface and REST API* (There are other computational chemistry REST APIs, including [Open Chemistry](https://doi.org/10.1186/s13321-017-0241-z), and [AFLOW-ML](https://doi.org/10.1016/j.commatsci.2018.03.075))<jupyter_code># dict mapping compound name to PubChemID
PubChemIDs = {
    'Methane': '297',
    'Benzene': '241',
    'Ethylammonium Iodide': '11116533'
}<jupyter_output><empty_output><jupyter_text>The URL below is the compound 'API endpoint' which I can send a request to for properties of compounds given their PubChem CIDs<jupyter_code>response = requests.get(('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound'          # What type of entity? Compound
                         f'/cid/{",".join(PubChemIDs.values())}/'                      # Which compounds? These
                         'property/MolecularFormula,MolecularWeight,CanonicalSMILES/'  # Which properties? 
                         'JSON'))                                                     # Which format?
                        #proxies=proxies)
response
response.json()<jupyter_output><empty_output><jupyter_text>The ESCALATE REST API * Send data back and forth between ML, chemists, and laboratories  Interactive List of all available ESCALATE endpointsSimply navigating to http://localhost:8000/api will show you a browsable list of all available API endpoints. We can also view that list programmatically:<jupyter_code>base_url = 'http://localhost:8000/api'  # local dev server
response = requests.get(base_url)
response.json()<jupyter_output><empty_output><jupyter_text>Logging into ESCALATETo be able to view and post to all endpoints, I will create a log in session that is managed by a token.<jupyter_code># demo login credentials
login_data = {
    'username': 'mtynes',
    'password': '<PASSWORD>'
}

r_login = requests.post(f'{base_url}/login', data=login_data)
token = r_login.json()['token']
token
token_header = {'Authorization': f'Token {token}'}

content_type_header = {'content-type': 'application/json'} # for most requests we'll want this header<jupyter_output><empty_output><jupyter_text>This token will allow me to validate my identity for this session  Simple helper functions for GET/POSTThese functions will do some minimal URL generation and JSON response parsing on top of what is done by the `requests` library<jupyter_code>def patch_data(endpoint, data={}, headers={**token_header, **content_type_header}):
    """PATCH `data` to `endpoint`in ESCALATE API using `headers`
    
    return: (dict|requests.Response), bool
    """
    r = requests.patch(f'{base_url}/{endpoint}/', 
                      data=json.dumps(data), 
                      headers=headers)
    print(r)
    if r.ok: 
        print('POST: OK, returning new resource dict')
        return r.json()
    print('POST: FAILED, returning response object')
    return r


def post_data(endpoint, data={}, headers={**token_header, **content_type_header}):
    """POST `data` to `endpoint`in ESCALATE API using `headers`
    
    return: (dict|requests.Response), bool
    """
    r = requests.post(f'{base_url}/{endpoint}/', 
                      data=json.dumps(data), 
                      headers=headers)
    print(r)
    if r.ok: 
        print('POST: OK, returning new resource dict')
        return r.json()
    print('POST: FAILED, returning response object')
    return r


def get_data(endpoint, data={}, headers={**token_header}):
    """Make GET request with `data` to `endpoint` in ESCALATE API using `headers`
    
    return: (dict|list|requests.Response), bool
    """
    r = requests.get(f'{base_url}/{endpoint}/', params=data, headers=headers)
    print(r)
    if r.ok: 
        print('GET: OK')
        
        resp_json = r.json()        
        
        # handle cases: one vs many results
        if resp_json.get('count') is None: # edge case: template edit
            return r.json()
        elif resp_json.get('count') == 1: 
            print('Found one resource, returning dict')
            return resp_json['results'][0]
        elif resp_json.get('count') >= 1: 
            print(f"Found {resp_json['count']} resources, returning list of dicts)")
            return r.json()['results']
        else:
            print('GET: FAILED, returning response object')
    return r<jupyter_output><empty_output><jupyter_text>GET all of materials defined in ESCALATE<jupyter_code>r = post_data(endpoint='material', data={
    "description": "Ethylammonium Iodide",
    "consumable": True,
    "material_class": "model",
})
r = get_data(endpoint='material')<jupyter_output><Response [201]>
POST: OK, returning new resource dict
<Response [200]>
GET: OK
Found one resource, returning dict<jupyter_text>All of these materials would probably take up too much space in this notebook, lets look at the first one. All lead iodide shares in this chemical formula, etc. Models are jsut 'what is is about these thigns that are all the same, as opposed to particular vials of gblthat has volumes, contanimanants, dates, masses. these all come from the object, but share in model. models dont contain provenance, are only abotu intenseive properties<jupyter_code>r<jupyter_output><empty_output><jupyter_text>These are the fields available for materials. Notably we can associate a material with arbitrary properties and material types.  POST a material property Current property definitions in ESCALATEESCALATE supports user defined property definitions, these are the one's we're using<jupyter_code>r = post_data('property-def', {
    "description": "molecular weight",
    "property_def_class": 'intrinsic',
    "short_description": "mw",
    "val_unit": "g/mol",
    "unit_type": "num",
    "val_type": None
})
#print(r.json())
r = get_data('property-def', 
             {'fields': ['description']} # we can select 'columns' with the fields parameter    
            )
r<jupyter_output><Response [201]>
POST: OK, returning new resource dict
<Response [200]>
GET: OK
Found one resource, returning dict<jupyter_text>Molecular Weight<jupyter_code>mw_property_def = get_data('property-def', 
                           {'description': 'molecular weight', # find an entity that has a particular description
                            'fields': ['url', 'description', 'val_unit']})
mw_property_def<jupyter_output><Response [200]>
GET: OK
Found one resource, returning dict<jupyter_text>Now we can associate an instance of this property with a material, say EthNH3I<jupyter_code>ethylammonium_iodide = get_data('material', 
                                {'description': 'Ethylammonium Iodide',
                                 'fields':['url', 'uuid', 'description', 'property']})
ethylammonium_iodide<jupyter_output><Response [200]>
GET: OK
Found one resource, returning dict<jupyter_text>Note the empty property list aboveLets fill that in with the molecular weight from PubChem<jupyter_code>response = requests.get('https://pubchem.ncbi.nlm.nih.gov/rest/pug/'+\
                        f'compound/cid/{PubChemIDs.get("Ethylammonium Iodide")}/'+\
                        'property/MolecularFormula,MolecularWeight,CanonicalSMILES/JSON')
                      # proxies=proxies)
pubchem_json = response.json()
eth_mw = pubchem_json['PropertyTable']['Properties'][0]['MolecularWeight']
eth_mw
r = post_data(f'material/{ethylammonium_iodide["uuid"]}/property', {
    "property_val": {
        "value": eth_mw,
        "type": "num",
        "unit": "g/mol"
    },
    "property_def": mw_property_def['url'],
    "property_class": "nominal"
})
r
r.json()
r = post_data('materialproperty',
                  {'material': ethylammonium_iodide['url'],
                   'property_def': mw_property_def['url'],
                   'value': f"{eth_mw}"
                  }
                 )<jupyter_output><Response [404]>
POST: FAILED, returning response object<jupyter_text>And we've stored it!<jupyter_code>get_data('material', 
         {'description':'Ethylammonium Iodide'})<jupyter_output><Response [200]>
GET: OK
Found one resource, returning dict<jupyter_text>* In practice we can use this functionality to store properties from any experiment or calculation. * We can also store metadata about where these values came from     (example to come in a further tutorial on tags, notes, edocs, calculations).  Action definitions * Just as we are free to define properties, we are free to define actions* Current definition are what we've needed to specify human/robot instructions for current workflows<jupyter_code>r = post_data('parameter-def', {'description': 'volume', 'required': True,
                                'default_val': {'value': 0.0, 'unit': 'mL', 'type': 'num'},
                                'unit_type': 'volume',})
r

r = post_data('action-def', {"description": "dispense"})

r = get_data('action-def', {'fields': ['description', 'uuid', 'url']})
r<jupyter_output><Response [201]>
POST: OK, returning new resource dict
<Response [200]>
GET: OK
Found one resource, returning dict<jupyter_text>Zooming in on the dispense action definition<jupyter_code>r = get_data('action-def',               
         {'description': 'dispense',  # which action def
          'expand': 'parameter_def',  # sub dictionary to expand
          })
r<jupyter_output><Response [200]>
GET: OK
Found one resource, returning dict<jupyter_text>Actions + Materials = Experiment Template * Experiment template = the form of an experiment that I wish to re-use, varying material choices and process parameters<jupyter_code>experiment_templates = get_data('experimenttemplate', 
                               {'fields':['description', 'url']})
experiment_templates<jupyter_output><empty_output><jupyter_text>Click on perovskite demo link and note 2 main nested fields: * Bill of materials = the initial materials for the perovskite workflow* Workflow = the set of actions that combine these materials into perovskite crystal trials<jupyter_code>perovskite_template = get_data('experimenttemplate',
                              {'description': 'test_wf_1', 
                              'expand': 'workflow' # expand the workflow subdictionary
                              })
perovskite_template<jupyter_output><empty_output><jupyter_text>Bill of MaterialsThe initial materials: think list of materials in methods section of paper These are the Bill of Materials' Material entries for this experiment:<jupyter_code>get_data('bommaterial', {'bom':perovskite_template['bill_of_materials'][0]['uuid'], 
                        'fields':['description']})
get_data('compositematerial', {'composite_description__startswith':'Stock', 
                               'fields': ['composite_description',
                                          'component_description']})<jupyter_output><Response [200]>
GET: OK
Found 7 resources, returning list of dicts)<jupyter_text>We could drill further to get the concentrations, properties, etc  Workflows: Logical groups of actions Each of these contains a set of parameters that we can edit<jupyter_code>perovskite_demo = get_data('experimenttemplate',
         {'description': 'perovskite_demo',
          'expand': 'workflow'})
[wf['description'] for wf in perovskite_demo['workflow']]
dispense_solvent_wf = perovskite_demo['workflow'][3] # pull out a workflow
example_steps = dispense_solvent_wf['step'][4]     # pull out some steps
example_steps<jupyter_output><empty_output><jupyter_text>Creating a new workflow from a templateIf I want create an instance experiment from a template I:<jupyter_code>editable_template = get_data(
    # template endpoint  /        template ID      / create an instance of this template 
    f'experimenttemplate/{perovskite_demo["uuid"]}/create'
)
editable_template
editable_template['experiment_name'] = 'test_perovskite_instance_updated'<jupyter_output><empty_output><jupyter_text>Suppose I edit this json then I can post the new template to the server<jupyter_code>%%time
resp = post_data(
    f'experimenttemplate/{perovskite_demo["uuid"]}/create',
    editable_template
)
resp<jupyter_output><empty_output><jupyter_text>This experiment then appears in the experiment queue.   http://localhost:8000/experiment_listThe experimentalist is notified, can download relevant robot input and upload observed values through forms  Getting ML-ready data from the lab Once the experimentalist finishes their work, we can query the REST API for the results of the requested experiment:<jupyter_code>%%time
new_experiment_json = get_data('experiment/' + resp['new_experiment_created'].split('/')[-2], 
                                {'expand': 'workflow.step.workflow_object.action.parameter'}) # expanding deeply nested fields is somewhat slow
experiment_json = [new_experiment_json]<jupyter_output><empty_output><jupyter_text>The above json is quite long, so we won't render it here. Instead, we'll parse it into a pandas Dataframe with the function below.<jupyter_code>def experiment_json_to_df(experiment_json):
    result = []
    for e in experiment_json: 
        for workflow in e['workflow']:
            if 'Dispense' not in workflow['description']:
                    continue
            for step in workflow['step']: 
                action = step['workflow_object']['action']
                for parameter in action['parameter']:
                    result.append(
                        dict(
                            experiment_url         = e['url'],
                            experiment_id          = e['url'].split('/')[-2],
                            action_source          = action['source_material_description'],
                            action_dest            = action['destination_material_description'],
                            action_parameter       = parameter['parameter_def_description'],
                            action_parameter_value = float(parameter['parameter_val_nominal']['value']),
                            action_parameter_unit  = parameter['parameter_val_nominal']['unit']
                            )
                        )
    return pd.DataFrame(result)
results = experiment_json_to_df(experiment_json)
results<jupyter_output><empty_output><jupyter_text>From here, its just a quick pivot to transform the dataset into a format where each reaction well is a row. This is ideal for ML input.<jupyter_code>results = results.pivot_table(index=['experiment_id', 'action_dest'], 
                             columns=['action_source'], 
                             values='action_parameter_value')
results<jupyter_output><empty_output><jupyter_text>Of course, we also want the measured outcomes (human rated crystal scores) for each of these wells. Currently those are accessible through this endpoint:<jupyter_code>crystal_scores = get_data('measure', 
                          {'measuredef':
                                (get_data('measuredef', 
                                          {'description': 'crystal_score'})['url']
                                )
                          })                        
results['crystal_score'] = crystal_scores['measure_value']['value']
results<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10210687022900763, 'fraction_numerical': 0.007083969465648855, 'mean_word_length': 3.304942166140904, 'pattern_counts': {'":': 16, '<': 105, '<?xml version=': 0, '>': 106, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '328285', 'license': 'MIT', 'n_tokens_mistral': 4745, 'n_tokens_neox': 4442, 'n_words': 1391, 'path': 'demonstrations/REST_API_DEMO.ipynb', 'repo_name': 'darkreactions/ESCALATE', 'size': '16377'}