<jupyter_start><jupyter_text>REST API for DataMartThis notebook showcases how to use the REST API for the DataMart system.For the augmentation, we use the FIFA 2018 Man of Match data, available here: https://gitlab.datadrivendiscovery.org/d3m/datasets/tree/master/seed_datasets_data_augmentation/DA_fifa2018_manofmatchThe documentation for the REST API is available here: https://docs.auctus.vida-nyu.org/rest/<jupyter_code>from io import BytesIO
import json
import os
import pandas as pd
from pprint import pprint
from pathlib import Path
import requests
import zipfile
def print_results(results):
    if not results:
        return
    for result in results:
        print(result['metadata']['name'])
        print('Score: ', result['score'])
        if 'augmentation' in result:
            aug_type = result['augmentation']['type']
            print('Augmentation: %s' % aug_type)
            print("Left Columns: %s" %
                  str(result['augmentation']['left_columns_names']))
            print("Right Columns: %s" %
                  str(result['augmentation']['right_columns_names']))
            
        print("-------------------")<jupyter_output><empty_output><jupyter_text>Initially, we have the supplied data.<jupyter_code># You can change this accordingly
fifa_manofmatch_dir = os.path.expanduser('~/projects/d3m/datasets/seed_datasets_data_augmentation/DA_fifa2018_manofmatch/DA_fifa2018_manofmatch_dataset')
fifa_manofmatch_file = os.path.join(fifa_manofmatch_dir, 'tables', 'learningData.csv')
fifa_manofmatch_table = pd.read_csv(fifa_manofmatch_file)
fifa_manofmatch_table.head()<jupyter_output><empty_output><jupyter_text>Searching for DatasetsLet's use DataMart to search for datasets that can be used to augment the supplied one.<jupyter_code>URL = 'https://auctus.vida-nyu.org/api/v1'
url = URL + '/search'

# http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
with open(fifa_manofmatch_file, 'rb') as data:
    response = requests.post(
        url,
        files={
            'data': data,
        }
    )
if response.status_code == 400:
    try:
        print('Error: %s' % response.json()['error'])
    except (KeyError, ValueError):
        pass
response.raise_for_status()
query_results = response.json()['results']
print_results(query_results)<jupyter_output>Water Consumption And Cost (2013 - March 2019)
Score:  1.0
Augmentation: join
Left Columns: [['Date']]
Right Columns: [['Revenue Month']]
-------------------
Board of Standards and Appeals (BSA) Applications Status
Score:  0.1
Augmentation: union
Left Columns: [['Date'], ['Blocked']]
Right Columns: [['Date'], ['Block']]
-------------------
Street Construction Permits
Score:  1.0
Augmentation: join
Left Columns: [['Date']]
Right Columns: [['ModifiedOn']]
-------------------
FIFA 2018 game statistics data
Score:  0.08125
Augmentation: union
Left Columns: [['GameID'], ['Off-Target']]
Right Columns: [['GameID'], ['On-Target']]
-------------------
Housing New York Units by Building
Score:  1.0
Augmentation: join
Left Columns: [['Date']]
Right Columns: [['Project Start Date']]
-------------------
Housing Maintenance Code Complaints
Score:  1.0
Augmentation: join
Left Columns: [['Date']]
Right Columns: [['ReceivedDate']]
-------------------
Capital Projects
Score:  1.0
Augmentation: join
Left[...]<jupyter_text>It is also possible to specify which column will be used for augmentation.<jupyter_code>url = URL + '/search'
query={
    'variables': [
        {
            'type': 'tabular_variable',
            'columns': [1],  # GameID
            'relationship': 'contains'
        }
    ]
}

# http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
with open(fifa_manofmatch_file, 'rb') as data:
    response = requests.post(
        url,
        files={
            'data': data,
            'query': ('query.json', json.dumps(query), 'application/json'),
        }
    )
if response.status_code == 400:
    try:
        print('Error: %s' % response.json()['error'])
    except (KeyError, ValueError):
        pass
response.raise_for_status()
gameID_results = response.json()['results']
print_results(gameID_results)<jupyter_output>FIFA 2018 game statistics data
Score:  0.98275864
Augmentation: join
Left Columns: [['GameID']]
Right Columns: [['GameID']]
-------------------<jupyter_text>Augmenting a Dataset Let's try to do our augmentation for the previous first query result then.<jupyter_code>url = URL + '/augment'
task = gameID_results[0]

# http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
with open(fifa_manofmatch_file, 'rb') as data:
    response = requests.post(
        url,
        files={
            'data': data,
            'task': ('task.json', json.dumps(task), 'application/json'),
        },
        stream=True,
    )
if response.status_code == 400:
    try:
        print('Error: %s' % response.json()['error'])
    except (KeyError, ValueError):
        pass
response.raise_for_status()
zip_ = zipfile.ZipFile(BytesIO(response.content), 'r')
learning_data = pd.read_csv(zip_.open('tables/learningData.csv'))
dataset_doc = json.load(zip_.open('datasetDoc.json'))
zip_.close()
learning_data.head()<jupyter_output><empty_output><jupyter_text>And we have our augmented data! Its corresponding datasetDoc JSON object is presented below.However, note that this datasetDoc JSON **does not** preserve the information from the supplied data's datasetDoc JSON. You need to use the Python DataMart API for that: https://gitlab.com/datadrivendiscovery/datamart-api/blob/master/datamart.py<jupyter_code>pprint(dataset_doc, indent=2)<jupyter_output>{ 'about': { 'approximateSize': '59300 B',
             'datasetID': 'e8a407b75568422eb33bb0338f1b989b',
             'datasetName': 'e8a407b75568422eb33bb0338f1b989b',
             'datasetSchemaVersion': '3.2.0',
             'datasetVersion': '0.0',
             'license': 'unknown',
             'redacted': False},
  'dataResources': [ { 'columns': [ { 'colIndex': 0,
                                      'colName': 'd3mIndex',
                                      'colType': 'integer',
                                      'role': ['index']},
                                    { 'colIndex': 1,
                                      'colName': 'GameID',
                                      'colType': 'integer',
                                      'role': ['attribute']},
                                    { 'colIndex': 2,
                                      'colName': 'Date',
                                      'colType': 'dateTime',
                                      'rol[...]
{'fraction_non_alphanumeric': 0.14802631578947367, 'fraction_numerical': 0.019736842105263157, 'mean_word_length': 2.7515423443634326, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '377610', 'license': 'Apache-2.0', 'n_tokens_mistral': 2125, 'n_tokens_neox': 2004, 'n_words': 474, 'path': 'examples/rest-api-fifa2018_manofmatch.ipynb', 'repo_name': 'VIDA-NYU/auctus', 'size': '6688'}