<jupyter_start><jupyter_text>ISI Datamart Demonstration---This demonstration illustrates the following capabilities:- Entity linking to  __[Wikidata](http://wikidata.org)__- Augmentation with data from the __[Wikidata](http://wikidata.org)__ knowledge graph- Augmentation with data from Excel, CSV and other structured sources- Augmentaiton with Wikipedia tables- Enriching __[Wikidata](http://wikidata.org)__<jupyter_code>import sys, os
# sys.stdout = open(os.devnull, 'w')
from wikifier import utils
from datamart.entries_new import D3MDatamart, D3MJoinSpec
import pandas as pd
d3mDatamart = D3MDatamart()
# this is our original input dataset
inputs_ds_loc = "/Users/pszekely/Downloads/datamart_demo/DA_poverty_estimation/TRAIN/dataset_TRAIN/datasetDoc.json"
# pd.set_option('display.max_columns', None)
original_dataset = utils.load_d3m_dataset(inputs_ds_loc)<jupyter_output>Using TensorFlow backend.
/Users/minazuki/miniconda3/envs/datamart/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)<jupyter_text>Our Original Dataset---We start with a dataset that has the number of people in poverty in different counties in the United States. In this demo we are using data from Florida and Georgia only (so it runs faster).<jupyter_code>#original_dataset['learningData'].head()
original_dataset['learningData'].sample(n=15)<jupyter_output><empty_output><jupyter_text>Linking To Wikidata---Wikidata contains over 80 million identifiers for entities. Datamart can scan a dataset, automatically identify columns containing entity identifiers, and link the identifiers to the appropriate entity in Wikidata.Here is the wikified data. Clicking on the links takes you to the corresponding wikidata pages where you can see all the data available for each entity.<jupyter_code>wikified_dataset = utils.wikifier_for_d3m_all(input_ds=original_dataset).value
# wikified_dataset['learningData']
utils.pretty_print(wikified_dataset,"wikifier")<jupyter_output><empty_output><jupyter_text>Searching Datamart Using Our Wikified Data---Datamart finds multiple datasets that can be used to augment the poverty dataset. The results show the title, the columns available in each dataset, and the columns that will be used to join the candidate dataset to the poverty dataset.<jupyter_code>search_results = d3mDatamart.search_with_data(supplied_data=wikified_dataset)
# wiki_search_results.display()
utils.print_search_results(search_results)<jupyter_output><empty_output><jupyter_text>Using Search Results To Augment Your Data---The first search result, from Wikidata, augments our data using population, area and inception date of counties.<jupyter_code>wiki_search_result = search_results[0]
augmented_dataset = wiki_search_result.augment(supplied_data=wikified_dataset)
utils.pretty_print(augmented_dataset,"wiki_augment")<jupyter_output><empty_output><jupyter_text>Repeat Augmentation With Additional Search Results From Wikidata---The second search result, also from Wikidata, augments our using data from states. This adds columns with information about the states.<jupyter_code>augmented_dataset = search_results[1].augment(supplied_data=augmented_dataset)
utils.pretty_print(augmented_dataset,"wiki_augment")<jupyter_output><empty_output><jupyter_text>Download Data---Some of the search results are datasets indexed from the web, they are present in the search results because they can be joined with our wikified dataset.Let's inspect a few of these datasets. __The 6th search result contains poverty information.__<jupyter_code>downloaded_dataset = search_results[6].download(supplied_data=wikified_dataset)
utils.pretty_print(downloaded_dataset)
# utils.pretty_print(downloaded_dataset,"download")<jupyter_output><empty_output><jupyter_text>__The 7th search result contains unemployment information.__<jupyter_code># %%script false

downloaded_dataset = search_results[7].download(supplied_data=wikified_dataset)
# utils.pretty_print(downloaded_dataset)
utils.pretty_print(downloaded_dataset,"download")<jupyter_output><empty_output><jupyter_text>Augment With Datasets From The Web---Datasets from the web can also be used to augment our original data.Let's augment using the poverty data as it is useful to predict the number of people in poverty. Many new columns appear at the end.<jupyter_code>augmented_dataset = search_results[6].augment(supplied_data=augmented_dataset)
utils.pretty_print(augmented_dataset,"wiki_augment")<jupyter_output><empty_output><jupyter_text>Discovering And Using More Data---Crime data may be useful to predict poverty, but no crime data is currently available in Datamart.Searching in Google for `fbi crime statistics by county`  produces this search result:[ ](https://ucr.fbi.gov/crime-in-the-u.s)After navigating to this page, click on `2016`, then `Crime in the U.S. 2016`, then `Violent Crime`. You can explore the various crime datasets. Let's choose `Table 8`, which has crime data for all states, broken doown by county. For example, the Georgia [page](https://ucr.fbi.gov/crime-in-the-u.s/2016/crime-in-the-u.s.-2016/tables/table-8/table-8-state-cuts/georgia.xls) contains crime data for counties in Georgia.This crime data can be downloaded in Excel using the `Download Excel` [link](https://ucr.fbi.gov/crime-in-the-u.s/2016/crime-in-the-u.s.-2016/tables/table-8/table-8-state-cuts/georgia.xls/output.xls).---Challenges for using this data:- The data for each state is in a separate file- The column headers start in row 6- The spreadsheet has notes at the end, and the notes start in different rows for different states- The name of the state and the year are in the metadata rows (rows 2 and 4)  Automatic Table Understanding (Poster)---The automatic table understanding software performs the following tasks on spreadsheets and CSV files:- Identifies the type of each cell (data, header, attribute, global metadata)- Segments the table into blocks- Identifies relationships among blocks__Add image here with blocks of GEORGIA table__  Augmenting Wikidata With Data Extracted From Tables--- After the table understanding step, the data can be indexed in Datamart and used for augmentation. The challenge with the FBI crime data is that the data for each state is in a separate file. Augmentation of our original dataset requires combining the data from multiple files.Datamart addresses this challenge by mapping the table data to Wikidata and uploading the data to Datamart's Wikidata clone where it can be queried regardless of the file where it came from.  Adding Crime Properties To WikidataWikidata provides a [user interface](https://test.wikidata.org/wiki/Special:NewProperty) to define properties.We defined [properties to represent crime data](http://tinyurl.com/y5g7juu6).  Download The FBI Crime Data<jupyter_code># download_fbi_crime_data("Georgia", "Florida")
# this script should print the URL of each downloaded file.
utils.download_FBI_data(["Georgia", "Florida"])<jupyter_output><empty_output><jupyter_text>Use DIG To Map The Spreadsheets To WikidataA DIG script converts the spreadsheet data to Wikidata using a simple API for augmenting Wikidata.<jupyter_code># extract_fbi_crime_data_to_wikidata("Georgia", "Florida")
# this script should print a line after processing each state, like the following
# Generated Wikidata RDF triples for Georgia
utils.generate_FBI_data(["Georgia", "Florida"])<jupyter_output><empty_output><jupyter_text>Upload The RDF Triples To Wikidata<jupyter_code>utils.upload_FBI_data(["Georgia", "Florida"])<jupyter_output><empty_output><jupyter_text>The uploaded FBI data can be visualized, taking advantage of latitude/longitude coordinates present in Wikidata:- [Map ](http://tinyurl.com/y2a7b7a2) of crime data by county, colored by file where data was present- [Map ](http://tinyurl.com/y2k66mcd) of crime data by county colored by severity - [Map ](http://tinyurl.com/yxwh24vr) of crime data by county colored by severity, per 100,000 inhabitants   Search Datamart Again---The FBI data is now available as new columns to augment the poverty data.<jupyter_code>search_results = d3mDatamart.search_with_data(supplied_data=wikified_dataset)
# wiki_search_results.display()
utils.print_search_results(search_results)
## Augment Using The FBI Data
# The first search result has the FBI data.
wiki_search_result = search_results[0]
fbi_augmented_dataset = wiki_search_result.augment(supplied_data=wikified_dataset)
utils.pretty_print(fbi_augmented_dataset,"wiki_augment")<jupyter_output><empty_output><jupyter_text>__Remove the FBI data from the Datamart Wikidata installation__<jupyter_code>utils.clean_FBI_data()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07763337893296854, 'fraction_numerical': 0.009119927040583675, 'mean_word_length': 6.743159752868491, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '540620', 'license': 'MIT', 'n_tokens_mistral': 2656, 'n_tokens_neox': 2520, 'n_words': 847, 'path': 'example/datamart_demo.ipynb', 'repo_name': 'xkgoodbest/wikidata-wikifier', 'size': '8772'}