<jupyter_start><jupyter_text>A Machine Learning Approach for Classifying Sinopia's RDF<NAME> July 2020  SetupTo run the RDF Classify code in the Colab environment, we will first need to clone the https://github.com/ld4p/rdf-classify repository.<jupyter_code>! git clone https://github.com/LD4P/rdf-classify.git<jupyter_output>Cloning into 'rdf-classify'...
remote: Enumerating objects: 196, done.[K
remote: Counting objects: 100% (196/196), done.[K
remote: Compressing objects: 100% (107/107), done.[K
remote: Total 196 (delta 101), reused 156 (delta 72), pack-reused 0[K
Receiving objects: 100% (196/196), 6.71 MiB | 5.44 MiB/s, done.
Resolving deltas: 100% (101/101), done.<jupyter_text>Now we will download the latest Sinopia stage and production RDF exports in zip format.<jupyter_code># Downloaded from https://stage.sinopia.io/exports
! wget -O sinopia-stage.zip https://sinopia-exports-staging.s3-us-west-2.amazonaws.com/sinopia_export_all_2020-07-05T00:02:02.510Z.zip
# Downloaded from https://sinopia.io/exports
! wget -O sinopia-prod.zip https://sinopia-exports-production.s3-us-west-2.amazonaws.com/sinopia_export_all_2020-07-07T19:04:22.771Z.zip<jupyter_output>--2020-07-08 19:39:10--  https://sinopia-exports-production.s3-us-west-2.amazonaws.com/sinopia_export_all_2020-07-07T19:04:22.771Z.zip
Resolving sinopia-exports-production.s3-us-west-2.amazonaws.com (sinopia-exports-production.s3-us-west-2.amazonaws.com)... 172.16.17.32
Connecting to sinopia-exports-production.s3-us-west-2.amazonaws.com (sinopia-exports-production.s3-us-west-2.amazonaws.com)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5693153 (5.4M) [application/zip]
Saving to: ‘sinopia-prod.zip’

sinopia-prod.zip    100%[===================>]   5.43M  4.80MB/s    in 1.1s    

2020-07-08 19:39:12 (4.80 MB/s) - ‘sinopia-prod.zip’ saved [5693153/5693153]<jupyter_text>Next, we will install the [rdflib](https://rdflib.readthedocs.io/en/stable/) and a [rdflib-jsonld](https://github.com/RDFLib/rdflib-jsonld) packages needed by the **RDF Classify** modules we will be using in this demo.<jupyter_code>! pip install rdflib rdflib-jsonld
! pip install rdflib-jsonld==0.4.0<jupyter_output>Requirement already satisfied: rdflib-jsonld==0.4.0 in /usr/local/lib/python3.6/dist-packages (0.4.0)
Requirement already satisfied: rdflib>=4.2 in /usr/local/lib/python3.6/dist-packages (from rdflib-jsonld==0.4.0) (4.2.2)
Requirement already satisfied: pyparsing in /usr/local/lib/python3.6/dist-packages (from rdflib>=4.2->rdflib-jsonld==0.4.0) (2.4.7)
Requirement already satisfied: isodate in /usr/local/lib/python3.6/dist-packages (from rdflib>=4.2->rdflib-jsonld==0.4.0) (0.6.0)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from isodate->rdflib>=4.2->rdflib-jsonld==0.4.0) (1.12.0)<jupyter_text>Finally, we  import the [fastai.tabular](https://docs.fast.ai/tabular.html) module and we add the Python helper modules from the `rdf-classify` repository<jupyter_code>from fastai.tabular import *
import sys
sys.path.append("/content/rdf-classify/src")
import data_loader
import data_prep
help(data_loader)
help(data_prep)<jupyter_output>Help on module data_prep:

NAME
    data_prep

DESCRIPTION
    Data preparation helper module for converting a RDF Panda DataFrame to
    FastAI TabularList

FUNCTIONS
    databunch(df:pandas.core.frame.DataFrame, dependent_var:str='resource_template') -> fastai.basic_data.DataBunch

FILE
    /content/rdf-classify/src/data_prep.py<jupyter_text>Sinopia RDF Panda DataFramesThe `data_loader` module provides a `from_zipfile` function that iterates through an export zip file, extracts the RDF (and displays an error message if it cannot parse the RDF), and returns a list of RDF graphs of each resource in the Sinopia enviroment along with what group that graph is associated with in Sinopia.<jupyter_code>production = data_loader.from_zipfile('/content/sinopia-prod.zip')
stage = data_loader.from_zipfile('/content/sinopia-stage.zip')
print(f"Production size: {len(production)} Stage size: {len(stage)}")<jupyter_output><empty_output><jupyter_text>Here is an example of one of the RDF graphs in production:<jupyter_code>print(production[34]['graph'].serialize(format='turtle').decode())<jupyter_output><empty_output><jupyter_text>Now that we have lists of RDF graphs, we use the `data_loader.to_dataframe` helper function along with an URL to the environment's stored resource templates to construct a Pandas data-frame.<jupyter_code>production_df = data_loader.to_dataframe(production, 'https://trellis.sinopia.io/repository/ld4p')
stage_df = data_loader.to_dataframe(stage, 'https://trellis.stage.sinopia.io/repository/ld4p')<jupyter_output>Starting convert 2774 to dataframe at 2020-07-08 02:10:22.254947
.0..........100..........200..........300..........400..........500..........600..........700..........800..........900..........1000..........1100..........1200..........1300..........1400..........1500..........1600..........1700..........1800..........1900..........2000..........2100..........2200..........2300..........2400..........2500..........2600..........2700.......
    Finished at 2020-07-08 02:15:13.785642 time 4.85 minutes for dataframe,
    size 2776
Starting convert 4338 to dataframe at 2020-07-08 02:15:14.354121
Failed https://trellis.stage.sinopia.io/repository/ld4p/UNAM:RT:BF2:RetenciÃ³n status 404
.0..........100..........200..........300..........400..........500..........600..........700..........800..........900..........1000..........1100..........1200..........1300..........1400..........1500..........1600..........1700..........1800..........1900..........2000..........2100..........2200..........[...]<jupyter_text>Each dataframe has a different shape based on the number of propertyTemplates contained in the resource templates for each environment.<jupyter_code>production_df.fillna(0)
stage_df.fillna(0.)<jupyter_output><empty_output><jupyter_text>FastAI TabularList DataBunchBefore we can start using the FastAI machine learning classes, we first need to do some data preparation to convert our dataframes to a FastAI DataBunch subclass, TabularList. This is available in our helper `data_bunch.databunch` function.<jupyter_code>production_bunch = data_prep.databunch(production_df)
production_df = production_df.fillna(0.)
stage_df = stage_df.fillna(0.)<jupyter_output><empty_output><jupyter_text>TabularLearnerWith a TabularList, we can use the TabularLearner class to create a learner object that we can train and then use the learner to predict what an incoming RDF graph best matches to a Sinopia resource template<jupyter_code>prod_learner = tabular_learner(production_bunch, layers=[300,100], metrics=accuracy)
prod_learner.lr_find()
prod_learner.recorder.plot()
prod_learner.fit(3, 1e-02)
stage_bunch = data_prep.databunch(stage_df)
stage_learner = tabular_learner(stage_bunch, layers=[300,100], metrics=accuracy)
stage_learner.lr_find()
stage_learner.recorder.plot()
stage_learner.fit(3, 1e-10)
all_df = production_df + stage_df
all_databunch = data_prep.databunch(all_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17912118667786175, 'fraction_numerical': 0.07794570389028828, 'mean_word_length': 3.898560657984921, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 47, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '97877', 'license': 'Apache-2.0', 'n_tokens_mistral': 2749, 'n_tokens_neox': 2362, 'n_words': 526, 'path': 'doc/ClassifyingSinopiaRDF.ipynb', 'repo_name': 'LD4P/rdf-classify', 'size': '7158'}