<jupyter_start><jupyter_text>Introduction to the Semantic Web  Brief History* 1969: ARPAnet delivers first message* 1983: TCP/IP added to internet* 1989: WWW Invented along with HTTP proposed and implemented by [<NAME>](https://en.wikipedia.org/wiki/Tim_Berners-Lee)* 1999: Term "Semantic Web" defined by [<NAME>](https://en.wikipedia.org/wiki/Tim_Berners-Lee)* 1999: RDF recommendation adopted by W3C* 2004: RDF 1.0 spec published  Key Definitions[Semantic Web](https://www.w3.org/standards/semanticweb/)In addition to the classic “Web of documents” W3C is helping to build a technology stack to support a “Web of data,” the sort of data you find in databases. The ultimate goal of the Web of data is to enable computers to do more useful work and to develop systems that can support trusted interactions over the network. The term “Semantic Web” refers to W3C’s vision of the Web of linked data. Semantic Web technologies enable people to create data stores on the Web, build vocabularies, and write rules for handling data. Linked data are empowered by technologies such as RDF, SPARQL, OWL, and SKOS.[Linked Data](https://www.w3.org/standards/semanticweb/data.html)It is important to have the huge amount of data on the Web available in a standard format, reachable and manageable by Semantic Web tools. Furthermore, not only does the Semantic Web need access to data, but relationships among data should be made available, too, to create a Web of Data (as opposed to a sheer collection of datasets). This collection of interrelated datasets on the Web can also be referred to as Linked Data.[RDF (Resource Description Framework)](https://en.wikipedia.org/wiki/Resource_Description_Framework)The RDF data model is similar to classical conceptual modeling approaches (such as entity–relationship or class diagrams). It is based on the idea of making statements about resources (in particular web resources) in expressions of the form subject–predicate–object, known as triples. The subject denotes the resource, and the predicate denotes traits or aspects of the resource, and expresses a relationship between the subject and the object.[Ontology](https://www.w3.org/standards/semanticweb/ontology.html)On the Semantic Web, vocabularies define the concepts and relationships (also referred to as “terms”) used to describe and represent an area of concern. Vocabularies are used to classify the terms that can be used in a particular application, characterize possible relationships, and define possible constraints on using those terms. In practice, vocabularies can be very complex (with several thousands of terms) or very simple (describing one or two concepts only).There is no clear division between what is referred to as “vocabularies” and “ontologies”. The trend is to use the word “ontology” for more complex, and possibly quite formal collection of terms, whereas “vocabulary” is used when such strict formalism is not necessarily used or only in a very loose sense. Vocabularies are the basic building blocks for inference techniques on the Semantic Web.  References and Resourceshttps://github.com/semantalytics/awesome-semantic-web  What is actually in a URL?<jupyter_code>from urllib.parse import urlparse

urlparse("https://www.w3.org/People/Berners-Lee/card#i")<jupyter_output><empty_output><jupyter_text>Building the URI (Uniform Resource Identifier)For most of our cases, the scheme will be http or https, but can really be any protocol used to communicate over a network.The netloc, also known as the authority, is the network device that hosts the data being referenced by the URI.The path represents the location of the component located at the authority, where the data can be retrieved.The fragment portion is part of the URN (Uniform Resource Name), and signifies which object at the URL, the URI is pointing to.Lastly, from here on we will be using the term IRI (International Resource Identifier), which is an international version of the URI.  Working with RDF dataRDF data is an implementation of an unstructured graph data structure. Not all RDF can be cleanly placed into a relational database, but all relational database data can be cleanly represented as a graph matrix where the Subject is the IRI pointing to the primary key, the Predicate is the IRI pointing to the definition of the column name in a table, and the Object is either a literaly value, or a IRI pointing to the foreign key.Data in RDFs can be one of 3 types:1. IRI   * When used in a RDF, the IRI is used to denote the absolute or relative location to the resource being referenced.2. Literal   * A basic value that are not IRIs. Such as strings, integers, or an instance of a concrete class.3. Blank Node   * An anonymous reference to an object without an IRI. Usually used as a container type to reference a collection of RDF statements* Subject  * IRI  * Blank Node* Predicate  * IRI* Object  * IRI (Denotes a forign key when describing relational database data)  * Literal  * Blank Node  Example:*Table Name: People*|ID|Name|Age|Type|Knows||--|----|---|----|------||0|Tory|32|Human|||1|Clyde|13|Cat|0|Using the above table, we could make the following RDF statements:```textSubject(People0) -> Predicate(has_name) -> Object(Tory)Subject(People1) -> Predicate(knows) -> Object(People0)```<jupyter_code>!pip install rdflib rdflib-jsonld
import rdflib
from rdflib.namespace import FOAF , XSD

# create a Graph to store our RDF objects
graph = rdflib.Graph()

# Delcare our namespace to stuff all our stuff
ns = rdflib.Namespace("http://example.org")

# Declare new types
Person = rdflib.URIRef(ns + "/person")
Human = rdflib.URIRef(ns + "/human")
Cat = rdflib.URIRef(ns + "/cat")

# Create our People
Tory = rdflib.URIRef(Person + "/0")
Clyde = rdflib.URIRef(Person + "/1")

# Start populating the graphs
graph.add( (Tory, FOAF.name, rdflib.Literal("Tory")) )
graph.add( (Tory, rdflib.RDF.type, Human ) )
graph.add( (Tory, FOAF.age, rdflib.Literal(32) ) )

graph.add( (Clyde, FOAF.name, rdflib.Literal("Clyde")) )
graph.add( (Clyde, rdflib.RDF.type, Cat) )
graph.add( (Clyde, FOAF.age, rdflib.Literal(13) ) )
graph.add( (Clyde, FOAF.knows, Tory ) )
for triple in graph:
    print(triple)<jupyter_output>(rdflib.term.URIRef('http://example.org/person/1'), rdflib.term.URIRef('http://www.w3.org/1999/02/22-rdf-syntax-ns#type'), rdflib.term.URIRef('http://example.org/cat'))
(rdflib.term.URIRef('http://example.org/person/1'), rdflib.term.URIRef('http://xmlns.com/foaf/0.1/name'), rdflib.term.Literal('Clyde'))
(rdflib.term.URIRef('http://example.org/person/1'), rdflib.term.URIRef('http://xmlns.com/foaf/0.1/age'), rdflib.term.Literal('13', datatype=rdflib.term.URIRef('http://www.w3.org/2001/XMLSchema#integer')))
(rdflib.term.URIRef('http://example.org/person/0'), rdflib.term.URIRef('http://xmlns.com/foaf/0.1/age'), rdflib.term.Literal('32', datatype=rdflib.term.URIRef('http://www.w3.org/2001/XMLSchema#integer')))
(rdflib.term.URIRef('http://example.org/person/0'), rdflib.term.URIRef('http://xmlns.com/foaf/0.1/name'), rdflib.term.Literal('Tory'))
(rdflib.term.URIRef('http://example.org/person/1'), rdflib.term.URIRef('http://xmlns.com/foaf/0.1/knows'), rdflib.term.URIRef('http://example.org/pers[...]<jupyter_text>RDF Serialization FormatsSome common serialization formats include:* XML (Extensible Markup Language)* N3 (Notation3)* TTL (Terse RDF Triple Language)* JSON-LD (JSON for Linked-Data)* And MANY more!<jupyter_code>for fmt in ["xml", "n3", "ttl", "json-ld"]:
    print("=" * 20, fmt, "=" * 20)
    print(graph.serialize(format=fmt).decode())<jupyter_output>==================== xml ====================
<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF
   xmlns:ns1="http://xmlns.com/foaf/0.1/"
   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
>
  <rdf:Description rdf:about="http://example.org/person/1">
    <rdf:type rdf:resource="http://example.org/cat"/>
    <ns1:name>Clyde</ns1:name>
    <ns1:age rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">13</ns1:age>
    <ns1:knows rdf:resource="http://example.org/person/0"/>
  </rdf:Description>
  <rdf:Description rdf:about="http://example.org/person/0">
    <ns1:age rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">32</ns1:age>
    <ns1:name>Tory</ns1:name>
    <rdf:type rdf:resource="http://example.org/human"/>
  </rdf:Description>
</rdf:RDF>

==================== n3 ====================
@prefix ns1: <http://xmlns.com/foaf/0.1/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<http://example.org/person/1> a <http://example.org/cat> ;
    ns1:age 13 ;
    ns1:knows <ht[...]<jupyter_text>Applying Semantics to Linked DataThe simplest version of the [data pyramid](https://en.wikipedia.org/wiki/DIKW_pyramid) goes:Data -> Information -> Knowledge -> WisdomData is a collection of observations stated as facts.Information is inferred from data by extracting the useful parts out of the data. This is normally done through some sort of ETL (Extract. Transform. Load) process.Knowledge is information that is enriched with domain knowledge to add context to the information, or a combination of multiple sources of information to add context that would otherwise not have that context when standing alone.Wisdom is the shared understanding of the knowledge and how to apply it to business objects, and why it is useful.The value or cell in a database somewhere represents a data point. Each triple in our graph represents a single piece of information. The graph as a whole represents our collection of information. The next step is to apply domain knowledge to lift the information graph to a knowledge graph.The way we will encode and apply our domain knowledge is by using an ontology.  Vocabularies, Ontologies, and SchemasThe two most common ways to encode domain knowledge are:1. [RDFS (RDF Schema)](https://www.w3.org/TR/rdf-schema/)2. [OWL (Web Ontology Language)](https://www.w3.org/TR/owl-overview/)Vocabularies are themselves also just RDF graphs, but contain an encoding of domain knowledge and a set of constraints to validate or extend an existing knowledge graph.Lets start by bringing in the ontology for foaf (Friend of a Friend), the social network encoding ontology used in our previous example.<jupyter_code>ontology = rdflib.Graph()

url = "http://xmlns.com/foaf/spec/20140114.rdf"
namespace = rdflib.Namespace("http://xmlns.com/foaf/0.1/")

ontology.parse(url)

subjects = set(ontology.subjects())
predicates = set(ontology.predicates())
objects = set(ontology.objects())

print(f"Triples({len(ontology)}), Subjects({len(subjects)}), Predicates({len(predicates)}), Objects({len(objects)})")<jupyter_output>Triples(631), Subjects(86), Predicates(15), Objects(192)<jupyter_text>From here, we are able to categorize our subjects into two groups:1. Subjects that belong to the FOAF namespace2. External subjects that enrich the FOAF namespace items<jupyter_code>internal_subjects = set(sub for sub in subjects if sub.startswith(namespace))
external_subjects = subjects - internal_subjects

print(f"Internal({len(internal_subjects)}), External({len(external_subjects)})")
from collections import namedtuple

IRIRef = namedtuple("IRIRef", ("iri", "delim", "urn"))

def split(item):
    if not isinstance(item, rdflib.term.URIRef):
        return item
    iri, delim, urn = item.rpartition("#" if "#" in item else "/")
    return IRIRef(iri, delim, urn)
[split(subject)[2] for subject in internal_subjects]
[split(subject) for subject in external_subjects]<jupyter_output><empty_output><jupyter_text>In our example, the predicates are where things get interesting. Lets start by only looking at the namespaces brought in.In the below example, you can see we are bringing in predicates from not only multiple vocabularies, but multiple types or standards of vocabularies such as RDFS, and OWL standards. When building RDFs, you are encouraged to include as much sematics as possible, and you aren't required to stick to a single namespace, or even a single domain when describing your graph.<jupyter_code>set(split(predicate)[0] for predicate in predicates)
set(split(predicate) for predicate in predicates)<jupyter_output><empty_output><jupyter_text>While most RDF graphs describe relationships in your data, a vocabulary describes the relationships in your TYPES of data. In this case, a vocabulary is similar to a UML diagram that would descibe the relationships between python classes and python subclasses, to include base types (see the above "http://www.w3.org/1999/02/22-rdf-syntax-ns" and "http://purl.org/dc/elements/1.1") Next, lets look at objects that are references (not literal or blank nodes) that are not internal references inside the namespace. This will tell us how FOAF depends on external vocabularies.<jupyter_code>foreign_references = tuple(split(i) for i in predicates.union(objects).difference(subjects) if isinstance(i, rdflib.term.URIRef))
foreign_references
!pip install owlrl

import owlrl

combined_graph = graph + ontology

owlrl.DeductiveClosure(owlrl.CombinedClosure.RDFS_OWLRL_Semantics).expand(combined_graph)
print(f"The original graph contained {len(graph)} triples and the ontoloty contained {len(ontology)} triples. But after automated deductive reasoning it now contains {len(combined_graph)} triples!")
# Search the combined graph for all triples where Clyde is the Subject

for s, p, o in combined_graph.triples( (Clyde, None, None) ):  # None is considered a wildcard for iteration purposes
    if (s, p, o) in graph:
        continue  # Skip items in original graph, we only want to see new data that was learned through deductive reasoning
    print(split(p), split(o))
# Search the combined graph for all triples where Clyde is the Object

for s, p, o in combined_graph.triples( (None, None, Clyde) ):  # None is considered a wildcard for iteration purposes
    if (s, p, o) in graph:
        continue  # Skip items in original graph, we only want to see new data that was learned through deductive reasoning
    print(f"{split(s)}\n\t{split(p)}\n\t{split(o)}")<jupyter_output>IRIRef(iri='http://example.org/person', delim='/', urn='1')
	IRIRef(iri='http://www.w3.org/2002/07/owl', delim='#', urn='sameAs')
	IRIRef(iri='http://example.org/person', delim='/', urn='1')
{'fraction_non_alphanumeric': 0.11246136555212138, 'fraction_numerical': 0.014189379039055915, 'mean_word_length': 5.10244320617231, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 1, '>': 56, 'https://': 11, 'lorem ipsum': 0, 'www.': 15, 'xml': 14}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '15607', 'license': 'MIT', 'n_tokens_mistral': 4435, 'n_tokens_neox': 4218, 'n_words': 1582, 'path': 'notebook.ipynb', 'repo_name': 'tclasen/rdf-playground', 'size': '14254'}