<jupyter_start><jupyter_text>Linked Data and Music SPARQL Exercises: Exploring unknown datasets This Jupyter notebook aims to support a basic understanding of how to explore unknown Linked Data datasets with the query language SPARQL (https://www.w3.org/TR/rdf-sparql-query/). The Notebook is created by [@musicenfanthen](https://github.com/musicEnfanthen) and inspired by <NAME>](https://twitter.com/bobdc)'s description of the approach in his blog entry: - http://www.bobdc.com/blog/jupytersparql/ and the corresponding notebook - https://github.com/bobdc/misc/blob/master/JupyterSPARQL/Jupyter%20and%20SPARQL%20and%20Dort%20or%20Dordrecht.ipynb.It uses the `sparqlkernel` written and released by <NAME>: https://github.com/paulovn/sparql-kernel. To execute a block of code in this notebook, click in a cell and press `Shift+Enter`. For general information about Jupyter notebooks see http://jupyter.org/.   1. Setup At first, we have to set up this Jupyter notebook and the `sparqlkernel` which SPARQL endpoint, query format or output format we want to use. The `sparqlkernel` provides some `magics` (keywords) for this, so it is pretty straightforward. You will find more information about the syntax of these `magics` here: https://github.com/paulovn/sparql-kernelsyntax Possible music-related endpoints (in alphabetical order):**DoReMus**- Endpoint: http://data.doremus.org/sparql- Website:  https://www.doremus.org/**JazzCats**- Endpoint: http://cdhr-linkeddata.anu.edu.au/jazzcats-sparql/sparql- Website:  http://jazzcats.cdhr.anu.edu.au/**MIDI Linked Data** (could face timeouts)- Endpoint: http://virtuoso-midi.amp.ops.labs.vu.nl/sparql- Website:  https://midi-ld.github.io/**MusicOWL**- Endpoint: http://linkeddata.uni-muenster.de:7200- Website:  http://linkeddata.uni-muenster.de/musicportal/**RISM**- Endpoint: https://data.rism.info/sparql- Website:  https://opac.rism.info/**SLICKMEM**- Endpoint: https://blazegraph.linkedmusic.org/blazegraph/namespace/SLICKMEM/sparql- Website:  https://slickmem.linkedmusic.org<jupyter_code># Set the SPARQL endpoint (feel free to change to your liking)
%endpoint http://data.doremus.org/sparql

# Don't show more than 50 results (event if more are fetched)
%show 50

# Set the data format requested to the SPARQL endpoint (%format JSON | XML | N3 | any | default)
%format any

# Set the output rendering shape (%display raw | table [withtypes] | diagram [svg|png] [withliterals])
%display table<jupyter_output><empty_output><jupyter_text>Short test if the selected endpoint is running and we are connecting to it properly.**IMPORTANT: Use the `LIMIT` modifier in all of the following queries to avoid stressing the endpoint too much. You may need to increase your limit in certain queries according to the total number of query results e.g. in 2.1.1.**<jupyter_code>SELECT * 
WHERE {

    ?s ?p ?o . 

}
LIMIT 5<jupyter_output><empty_output><jupyter_text>2. SPARQL Queries  2.1. Exploring an unknown dataset – Concepts/Classes  2.1.0 Find all Concepts/Classes of the dataset  Explanation:General information about and specification of the SPARQL Query language can be found here: https://www.w3.org/TR/sparql11-query/ `SELECT`    = one of SPARQL's query forms; returns all, or a subset of, the variables bound in a query pattern match.`DISTINCT`  = ensure solutions in the sequence are unique, i.e. duplicates are merged into one result`*`         = shortcut to return all bound variables (`?xyz`) as results, here the results bound in `?Concept``WHERE {…}` = so-called WHERE-clause, basic graph pattern to match against the data graph`[]`        = blank node, see [4.1.4 Syntax for Blank Nodes](https://www.w3.org/TR/sparql11-query/QSynBlankNodes)`a`         = `rdf:type``?Concept` = random variable name for any URI that matches the given pattern in the object position of a triple.`LIMIT`     = modifier to limit the result number (important!)<jupyter_code>SELECT DISTINCT * 
WHERE {                 

    [] a ?Concept .     
    
}
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.1.1 Get total number of classes available in the dataset  Explanation`(COUNT(DISTINCT ?Concept)`= count number of distinct values bound to variable `?Concept`You may need to increase your limit in subsequent queries according to the resulting total number to see all query results.<jupyter_code>SELECT (COUNT(DISTINCT ?Concept) as ?count)
WHERE   
{ 
   
    [] a ?Concept .
 
}
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.1.2 Get the number of the individual concepts/classes in the dataset  Explanation`(COUNT(*) AS ?n)` = bind the number of results of all bound variables to the variable `?n``GROUP BY`         = group the results by the given value/variable`ORDER BY DESC`    = sort the results in descending order by the given value/variable (leave out `DESC` or use `ASC` for ascending order<jupyter_code>SELECT DISTINCT ?Concept (COUNT(*) AS ?n) 
WHERE {

    [] a ?Concept .

}
GROUP BY ?Concept
ORDER BY DESC(?n)
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.1.3 Get the number of these concepts/classes in the dataset plus a sample of every class  Explanationsee also: https://twitter.com/kidehen/status/1129143936839294976 `(SAMPLE(?s) AS ?sample)` = retrieve one random sample of the result set for the variable `?s` and bind the sample to the variable `?sample``COUNT(1)`                = equivalent of `COUNT(*)``?s`                      = random variable name for any URI that matches the given pattern in the subject position of a triple`FILTER(isIri(?s))`       = filter out only those solutions for the variable `?s` that are an IRI<jupyter_code>SELECT (SAMPLE(?s) AS ?sample) (COUNT(1) AS ?n) (?Concept AS ?entityType) 
WHERE {

    ?s a ?Concept .
    FILTER(isIri(?s)) .
}
GROUP BY ?Concept
ORDER BY DESC(?n)
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.1.4 Get all classes/concepts with a given prefix/namespace  Explanationsee also: https://codyburleson.com/sparql-examples-select/ `FILTER( STRSTARTS(STR(?Concept),str(owl:)) )`       = filter out only those solutions for the variable `?Concept`  that start with the given string `owl:` (try to replace with `foaf:`, `rdfs:`, `mo:` or others)<jupyter_code>PREFIX doremus: <http://data.doremus.org/ontology#>
PREFIX ecrm: <http://erlangen-crm.org/current/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX mo: <http://purl.org/ontology/mo/>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>


SELECT DISTINCT ?Concept
WHERE {
    
     [] a ?Concept .
     FILTER( STRSTARTS(STR(?Concept),str(owl:)) )
    
}
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.1.5 Get and count the namespaces of all classes/concepts used in the dataset  Explanationsee also: https://stackoverflow.com/a/20054523`COUNT (DISTINCT ?Concept)` = count only distinct values bound to variable `?Concept`, i.e. make sure all occurences of classes/concepts are counted only once`REPLACE(str(?Concept), "(|/)[^/]*$", "$1")` = REGEX expression that is used to remove the individual class name from the namespaces bound to `?Concept``BIND(… AS ?ns)` = binds the result of the regex replacement to the variable `?ns`<jupyter_code>SELECT ?ns (COUNT (DISTINCT ?Concept) AS ?nsCount)
WHERE   
{ 
   [] a ?Concept .
   BIND(REPLACE(str(?Concept), "(#|/)[^#/]*$", "$1") AS ?ns)
 
}
ORDER BY DESC(?nsCount)
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.2 Exploring an unknown dataset – Properties  2.2.0 Get all properties used in the dataset  Explanationsee above for classes/concepts<jupyter_code>SELECT DISTINCT ?p
WHERE   
{ 
   
    ?s ?p ?o .
 
}
ORDER BY ?p
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.2.1 Get total number of properties used in the dataset  Explanation`(COUNT(DISTINCT ?p)`= count number of distinct values bound to variable `?p`<jupyter_code>SELECT (COUNT(DISTINCT ?p) as ?pCount)
WHERE   
{ 
   
    ?s ?p ?o .
 
}
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.2.2 Get properties with a given prefix/namespace  Explanationsee above for classes/concepts `FILTER( STRSTARTS(STR(?Concept),str(owl:)) )`       = filter out only those solutions for the variable `?Concept`  that start with the given string `owl:` (try to replace with `foaf:`, `rdfs:`, or `mo:`)<jupyter_code>PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX mo: <http://purl.org/ontology/mo/>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>


SELECT DISTINCT ?p
WHERE {
    
     ?s ?p ?o .
     FILTER( STRSTARTS(STR(?p),str(owl:)) )
    
}
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.2.3 Get and count the namespaces of all properties used in the dataset  Explanationsee also: https://stackoverflow.com/a/20054523`COUNT (DISTINCT ?Concept)` = count only distinct values bound to variable `?Concept`, i.e. make sure all occurences of classes/concepts are counted only once`REPLACE(str(?Concept), "(|/)[^/]*$", "$1")` = REGEX expression that is used to remove the individual class name from the namespaces bound to `?Concept``BIND(… AS ?ns)` = binds the result of the regex replacement to the variable `?ns`<jupyter_code>SELECT ?ns (COUNT (DISTINCT ?p) AS ?nsCount)
WHERE   
{ 
   ?s ?p ?o .
   BIND(REPLACE(str(?p), "(#|/)[^#/]*$", "$1") AS ?ns)
 
}
ORDER BY DESC(?nsCount)
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.3 Exploring a certain class/concept  2.3.0 Find all persons of the dataset Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2. Explanation:`PREFIX foaf: ` = shortcut prefix for the [FOAF](http://xmlns.com/foaf/spec/) vocabulary and namespace`SELECT DISTINCT ?person` = return all unique values bound in the variable `?person` as results`foaf:Person`        = concept of Person from the [FOAF](http://xmlns.com/foaf/spec/) vocabulary (replace with the corresponding Person concept according to your dataset)<jupyter_code>PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT DISTINCT ?person 
WHERE {
    
    ?person a foaf:Person .
} 
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.3.1 Count all persons in the dataset Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2. Explanation:see 2.1.1`BIND(foaf:Person as ?personConcept )` = bind `foaf:Person` to the variable `?personConcept`<jupyter_code>PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT  ?personConcept (COUNT(?person) AS ?personCount)
WHERE {
    
        ?person a foaf:Person .
        BIND(foaf:Person as ?personConcept )
}
GROUP BY ?personConcept
ORDER BY DESC (?personCount)
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.3.2 Find all properties assigned to the Person concept (as subject) in the dataset Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2. Explanation:`?p` = random variable name for any URI that matches the given pattern in the predicate position of a triple<jupyter_code>PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT DISTINCT ?p
WHERE {
    
        ?person a foaf:Person .
        ?person ?p ?o .

}
ORDER BY ?p
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.3.3 Find and count all properties assigned to the Person concept (as subject) in the dataset Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2. Explanation:`?p` = random variable name for any URI that matches the given pattern in the predicate position of a triple<jupyter_code>PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT ?p (COUNT(?p) AS ?pCount) 
WHERE
{
      ?person a foaf:Person .
      ?person ?p ?o .
}
GROUP BY ?p
ORDER BY DESC (?pCount)
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.3.4 Find and count all properties assigned to the Person concept (as object) in the dataset Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2. Explanation:see 2.2.3<jupyter_code>PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT DISTINCT ?p (COUNT(?p) AS ?pCount)
WHERE {
    
        ?person a foaf:Person .
        ?s ?p ?person .
        
}
GROUP BY ?p
ORDER BY DESC(?pCount)
LIMIT 50<jupyter_output><empty_output><jupyter_text>2.3.5 Find and count all properties assigned to the Person concept (as subject & object) in the dataset The same query merging the two columns into one thus merging properties that are available for Persons in the position of subject and object. Explanation:`{…} UNION {…}` = keyword to form a disjunction of two graph patterns in terms of a logical OR; solutions to both sides of the UNION are included in the results.<jupyter_code>PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT DISTINCT ?p (COUNT(?p) AS ?pCount)
WHERE
{
        ?person a foaf:Person .
        { ?person ?p [] . }
        UNION
        { [] ?p ?person . }
}
GROUP BY ?p
ORDER BY DESC (?pCount)
LIMIT 50<jupyter_output><empty_output><jupyter_text>Of which type are possible subjects and objects of a property? And what does a sample look like?<jupyter_code>PREFIX mo: <http://purl.org/ontology/mo/>

SELECT (SAMPLE(?s) AS ?perfomance) ?sType (COUNT(?s) AS ?pCount) ?oType (SAMPLE(?o) AS ?performer)
WHERE
{
    
      ?s mo:performer ?o .  # replace with a property that is in your dataset
      ?s a ?sType .
      ?o a ?oType .

}
GROUP BY ?sType ?oType
ORDER BY DESC (?pCount)
LIMIT 50<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11278516096936962, 'fraction_numerical': 0.017767038590007816, 'mean_word_length': 4.278319579894974, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 0, '>': 99, 'https://': 18, 'lorem ipsum': 0, 'www.': 12, 'xml': 10}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '59800', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 4817, 'n_tokens_neox': 4557, 'n_words': 1600, 'path': 'NB_Explore unkown datasets with SPARQL.ipynb', 'repo_name': 'jgarte/music21playground', 'size': '14087'}