<commit_before>\documentclass[11pt,a4paper]{article}

% \usepackage{authblk}
\usepackage{indentfirst}
\usepackage[linewidth=1pt]{mdframed}
\usepackage{listings}
\usepackage{url}
\usepackage{amsmath}

\title{Umaka Score}

\begin{document}

\maketitle

\begin{abstract}
Umaka Scores represents how valuable endpoints are.
Umaka Score is calculated on the basis of the evaluation from the 6 aspects, Availability, Freshness, Operation, Usefulness, Validity and Performance.
We also rank the endpoints on a scale of A to E according to the Umaka score.

In this document, we show how to calculate Umaka score.
\end{abstract}

\section{Umaka Score}

Umaka Score represents how valuable endpoints are.
We believe there are six aspects, Availability, Freshness, Operation, Usefulness, Validity and Performance, for  valuable endpoints. We evaluate and score endpoints from these aspects.
Then Umaka Score is average of these score:

\begin{mdframed}
  \center
Umaka Score = $\frac{\displaystyle \sum_{aspects}score}{6}$

where

$\begin{array}{llll}
{\rm aspects} = [ & {\rm Availability}, & {\rm Freshness}, & {\rm Operation}, \\
                  & {\rm Usefulness},   & {\rm Validity},  & {\rm Performance} ]
\end{array}$

\end{mdframed}

We rank the endpoint as shown in Table \ref{table:umaka_rank}

\begin{table}
  \center
  \begin{tabular}{cc}
    \hline
    Umaka Score & Umaka rank \\
    \hline
    81 - 100 & A \\
    61 - 80  & B \\
    41 - 60  & C \\
    21 - 40  & D \\
    0  - 20  & E \\
    \hline
  \end{tabular}
  \caption{Umaka Rank}
  \label{table:umaka_rank}
\end{table}

In Section \ref{metrics}, we show how to score endpoints from each aspect.

\section{Metrics of Umaka Score} \label{metrics}

\subsection{Availability}

Availability represents the degree of ready for use. High availability value means we can access the endpoint most of all the time. Low availability value means the endpoint is often down.
We measure the following metrics for availability:

\begin{itemize}

\item Alive

We send HTTP request to the endpoint URI daily. If the endpoint return 200 HTTP response, alive is true, otherwise false.
Note that we assume the endpoint as dead when the endpoint returns 302 Found.

\item Alive Rate

Alive rate is the percentage of alive in 30 days.

\end{itemize}

The Availability score is calculated as:

\begin{mdframed}
  \center
Availability = Alive~Rate
\end{mdframed}

\subsection{Freshness}

Freshness represents how often data in the endpoint is updated.
We measure the following metrics for freshness:

\begin{itemize}
  \item Last Updated

  We retrieve Service Description and VoID and get the literals specified by dcterms:modified.
  Then we assume the last updated as the latest date among those literals.

  Unfortunately, most of all endpoints provide Service Description and VoID without dcterms:modified statement.

  Thus, we determine the data modification thought the following adhoc procedure:

  First, we retrieve a number of statements using a query described in Listing \ref{list:query_for_number_of_statements} and compare the number with the previous one. If the number of statements has changed, we assume the endpoint seems to be updated today.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A query for retrieving a number of statements,label=list:query_for_number_of_statements]
SELECT COUNT(*) AS ?c
WHERE {?s ?p ?o}
  \end{lstlisting}

  When the number of statements is not changed, we retrieve the first and the last statements using queries described in Listing \ref{list:query_for_the_first_statement} and \ref{list:query_for_the_last_statement}.
  Then we compare them with those of a previous day. If one of them is different, we assume the endpoint seems to be updated today.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A query for the first statement,label=list:query_for_the_first_statement]
SELECT *
WHERE {?s ?p ?o}
OFFSET 0 LIMIT 1
  \end{lstlisting}

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A query for the last statement,label=list:query_for_the_last_statement]
SELECT *
WHERE {?s ?p ?o}
OFFSET ($COUNT - 1) LIMIT 1
  \end{lstlisting}

  Note that \$COUNT represents the number of total statements retrieved by Listing \ref{list:query_for_number_of_statements}.

  Note that Virtuoso has trouble to count the number of statements. So this adhoc procedure does not work well for the endpoint based on Virtuoso.

  \item Update Interval

  Update Interval is average of the interval between last updated.
  Update Interval is N/A if there are less than two last updated dates for the endpoint.

\end{itemize}

Even though we would like to score freshness on the basis of the Update Interval,
we give up to score and set 50 for all endpoints since the adhoc approach does not work well for Virtuoso.

\begin{mdframed}
  \center
Freshness = 50
\end{mdframed}

\subsection{Operation}

Operation represents the degree of the maintenance.
We measure the two metrics:

\begin{itemize}

  \item Service Description

  true if Service Description can be retrieved in Turtle format or RDFXML format, otherwise false.

  \item VoID

  true if VoID can be retrieved from well-known URI\cite{Alexandar:11:VoID} in Turtle format or RDFXML format, otherwise false.

\end{itemize}

We calculate Operation score as follows:

\begin{mdframed}
\center
Operation = $\left\{
    \begin{array}{ll}
      0   & {\rm if~both~of~them~are~false } \\
      50  & {\rm if~one~of~them~is~false } \\
      100 & {\rm if~both~of~them~are~true }
  \end{array}
  \right.$
\end{mdframed}

\subsection{Usefulness}

Usefulness represents the degree how easily we can link data in the endpoint.
We measure the three metrics:

\begin{itemize}
  \item Metadata Score

  Metadata Score represents how much the endpoint contains metadata defined in \cite{SparqleBuilderMetadata}.

  First we retrieve a list of graphs in the endpoint using a query described in Listing \ref{list:query_for_a_list_of_graphs}.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain graph URIs on a SPARQL endpoint,label=list:query_for_a_list_of_graphs]
SELECT DISTINCT ?g
WHERE{
 GRAPH ?g{ ?s ?p ?o.}
}
  \end{lstlisting}

  Then we try to retrieve the metadata for each graph except for Table \ref{table:ignore_graphs} as follows:

  \begin{table}[htbp]
    \center
    \begin{tabular}{|l|}
      \hline
      Graph URI \\
      \hline
      \url{http://www.openlinksw.com/schemas/virtrdf#} \\
      \hline
    \end{tabular}
    \caption{List of Ignore Graphs}
    \label{table:ignore_graphs}
  \end{table}

  \begin{enumerate}

    \item Classes

    We retrieve a list of classes using a query described in Listing \ref{list:query_for_classes_on_a_graph} and \ref{list:query_for_classes_having_instances_on_a_graph}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the classes on a graph g,label=list:query_for_classes_on_a_graph]
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT DISTINCT ?c
FROM <g>
WHERE {
  { ?c rdf:type rdfs:Class. }
  UNION
  { [] rdf:type ?c. }
  UNION
  { [] rdfs:domain ?c. }
  UNION
  { [] rdfs:range ?c. }
  UNION
  { ?c rdfs:subclassOf []. }
  UNION
  { [] rdfs:subclassOf ?c. }
}
LIMIT 100
    \end{lstlisting}

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the classes having instances on a graph g,label=list:query_for_classes_having_instances_on_a_graph]
PREFIX rdf:
SELECT DISTINCT ?c
        FROM <g>
WHERE{
        [] rdf:type ?c.
}
    \end{lstlisting}

    \item Labels

    We retrieve a list of labels using a query described in Listing \ref{list:query_for_labels_of_classes}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain labels of the classes c1 c2 ... cn from a graph g,label=list:query_for_labels_of_classes]
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT DISTINCT ?c ?label
WHERE {
    graph <g> {
      ?c rdfs:label ?label.
      filter (
        ?c IN (<c1>, <c2>, ..., <cn>)
      )
    }
}
    \end{lstlisting}

    \item Datatypes

    We retrieve a list of datatypes using a query described in Listing \ref{list:query_for_datatypes_on_a_graph}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the datatypes on a graph g,label=list:query_for_datatypes_on_a_graph]
SELECT DISTINCT (datatype(?o) AS ?ldt)
FROM <g>
WHERE{
  [] ?p ?o.
  FILTER(isLiteral(?o))
}
    \end{lstlisting}

    \item Properties

    We retrieve a list of properties using a query described in Listing \ref{list:query_for_properties_on_a_graph}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the properties on a graph g,label=list:query_for_properties_on_a_graph]
SELECT DISTINCT ?p
        FROM <g>
WHERE{
        ?s ?p ?o.
}
    \end{lstlisting}

  \end{enumerate}

  We evaluate Metadata score as follows:

  \begin{mdframed}
    \center
Metadata Score = $\frac{\displaystyle \sum_{graphs}^{g}(c(g) + l(g) + p(g) + d(g))}{N}$

where

$N$ = Number of Graphs

$c(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~classes} \\
            25  & {\rm if~g~contains~more~than~zero~classes}
        \end{array}
        \right.$

$l(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~labels} \\
            25  & {\rm if~g~contains~more~than~zero~labels}
        \end{array}
        \right.$

$p(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~properties} \\
            25  & {\rm if~g~contains~more~than~zero~properties}
        \end{array}
        \right.$

$d(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~datatypes} \\
            25  & {\rm if~g~contains~more~than~zero~datatypes}
        \end{array}
        \right.$

  \end{mdframed}

\item Vocabulary Score

  Vocabulary Score, which is calculated based on metadata, represents how many vocabularies data in the endpoint use.

  Vocabulary Score is calculated as follows:

  \begin{mdframed}
    \center

Vocabulary Score = $\frac{\displaystyle \sum_{graphs}^{g}v(g)}{N}$

where

$N$ = Number of Graphs

$v(g)$ = Number of Properties in Graph g

  \end{mdframed}

\item Ontology Score

  Ontology Score, which is calculated based on metadata, represents how much common ontologies data in the endpoint use.

  Ontology Score is calculated as follows:

  \begin{mdframed}
    \center

Vocabulary Score = $\frac{\displaystyle \sum_{graphs}^{g}o(g)}{N}$

where

$N$ = Number of Graphs

$o(g)$ = $\frac{NCO}{NO}$

$NO$ = Number of Ontologies used for Properties

$NCO$ = Number of Ontologies used for Properties in Table \ref{table:common_ontologies}

  \end{mdframed}

  \begin{table}[htbp]
    \center
    \begin{tabular}{|l|}
      \hline
      Ontology URI \\
      \hline
      \url{http://www.w3.org/2000/01/rdf-schema} \\
      \url{http://www.w3.org/1999/02/22-rdf-syntax-ns} \\
      \url{http://www.socrata.com/rdf/terms} \\
      \url{http://www.w3.org/2003/01/geo/wgs84_pos} \\
      \url{http://xmlns.com/foaf/0.1/} \\
      \url{http://www.w3.org/2002/07/owl} \\
      \url{http://purl.org/dc/elements/1.1/} \\
      \url{http://purl.org/dc/terms/} \\
      \url{http://www.w3.org/2000/10/swap/pim/usps} \\
      \url{http://dublincore.org/documents/dcmi-box/} \\
      \url{http://www.territorio.provincia.tn.it/geodati/ontology/} \\
      \url{http://www.w3.org/2004/02/skos/core} \\
      \hline
    \end{tabular}
    \caption{List of Common Ontologies}
    \label{table:common_ontologies}
  \end{table}

\end{itemize}

At last, we evaluate Usefulness Score as follows:

\begin{mdframed}
  \center
  $\begin{array}{lll}
  {\rm Usefulness} & = & 30.0 * {\rm Metadata~Score} \\
                   & + & 40.0 * f10({\rm Vocabulary~Score}) \\
                   & + & 30.0 * {\rm Ontology~Score}
  \end{array}$

  where

  $f10(x) = \left\{
    \begin{array}{ll}
      10 & {\rm if}~x>10 \\
      x  & {\rm Otherwise}
    \end{array}
  \right.$
\end{mdframed}

\subsection{Validity}

  Validity represents how endpoint and data in it obey the rules.
  We measure the two metrics:

  \begin{itemize}

    \item Cool URI

    The URI of endpoints is preferred to be Cool URI\cite{Leo:08:CoolUri}, \cite{Leigh:12:LinkedDataPatterns}.

    We check four criteria:
    \begin{enumerate}
      \item A host of URI of endpoints should not be specified by IP address
      \item A port of URI of endpoints should be 80
      \item A URI of endpoints should not contain query parameters
      \item A length of URI of endpoints should be less than 30 characters
    \end{enumerate}

    Cool URI Score is a percentage of the satisfied rules.

    \item Linked Data Rule

    The endpoints are preferred to be satisfied with the four rules of linked data\cite{Tim:06:LinkedDataRules}.

    We check four criteria:
    \begin{enumerate}
      \item Use URIs as names for things

      We assume all subjects of statements are things. We search invalid statement using a query described in Listing \ref{list:non_uri_subject}, and if nothing is found the endpoint satisfied this rule.

      Note that we ignore Virtuoso specific graphs since Virtuoso contains a graph which contains invalid statements.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for searching non-URI subjects,label=list:non_uri_subject]
SELECT
  *
WHERE {
GRAPH ?g { ?s ?p ?o } .
  filter (!isURI(?s) && !isBLANK(?s) && ?g NOT IN (
    <http://www.openlinksw.com/schemas/virtrdf#>
  ))
}
LIMIT 1
      \end{lstlisting}

      \item Use HTTP URIs so that people can look up those names

      We assume all subjects of statements are things. We search invalid statement using a query described in Listing \ref{list:non_http_uri_subject}, and if nothing is found the endpoint satisfied this rule.

      Note that we ignore Virtuoso specific graphs since Virtuoso contains a graph which contains invalid statements.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for searching non-HTTP-URI subjects,label=list:non_http_uri_subject]
SELECT
  *
WHERE {
  GRAPH ?g { ?s ?p ?o } .
  filter (!regex(?s, "http://", "i") && !isBLANK(?s) && ?g NOT IN (
    <http://www.openlinksw.com/schemas/virtrdf#>
  ))
}
LIMIT 1
      \end{lstlisting}

      \item When someone looks up a URI, provide useful information, using the standards (RDF*, SPARQL)

      We assess this rule by obtaining a subject (URI) using a query described in Listing \ref{list:query_for_a_subject} and accessing the URI via HTTP protocol. We assume that the endpoint is satisfied with the rule if the URI returns any data.

      Note that we ignore Virtuoso specific graphs since Virtuoso contains a graph which contains invalid statements.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for a Subject,label=list:query_for_a_subject]
SELECT
  ?s
WHERE {
  GRAPH ?g { ?s ?p ?o } .
  filter (isURI(?s) && ?g NOT IN (
    <http://www.openlinksw.com/schemas/virtrdf#>
  ))
}
LIMIT 1
OFFSET 100
      \end{lstlisting}

      \item Include links to other URIs. so that they can discover more things

      We assume the statement representing the link to other URI uses the vocabularies owl:sameAs or rdfs:seeAlso. We think if there are any statement of which property is owl:sameAs or rdfs:seeAlso, the endpoint is satisfied with the rule.
      Thus we check the feasibility of the rule by using queries described in Listing \ref{list:query_for_same_as}, \ref{list:query_for_see_also}.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for a Same AS Statement,label=list:query_for_same_as]
PREFIX owl:<http://www.w3.org/2002/07/owl#>
SELECT
  *
WHERE {
  GRAPH ?g { ?s owl:sameAs ?o } .
}
LIMIT 1
      \end{lstlisting}

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for a See Also Statement,label=list:query_for_see_also]
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT
  *
WHERE {
  GRAPH ?g { ?s rdfs:seeAlso ?o } .
}
LIMIT 1
      \end{lstlisting}

    \end{enumerate}

    Linked Data Score is a percentage of the satisfied rules.

  \end{itemize}

  We evaluate Validity as follows:

  \begin{mdframed}
    \center
Validity = 40 * Cool URI Score + 60.0 * Linked Data Rule Score
  \end{mdframed}

\subsection{Performance}

  Performace suggests how powerful the endpoint is.

  We measure the response times of the two queries, Listing \ref{list:query_ask}, \ref{list:query_for_list_of_graphs}. The former query is a most simple query and we use this query to estimate the transfer time. The latter query requires a little computations for endpoints. We believe the execution cost of this query does not differ very much according to the size of data.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Most Simple Query,label=list:query_ask]
ASK {}
  \end{lstlisting}

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for Listing Graphs,label=list:query_for_list_of_graphs]
SELECT DISTINCT
  ?g
WHERE {
  GRAPH ?g { ?s ?p ?o }
}
  \end{lstlisting}

  We assume the execution time as:

  \begin{mdframed}
    \center
Execution Time = Differences of the response time for those queries.
  \end{mdframed}

  After that, we evaluate Performance as:

  \begin{mdframed}
    \center
Performance = $\left\{
    \begin{array}{ll}
        \multicolumn{2}{l}{100.0 * (1.0 - {\rm Execution~Time})} \\
          & {\rm if~Execution~Time~is~less~than~1~second}  \\
        0 & {\rm Otherwise}
    \end{array}
    \right.$
  \end{mdframed}


\bibliographystyle{plain}
\bibliography{umaka-score}

\end{document}
<commit_msg>change 'RDFXML' to 'RDF/XML'<commit_after>\documentclass[11pt,a4paper]{article}

% \usepackage{authblk}
\usepackage{indentfirst}
\usepackage[linewidth=1pt]{mdframed}
\usepackage{listings}
\usepackage{url}
\usepackage{amsmath}

\title{Umaka Score}

\begin{document}

\maketitle

\begin{abstract}
Umaka Scores represents how valuable endpoints are.
Umaka Score is calculated on the basis of the evaluation from the 6 aspects, Availability, Freshness, Operation, Usefulness, Validity and Performance.
We also rank the endpoints on a scale of A to E according to the Umaka score.

In this document, we show how to calculate Umaka score.
\end{abstract}

\section{Umaka Score}

Umaka Score represents how valuable endpoints are.
We believe there are six aspects, Availability, Freshness, Operation, Usefulness, Validity and Performance, for  valuable endpoints. We evaluate and score endpoints from these aspects.
Then Umaka Score is average of these score:

\begin{mdframed}
  \center
Umaka Score = $\frac{\displaystyle \sum_{aspects}score}{6}$

where

$\begin{array}{llll}
{\rm aspects} = [ & {\rm Availability}, & {\rm Freshness}, & {\rm Operation}, \\
                  & {\rm Usefulness},   & {\rm Validity},  & {\rm Performance} ]
\end{array}$

\end{mdframed}

We rank the endpoint as shown in Table \ref{table:umaka_rank}

\begin{table}
  \center
  \begin{tabular}{cc}
    \hline
    Umaka Score & Umaka rank \\
    \hline
    81 - 100 & A \\
    61 - 80  & B \\
    41 - 60  & C \\
    21 - 40  & D \\
    0  - 20  & E \\
    \hline
  \end{tabular}
  \caption{Umaka Rank}
  \label{table:umaka_rank}
\end{table}

In Section \ref{metrics}, we show how to score endpoints from each aspect.

\section{Metrics of Umaka Score} \label{metrics}

\subsection{Availability}

Availability represents the degree of ready for use. High availability value means we can access the endpoint most of all the time. Low availability value means the endpoint is often down.
We measure the following metrics for availability:

\begin{itemize}

\item Alive

We send HTTP request to the endpoint URI daily. If the endpoint return 200 HTTP response, alive is true, otherwise false.
Note that we assume the endpoint as dead when the endpoint returns 302 Found.

\item Alive Rate

Alive rate is the percentage of alive in 30 days.

\end{itemize}

The Availability score is calculated as:

\begin{mdframed}
  \center
Availability = Alive~Rate
\end{mdframed}

\subsection{Freshness}

Freshness represents how often data in the endpoint is updated.
We measure the following metrics for freshness:

\begin{itemize}
  \item Last Updated

  We retrieve Service Description and VoID and get the literals specified by dcterms:modified.
  Then we assume the last updated as the latest date among those literals.

  Unfortunately, most of all endpoints provide Service Description and VoID without dcterms:modified statement.

  Thus, we determine the data modification thought the following adhoc procedure:

  First, we retrieve a number of statements using a query described in Listing \ref{list:query_for_number_of_statements} and compare the number with the previous one. If the number of statements has changed, we assume the endpoint seems to be updated today.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A query for retrieving a number of statements,label=list:query_for_number_of_statements]
SELECT COUNT(*) AS ?c
WHERE {?s ?p ?o}
  \end{lstlisting}

  When the number of statements is not changed, we retrieve the first and the last statements using queries described in Listing \ref{list:query_for_the_first_statement} and \ref{list:query_for_the_last_statement}.
  Then we compare them with those of a previous day. If one of them is different, we assume the endpoint seems to be updated today.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A query for the first statement,label=list:query_for_the_first_statement]
SELECT *
WHERE {?s ?p ?o}
OFFSET 0 LIMIT 1
  \end{lstlisting}

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A query for the last statement,label=list:query_for_the_last_statement]
SELECT *
WHERE {?s ?p ?o}
OFFSET ($COUNT - 1) LIMIT 1
  \end{lstlisting}

  Note that \$COUNT represents the number of total statements retrieved by Listing \ref{list:query_for_number_of_statements}.

  Note that Virtuoso has trouble to count the number of statements. So this adhoc procedure does not work well for the endpoint based on Virtuoso.

  \item Update Interval

  Update Interval is average of the interval between last updated.
  Update Interval is N/A if there are less than two last updated dates for the endpoint.

\end{itemize}

Even though we would like to score freshness on the basis of the Update Interval,
we give up to score and set 50 for all endpoints since the adhoc approach does not work well for Virtuoso.

\begin{mdframed}
  \center
Freshness = 50
\end{mdframed}

\subsection{Operation}

Operation represents the degree of the maintenance.
We measure the two metrics:

\begin{itemize}

  \item Service Description

  true if Service Description can be retrieved in Turtle format or RDF/XML format, otherwise false.

  \item VoID

  true if VoID can be retrieved from well-known URI\cite{Alexandar:11:VoID} in Turtle format or RDFXML format, otherwise false.

\end{itemize}

We calculate Operation score as follows:

\begin{mdframed}
\center
Operation = $\left\{
    \begin{array}{ll}
      0   & {\rm if~both~of~them~are~false } \\
      50  & {\rm if~one~of~them~is~false } \\
      100 & {\rm if~both~of~them~are~true }
  \end{array}
  \right.$
\end{mdframed}

\subsection{Usefulness}

Usefulness represents the degree how easily we can link data in the endpoint.
We measure the three metrics:

\begin{itemize}
  \item Metadata Score

  Metadata Score represents how much the endpoint contains metadata defined in \cite{SparqleBuilderMetadata}.

  First we retrieve a list of graphs in the endpoint using a query described in Listing \ref{list:query_for_a_list_of_graphs}.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain graph URIs on a SPARQL endpoint,label=list:query_for_a_list_of_graphs]
SELECT DISTINCT ?g
WHERE{
 GRAPH ?g{ ?s ?p ?o.}
}
  \end{lstlisting}

  Then we try to retrieve the metadata for each graph except for Table \ref{table:ignore_graphs} as follows:

  \begin{table}[htbp]
    \center
    \begin{tabular}{|l|}
      \hline
      Graph URI \\
      \hline
      \url{http://www.openlinksw.com/schemas/virtrdf#} \\
      \hline
    \end{tabular}
    \caption{List of Ignore Graphs}
    \label{table:ignore_graphs}
  \end{table}

  \begin{enumerate}

    \item Classes

    We retrieve a list of classes using a query described in Listing \ref{list:query_for_classes_on_a_graph} and \ref{list:query_for_classes_having_instances_on_a_graph}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the classes on a graph g,label=list:query_for_classes_on_a_graph]
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT DISTINCT ?c
FROM <g>
WHERE {
  { ?c rdf:type rdfs:Class. }
  UNION
  { [] rdf:type ?c. }
  UNION
  { [] rdfs:domain ?c. }
  UNION
  { [] rdfs:range ?c. }
  UNION
  { ?c rdfs:subclassOf []. }
  UNION
  { [] rdfs:subclassOf ?c. }
}
LIMIT 100
    \end{lstlisting}

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the classes having instances on a graph g,label=list:query_for_classes_having_instances_on_a_graph]
PREFIX rdf:
SELECT DISTINCT ?c
        FROM <g>
WHERE{
        [] rdf:type ?c.
}
    \end{lstlisting}

    \item Labels

    We retrieve a list of labels using a query described in Listing \ref{list:query_for_labels_of_classes}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain labels of the classes c1 c2 ... cn from a graph g,label=list:query_for_labels_of_classes]
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT DISTINCT ?c ?label
WHERE {
    graph <g> {
      ?c rdfs:label ?label.
      filter (
        ?c IN (<c1>, <c2>, ..., <cn>)
      )
    }
}
    \end{lstlisting}

    \item Datatypes

    We retrieve a list of datatypes using a query described in Listing \ref{list:query_for_datatypes_on_a_graph}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the datatypes on a graph g,label=list:query_for_datatypes_on_a_graph]
SELECT DISTINCT (datatype(?o) AS ?ldt)
FROM <g>
WHERE{
  [] ?p ?o.
  FILTER(isLiteral(?o))
}
    \end{lstlisting}

    \item Properties

    We retrieve a list of properties using a query described in Listing \ref{list:query_for_properties_on_a_graph}.

    \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=Obtain the properties on a graph g,label=list:query_for_properties_on_a_graph]
SELECT DISTINCT ?p
        FROM <g>
WHERE{
        ?s ?p ?o.
}
    \end{lstlisting}

  \end{enumerate}

  We evaluate Metadata score as follows:

  \begin{mdframed}
    \center
Metadata Score = $\frac{\displaystyle \sum_{graphs}^{g}(c(g) + l(g) + p(g) + d(g))}{N}$

where

$N$ = Number of Graphs

$c(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~classes} \\
            25  & {\rm if~g~contains~more~than~zero~classes}
        \end{array}
        \right.$

$l(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~labels} \\
            25  & {\rm if~g~contains~more~than~zero~labels}
        \end{array}
        \right.$

$p(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~properties} \\
            25  & {\rm if~g~contains~more~than~zero~properties}
        \end{array}
        \right.$

$d(g) = \left\{
        \begin{array}{ll}
            0   & {\rm if~g~does~not~contains~any~datatypes} \\
            25  & {\rm if~g~contains~more~than~zero~datatypes}
        \end{array}
        \right.$

  \end{mdframed}

\item Vocabulary Score

  Vocabulary Score, which is calculated based on metadata, represents how many vocabularies data in the endpoint use.

  Vocabulary Score is calculated as follows:

  \begin{mdframed}
    \center

Vocabulary Score = $\frac{\displaystyle \sum_{graphs}^{g}v(g)}{N}$

where

$N$ = Number of Graphs

$v(g)$ = Number of Properties in Graph g

  \end{mdframed}

\item Ontology Score

  Ontology Score, which is calculated based on metadata, represents how much common ontologies data in the endpoint use.

  Ontology Score is calculated as follows:

  \begin{mdframed}
    \center

Vocabulary Score = $\frac{\displaystyle \sum_{graphs}^{g}o(g)}{N}$

where

$N$ = Number of Graphs

$o(g)$ = $\frac{NCO}{NO}$

$NO$ = Number of Ontologies used for Properties

$NCO$ = Number of Ontologies used for Properties in Table \ref{table:common_ontologies}

  \end{mdframed}

  \begin{table}[htbp]
    \center
    \begin{tabular}{|l|}
      \hline
      Ontology URI \\
      \hline
      \url{http://www.w3.org/2000/01/rdf-schema} \\
      \url{http://www.w3.org/1999/02/22-rdf-syntax-ns} \\
      \url{http://www.socrata.com/rdf/terms} \\
      \url{http://www.w3.org/2003/01/geo/wgs84_pos} \\
      \url{http://xmlns.com/foaf/0.1/} \\
      \url{http://www.w3.org/2002/07/owl} \\
      \url{http://purl.org/dc/elements/1.1/} \\
      \url{http://purl.org/dc/terms/} \\
      \url{http://www.w3.org/2000/10/swap/pim/usps} \\
      \url{http://dublincore.org/documents/dcmi-box/} \\
      \url{http://www.territorio.provincia.tn.it/geodati/ontology/} \\
      \url{http://www.w3.org/2004/02/skos/core} \\
      \hline
    \end{tabular}
    \caption{List of Common Ontologies}
    \label{table:common_ontologies}
  \end{table}

\end{itemize}

At last, we evaluate Usefulness Score as follows:

\begin{mdframed}
  \center
  $\begin{array}{lll}
  {\rm Usefulness} & = & 30.0 * {\rm Metadata~Score} \\
                   & + & 40.0 * f10({\rm Vocabulary~Score}) \\
                   & + & 30.0 * {\rm Ontology~Score}
  \end{array}$

  where

  $f10(x) = \left\{
    \begin{array}{ll}
      10 & {\rm if}~x>10 \\
      x  & {\rm Otherwise}
    \end{array}
  \right.$
\end{mdframed}

\subsection{Validity}

  Validity represents how endpoint and data in it obey the rules.
  We measure the two metrics:

  \begin{itemize}

    \item Cool URI

    The URI of endpoints is preferred to be Cool URI\cite{Leo:08:CoolUri}, \cite{Leigh:12:LinkedDataPatterns}.

    We check four criteria:
    \begin{enumerate}
      \item A host of URI of endpoints should not be specified by IP address
      \item A port of URI of endpoints should be 80
      \item A URI of endpoints should not contain query parameters
      \item A length of URI of endpoints should be less than 30 characters
    \end{enumerate}

    Cool URI Score is a percentage of the satisfied rules.

    \item Linked Data Rule

    The endpoints are preferred to be satisfied with the four rules of linked data\cite{Tim:06:LinkedDataRules}.

    We check four criteria:
    \begin{enumerate}
      \item Use URIs as names for things

      We assume all subjects of statements are things. We search invalid statement using a query described in Listing \ref{list:non_uri_subject}, and if nothing is found the endpoint satisfied this rule.

      Note that we ignore Virtuoso specific graphs since Virtuoso contains a graph which contains invalid statements.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for searching non-URI subjects,label=list:non_uri_subject]
SELECT
  *
WHERE {
GRAPH ?g { ?s ?p ?o } .
  filter (!isURI(?s) && !isBLANK(?s) && ?g NOT IN (
    <http://www.openlinksw.com/schemas/virtrdf#>
  ))
}
LIMIT 1
      \end{lstlisting}

      \item Use HTTP URIs so that people can look up those names

      We assume all subjects of statements are things. We search invalid statement using a query described in Listing \ref{list:non_http_uri_subject}, and if nothing is found the endpoint satisfied this rule.

      Note that we ignore Virtuoso specific graphs since Virtuoso contains a graph which contains invalid statements.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for searching non-HTTP-URI subjects,label=list:non_http_uri_subject]
SELECT
  *
WHERE {
  GRAPH ?g { ?s ?p ?o } .
  filter (!regex(?s, "http://", "i") && !isBLANK(?s) && ?g NOT IN (
    <http://www.openlinksw.com/schemas/virtrdf#>
  ))
}
LIMIT 1
      \end{lstlisting}

      \item When someone looks up a URI, provide useful information, using the standards (RDF*, SPARQL)

      We assess this rule by obtaining a subject (URI) using a query described in Listing \ref{list:query_for_a_subject} and accessing the URI via HTTP protocol. We assume that the endpoint is satisfied with the rule if the URI returns any data.

      Note that we ignore Virtuoso specific graphs since Virtuoso contains a graph which contains invalid statements.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for a Subject,label=list:query_for_a_subject]
SELECT
  ?s
WHERE {
  GRAPH ?g { ?s ?p ?o } .
  filter (isURI(?s) && ?g NOT IN (
    <http://www.openlinksw.com/schemas/virtrdf#>
  ))
}
LIMIT 1
OFFSET 100
      \end{lstlisting}

      \item Include links to other URIs. so that they can discover more things

      We assume the statement representing the link to other URI uses the vocabularies owl:sameAs or rdfs:seeAlso. We think if there are any statement of which property is owl:sameAs or rdfs:seeAlso, the endpoint is satisfied with the rule.
      Thus we check the feasibility of the rule by using queries described in Listing \ref{list:query_for_same_as}, \ref{list:query_for_see_also}.

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for a Same AS Statement,label=list:query_for_same_as]
PREFIX owl:<http://www.w3.org/2002/07/owl#>
SELECT
  *
WHERE {
  GRAPH ?g { ?s owl:sameAs ?o } .
}
LIMIT 1
      \end{lstlisting}

      \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for a See Also Statement,label=list:query_for_see_also]
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT
  *
WHERE {
  GRAPH ?g { ?s rdfs:seeAlso ?o } .
}
LIMIT 1
      \end{lstlisting}

    \end{enumerate}

    Linked Data Score is a percentage of the satisfied rules.

  \end{itemize}

  We evaluate Validity as follows:

  \begin{mdframed}
    \center
Validity = 40 * Cool URI Score + 60.0 * Linked Data Rule Score
  \end{mdframed}

\subsection{Performance}

  Performace suggests how powerful the endpoint is.

  We measure the response times of the two queries, Listing \ref{list:query_ask}, \ref{list:query_for_list_of_graphs}. The former query is a most simple query and we use this query to estimate the transfer time. The latter query requires a little computations for endpoints. We believe the execution cost of this query does not differ very much according to the size of data.

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Most Simple Query,label=list:query_ask]
ASK {}
  \end{lstlisting}

  \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true,frame=single,caption=A Query for Listing Graphs,label=list:query_for_list_of_graphs]
SELECT DISTINCT
  ?g
WHERE {
  GRAPH ?g { ?s ?p ?o }
}
  \end{lstlisting}

  We assume the execution time as:

  \begin{mdframed}
    \center
Execution Time = Differences of the response time for those queries.
  \end{mdframed}

  After that, we evaluate Performance as:

  \begin{mdframed}
    \center
Performance = $\left\{
    \begin{array}{ll}
        \multicolumn{2}{l}{100.0 * (1.0 - {\rm Execution~Time})} \\
          & {\rm if~Execution~Time~is~less~than~1~second}  \\
        0 & {\rm Otherwise}
    \end{array}
    \right.$
  \end{mdframed}


\bibliographystyle{plain}
\bibliography{umaka-score}

\end{document}

{'fraction_non_alphanumeric': 0.10679452054794521, 'fraction_numerical': 0.011232876712328766, 'mean_word_length': 3.555791313030454, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 34, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5754835', 'n_tokens_mistral': 12352, 'n_tokens_neox': 11639, 'n_words': 3693}