the usability of the tool.

The scores range was on a scale of 1 (very easy/rapid/usable) to 5 (very difficult/slow/unusable).
To evaluate the general level of user satisfaction, we added to the survey a section containing the SUS-System Usability Scale questionnaire [1]}.
Before starting the test, the users were asked to perform a brief tutorial defined for the tool.
The tutorial presents the user the three visualization levels (ODP level, exploration level, visualization level) and, in each of them, it shows the position of the filters.
We measured the actual time occurred for performing each task and asked the participants to record their screen and send us the recording. The latter was useful for understanding the reason of possible failures or slowdowns.
10 of the 11 participants agreed to record their screen and share the recording.
The results of the evaluation according to the SUS scale are presented in Figure REF . They show that the tool is considered excellent by more than half of the users and good by the remaining ones except one, whose judgment was poor. The average score was \(81.4\) , with a standard deviation of \(12.6\) .
<FIGURE>Assessment of the visualization level. A first group of tasks, reported in Table REF , was designed to evaluate the clarity of the visualization level (therefore assessing the hypothesis H4).
In order to compare ODPReactor with existing tools we asked each user to perform the same task also using LodView (a widely known KGs viewer adopted for ArCo).
We considered one task per ODP View.
Each task was executed twice per tool in order to measure also the improvement with the experience. For each execution the user was asked questions about the dimensions of the ODP.
We considered as evaluation metric the number of correct answers, the time of execution and the users judgment of usability.
An answer has been evaluated as correct if the user correctly identified the entry containing the data, although the answer is only partial.
For example in tasks TITL1 and TITL2, for little-known cities the users had difficulties in distinguishing the city from the village or from the county (all reported in the same entry). Quanti sono?
Therefore we considered all such answers as correct. The results are reported in Figure REF .
Figure REF  shows the number of correct task executions. ODPReactor always performs similarly or better than LodView. Figure REF  shows the average task execution time. ODPReactor always performs better than LodView, with the exception of TITL 1, which regards retrieving the location of a cultural property.
By analyzing the recordings it emerges that the users have quickly identified the data, but have delayed the transcription, because of difficulties in distinguishing the town (specifically Caravino and Vittorio Veneto) from the village or from the province. The same behaviour is not reflected in the LodView tasks, since the answers consist in widely-known cities (specifically Rome and Turin).
Figure REF  shows the user judgment of the tool, where ODPReactor was considered significantly more adequate than LodView. This result is emphasized for the Measurement Collection tasks, where ODPReactor was evaluated unanimuosly very adequate while the judgment for LodView was almost equally distributed from Very adequate to Inadequate.
<TABLE>Assessment of the ODP and exploration level.
In order to assess H1-3 hypotheses we designed a group of tasks involving the ODP level and exploration level. These tasks, reported in Table REF , are designed for evaluating the benefits of the approach in streamline the navigation and search of information in the KG.
All the tasks required to identify the number of occurrences in the KG satisfying a given searching criterion.
The first two tasks (referred to Measurement collection ODP and Cultural property component of ODP) can be performed by applying just one filter, while the last three ones (referred to Time indexed typed location ODP) require the activation of more than one filter. We chose the ODP Time Indexed Typed Location for the multi-filter tasks since this pattern is more complex then the others and hence it has more filter controls.
To evaluate the general level of user satisfaction, we asked the users to answer a survey defined according to the SUS-System Usability Scale questionnaire [1]}.
<TABLE><FIGURE>Finally, we report on the results of the exploratory research tasks on ODPReactor. We did not compare this task with LodView since it does not provide exploratory search functionalities. In counting the number of occurrences that satisfy the given constraints, we did not make any assumption about open or close world, and considered correct both the answers that include or exclude objects that do not contain some of the properties necessary for applying the filter.
Results are summed up in Figure REF . We collected 55 (\(11 \times 5\) ) responses by aggregating correctness, usability and difficulty, from all tasks and all users. Most of the users performed the task correctly (Fig. REF ). We analyzed the recordings for understanding the reasons of failure or difficulties in executing the task. It emerged that among 10 wrong answers, 9 ones are due to failures to apply all filters necessary for the complex multi-filter tasks, while for the other one the data was correctly identified but wrongly reported.
Specifically, the testers wrongly set the filter of the Time-indexed Typed Location instances for selecting only those having a certain type (e.g. Current Location).
This indicates that the usage of such filter was not intuitive for the testers and it has to redesigned.
Finally, Figure REF  shows the result of the questions about usability of the tool.
Most of the users judged the tool very adequate or adequate. 51 answers out of 55 were positive or intermediate.
Figure REF  shows similar positive results about difficulty.
<FIGURE>Conclusions and Future Work
This work proposed a novel approach to Knowledge Graph visualization that uses Ontology Design Patterns as first-class citizens for accessing and navigating KGs.
We described a general framework that enables reusing an ODP-related visual frame to different KGs and implemented a tool for validating the concept.
The results of the execution of a set of tasks on the tool demonstrated the validity of the proposed approach, confirmed by considering both objective parameters and user perspective.
In future we plan to enrich our library of visual frames with new ODP-based views and to explore general methods for automatizing the annotation of pattern instances.
We also plan to study an automatic system for building filter facets and to make an extensive evaluation on a larger number of users.
