We are reporting the SHINRA project, a project for structuring Wikipedia with collaborative construction scheme. The goal of the project is to create a huge and well-structured knowledge base to be used in NLP applications, such as QA, Dialogue systems and explainable NLP systems. It is created based on a scheme of ”Resource by Collaborative Contribution (RbCC)”. We conducted a shared task of structuring Wikipedia, and at the same, submitted results are used to construct a knowledge base.
 There are machine readable knowledge bases such as CYC, DBpedia, YAGO, Freebase Wikidata and so on, but each of them has problems to be solved. CYC has a coverage problem, and others have a coherence problem due to the fact that these are based on Wikipedia and/or created by many but inherently incoherent crowd workers. In order to solve the later problem, we started a project for structuring Wikipedia using automatic knowledge base construction shared-task.
 The automatic knowledge base construction shared-tasks have been popular and well studied for decades. However, these tasks are designed only to compare the performances of different systems, and to find which system ranks the best on limited test data. The results of the participated systems are not shared and the systems may be abandoned once the task is over.
 We believe this situation can be improved by the following changes:
 1. designing the shared-task to construct knowledge base rather than evaluating only limited test data
 2. making the outputs of all the systems open to public so that we can run ensemble learning to create the better results than the best systems
 3. repeating the task so that we can run the task with the larger and better training data from the output of the previous task (bootstrapping and active learning)
 We conducted “SHINRA2018” with the above mentioned scheme and in this paper
 we report the results and the future directions of the project. The task is to extract the values of the pre-defined attributes from Wikipedia pages. We have categorized most of the entities in Japanese Wikipedia (namely 730 thousand entities) into the 200 ENE categories. Based on this data, the shared-task is to extract the values of the attributes from Wikipedia pages. We gave out the 600 training data and the participants are required to submit the attribute-values for all remaining entities of the same category type. Then 100 data out of them for each category are used to evaluate the system output in the shared-task.
 We conducted a preliminary ensemble learning on the outputs and found 15 F1 score improvement on a category and the average of 8 F1 score improvements on all 5 categories we tested over a strong baseline. Based on this promising results, we decided to conduct three tasks in 2019; multi-lingual categorization task (ML), extraction for the same 5 categories in Japanese with a larger training data (JP-5) and extraction for 34 new categories in Japanese (JP-34).
 Wikipedia is a great resource as a knowledge base of the entities in the world. However, Wikipedia is created for human to read rather than machines to process. Our goal is to transform the current Wikipedia to a machine readable format based on a clean structure. There are several machine readable knowledge bases (KB) such as CYC BID4 , DBpedia BID3 , YAGO BID7 , Freebase BID0 , Wikidata BID13 and so on, but each of them has problems to be solved. CYC has a coverage problem, and others have a coherence problem due to the fact that these are based on Wikipedia and/or created by many but inherently incoherent crowd workers. In order to solve these problems, we started a project for structuring Wikipedia using automatic knowledge base construction (AKBC) shared-task using a cleaner ontology definition.The automatic knowledge base construction shared-tasks have been popular for decades. In particular, there are popular shared-tasks in the field of Information Extraction, Knowledge Base population and attribute extraction, such as KBP[U.S. National Institute of Standards and Technology (NIST) , 2018] and CoNLL. However, most of these tasks are designed only to compare the performances of participated systems, and to find which system ranks the best on limited test data. The outputs of the participated systems are not shared and the results and the systems may be abandoned once the evaluation task is over.We believe this situation can be improved by the following changes:1. designing the shared-task to construct knowledge base rather than only evaluating on limited test data 2. making the outputs of all the systems open to public so that anyone can run ensemble learning to create the better results than the best single system 3. repeating the task so that we can run the task with the larger and better training data from the output of the previous task (active learning and bootstrapping)We conducted "SHINRA2018" with the aforementioned ideas, we call it "Resource by Collaborative Contribution (RbCC)". In this paper we report the first results and the future directions of the project.The task is to extract the values of the pre-defined attributes from Wikipedia entity pages. We used Extended Named Entity (ENE) as the definition of the category (in total 200 categories in the ontology) and the attributes (average of 20 attributes) for each category. We have categorized most of the entities in Japanese Wikipedia (namely 730 thousand entities) into the ENE categories prior to this project. Based on this data, the sharedtask is to extract values of the attributes defined for the category of each entity. At the SHINRA2018 project, we limited the target categories to 5, namely, person, company, city, airport and chemical compound. We gave out the 600 training data each for 5 categories at and the participants are supposed to submit the attribute-values for all remaining entities of the categories in Japanese Wikipedia. Then 100 data out of the entire pages of the category are used at the evaluation of the participated systems in the shared-task. For example, there are about 200K person entities in Japanese Wikipedia, and the participants have to extract the attribute-values, such as "birthday", "the organizations he/she have belonged", "mentor" and "awards" from all the remaining entities (i.e. 199.4K = 200K-600 entities). Before starting the project, the participants signed the contract that all the output will be shared among all participants, so that anyone can conduct the ensemble learning on those outputs, and hence create a better knowledge base than the best system in the task. Note that, for the sake of participant's interest, i.e. a company may want to keep the system as their property, the outputs are required to be shared, but their systems are not necessarily to be shared. A promising results of the ensemble learning is achieved and we envision that it will lead to the cleaner machine readable knowledge base construction. We proposed a scheme of knowledge base creation: "Resource by Collaborative Contribution". We conducted the Japanese Wikipedia structuring project, SHINRA2018, based on that scheme. Based on Extended Named Entity, the top-down definition of categories and attributed for named entities, the task is to extract the attribute-values from Japanese Wikipedia pages. 8 groups participated to the task, and the ensemble learning results shows that the RbCC scheme is practical and promising. A quite big improvement over the the best single system was achieved on "airport" category (more than 15 F-score), and the average of 8 F-score improvement was achieved using the weighted voting methods. We are planning to conduct SHINRA2019 based on the RbCC scheme on 3 tasks. These are the multi-lingual categorization, the extraction of attribute-value on the same 5 categories, and the extraction of attribute-values on 30 new categories in Japanese.We'd like to express our deep appreciation to all the participants and collaborators who helped this project. Without the participation, we couldn't even try the ensemble learning and achieve the goal. We are hoping to expand and spread the idea of RbCC scheme, not only limited to this kind of task and resource.
We introduce a "Resource by Collaborative Construction" scheme to create KB, structured Wikipedia 
{"src": "scitldr-AIC_train", "paper_id": "HygfXWqTpm"}