RITE is a generic benchmark task that addresses major text understanding needs in various NLP/Information Access research areas. We will evaluate systems which automatically detect entailment, paraphrase, and contradiction in texts written in Japanese, Simplified Chinese, or Traditional Chinese. In BC subtask, an input text pair looks as follows:
- t1: Yasunari Kawabata won the Nobel Prize in Literature for his novel "Snow Country"
- t2: Yasunari Kawabata is the writer of "Snow Country"
A system is given a credit for this pair if it returns the same label as human assessors'; in this specific case, "Y" (meaning "Yes", a human reading the premise t1 would infer that the hypothesis t2 is most likely true).
NTCIR-9 RITE Subtasks
- Binary-class (BC) subtask - Given a text pair (t1,t2) identify whether t1 entails (infers) a hypothesis t2 or not.
- Multi-class (MC) subtask - 5-way labeling task to detect (forward / reverse / bidirection) entailment or no entailment (contradiction / independence) in a text pair.
- Entrance Exam subtask (Japanese Only) - Same as the BC subtask in terms of input and output. All the data is created from actual college-level entrance exams.
- RITE4QA subtask - Same as the BC subtask in terms of input and output, but as an embedded answer validation component in Question Answering system. This way, the impact of RITE to an overall end-to-end application can be measured.
Who should participate?
RITE is a generic task that addresses major Text Understanding needs in various research areas e.g. Information Retrieval, Question Answering, Text Summarization, Opinion Analysis. Researchers in Natural Language Processing, Machine Learning community are also very welcome. Since RITE is a friendly evaluation effort, do not hesitate to try ambitious/exploratory approaches that do not necessarily work. A uniqueness in your method would also be a big contribution to the community. Don't worry about the lack of language resource/tool to participate. We will build the Resource Pool on this wiki, so that useful resources can be shared among participants, and they can be used to build a system for the formal evaluation.
- Task Guideline defines the task in more detail, such as input/output format.
- Data - sample data, development data (and formal data during the formal run period) are downloadable.
- Resource Pool contains useful material for participating in RITE.
- Detailed Schedule - contains detailed schedule.
- RITE-SDK - a Java framework which enables one to build a RITE system easily.
- Formal run for all 4 subtasks are done. We received nearly 200 runs from 24 teams, which is one of the largest numbers in a single NTCIR task ever! The NTCIR-9 workshop has been held in Dec 2011. You can access an online proceedings here.
If you are interested in the task, join our mailing list by sending an empty email to ntc9-rite-list-subscribe [at] cs.cmu.edu
Email ntc9-rite-organizers [at] cs.cmu.edu if you have any questions.