<commit_before># FelisCatus Zero-multilingual 
[![Build Status](https://travis-ci.org/ktr-skmt/FelisCatusZero-multilingual.svg?branch=master)](https://travis-ci.org/ktr-skmt/FelisCatusZero-multilingual)
[![codecov](https://codecov.io/gh/ktr-skmt/FelisCatusZero-multilingual/branch/master/graph/badge.svg)](https://codecov.io/gh/ktr-skmt/FelisCatusZero-multilingual)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/FelisCatusZero-multilingual/Lobby?utm_source=share-link&utm_medium=link&utm_campaign=share-link)

This system answers world history essay questions in Japanese and/or English and evaluate the answers.  

(past version: <a href="https://github.com/ktr-skmt/FelisCatusZero">FelisCatus Zero</a>.)

## Trial

1. Install Scala and Simple Build Tool (SBT).
1. Git-clone or download this repository.
1. Run <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/blob/master/trial.sh">trial.sh</a>
```bash
$ bash trial.sh
```

## Quick Start Guide
### (1) Software Installation for FelisCatus Zero Running
Install the following softwares before running (non-trial) FelisCatus Zero
* Scala
* Simple Build Tool (SBT)
* Indri search engine. See <a href="https://github.com/ktr-skmt/FelisCatusZero/wiki/Install-Guide">here</a>  
* Japanese language analyzer (MeCab). See <a href="https://github.com/ktr-skmt/FelisCatusZero/wiki/Install-Guide">here</a>  

Note: I am planning to make remote services to enable to run without local Indri and MeCab.

### (2) Knowledge Source Installation
See [Knowledge Source Installation](tutorial/KnowledgeSourceInstallation.md)

Note: Knowledge source accessible people are currently limited because of a license issue.  
Note: I am planning to make remote services to enable to run without local knowledge sources.

### (3) Download FelisCatus Zero
Git-clone or download this repository

Git-clone via HTTPS
```bash
cd (directory for downloading this repository)
git clone https://github.com/ktr-skmt/FelisCatusZero-multilingual.git
```
Git-clone via SSH
```bash
cd (directory for downloading this repository)
git clone git<EMAIL>:ktr-skmt/FelisCatusZero-multilingual.git
```
Download
```bash
cd (directory for downloading this repository)
wget https://github.com/ktr-skmt/FelisCatusZero-multilingual/archive/master.zip
```

### (4) Question-and-gold-standard Pair Installation
Sample questions have already been installed in this repository.

If you want to Install other pairs of question and gold standard, do the following 1. and/or 2.

1. Put QA Lab-3 essay subtask datasets into <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/tree/master/src/main/resources/qalab_dataset">src/main/resources/qalab_dataset</a>
1. Put QA Corpus into <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/tree/master/src/main/resources/qa_corpus">src/main/resources/qa_courpus</a>

Note: I am planning to make remote services to enable to download training data and test data without installing them locally.

### (5) How to Run
#### Run through SBT Shell (Recommended)
See [How to Run through SBT Shell](tutorial/HowToRunThroughSBTShell.md)

#### Run through SBT Batch Mode
See <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/blob/master/tutorial/Run.ipynb">Run.ipynb</a>

## Libraries
FelisCatusZero-multilingual has three libraries as sub-projects:
* [FelisCatus Zero-Libraries](libraries/maven-repo)
* [FelisCatus Zero-Libraries4UIMA](libraries4uima/maven-repo)
* [FelisCatus Zero-Libraries4JCas](libraries4jcas/maven-repo)

FelisCatusZero-multilingual is dependent on FelisCatus Zero-Libraries4JCas, FelisCatus Zero-Libraries4UIMA and JCas generated from [FelisCatusZero's type system](src/main/resources/desc/ts) by [JCasGen](jcasgen).
FelisCatus Zero-Libraries4JCas is dependent on FelisCatus Zero-Libraries and FelisCatus Zero-Libraries4UIMA.
The three libraries' dependencies are shown in the following table.

LIBRARY | DEPENDENT on UIMA | DEPENDENT on JCas generated from FelisCatusZero's type system
---|---|---
FelisCatus Zero-Libraries | NO | NO
FelisCatus Zero-Libraries4UIMA | YES | NO
FelisCatus Zero-Libraries4JCas | YES | YES

See [build.sbt](build.sbt) if you want to look in more detail.
<commit_msg>bug fix for indri indexer<commit_after># FelisCatus Zero-multilingual 
[![Build Status](https://travis-ci.org/ktr-skmt/FelisCatusZero-multilingual.svg?branch=master)](https://travis-ci.org/ktr-skmt/FelisCatusZero-multilingual)
[![codecov](https://codecov.io/gh/ktr-skmt/FelisCatusZero-multilingual/branch/master/graph/badge.svg)](https://codecov.io/gh/ktr-skmt/FelisCatusZero-multilingual)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/FelisCatusZero-multilingual/Lobby?utm_source=share-link&utm_medium=link&utm_campaign=share-link)

This system answers world history essay questions in Japanese and/or English and evaluate the answers.  

(past version: <a href="https://github.com/ktr-skmt/FelisCatusZero">FelisCatus Zero</a>.)

## Trial

1. Install Scala and Simple Build Tool (SBT).
1. Git-clone or download this repository.
1. Run <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/blob/master/trial.sh">trial.sh</a>
```bash
$ bash trial.sh
```

## Quick Start Guide
### (1) Software Installation for FelisCatus Zero Running
Install the following softwares before running (non-trial) FelisCatus Zero
* Scala
* Simple Build Tool (SBT)
* Indri search engine. See <a href="https://github.com/ktr-skmt/FelisCatusZero/wiki/Install-Guide">here</a>  
* Japanese language analyzer (MeCab). See <a href="https://github.com/ktr-skmt/FelisCatusZero/wiki/Install-Guide">here</a>  

Note: I am planning to make remote services to enable to run without local Indri and MeCab.

### (2) Knowledge Source Installation
See [Knowledge Source Installation](tutorial/KnowledgeSourceInstallation.md)

Note: Knowledge source accessible people are currently limited because of a license issue.  
Note: I am planning to make remote services to enable to run without local knowledge sources.

### (3) Download FelisCatus Zero
Git-clone or download this repository

Git-clone via HTTPS
```bash
cd (directory for downloading this repository)
git clone https://github.com/ktr-skmt/FelisCatusZero-multilingual.git
```
Git-clone via SSH
```bash
cd (directory for downloading this repository)
git clone <EMAIL>:ktr-skmt/FelisCatusZero-multilingual.git
```
Download
```bash
cd (directory for downloading this repository)
wget https://github.com/ktr-skmt/FelisCatusZero-multilingual/archive/master.zip
```

### (4) Question-and-gold-standard Pair Installation
Sample questions have already been installed in this repository.

If you want to Install other pairs of question and gold standard, do the following 1. and/or 2.

1. Put QA Lab-3 essay subtask datasets into <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/tree/master/src/main/resources/qalab_dataset">src/main/resources/qalab_dataset</a>
1. Put QA Corpus into <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/tree/master/src/main/resources/qa_corpus">src/main/resources/qa_courpus</a>

Note: I am planning to make remote services to enable to download training data and test data without installing them locally.

### (5) How to Run
#### Run through SBT Shell (Recommended)
See [How to Run through SBT Shell](tutorial/HowToRunThroughSBTShell.md)

#### Run through SBT Batch Mode
See <a href="https://github.com/ktr-skmt/FelisCatusZero-multilingual/blob/master/tutorial/Run.ipynb">Run.ipynb</a>

## Libraries
FelisCatusZero-multilingual has three libraries as sub-projects:
* [FelisCatus Zero-Libraries](libraries/maven-repo)
* [FelisCatus Zero-Libraries4UIMA](libraries4uima/maven-repo)
* [FelisCatus Zero-Libraries4JCas](libraries4jcas/maven-repo)

FelisCatusZero-multilingual is dependent on FelisCatus Zero-Libraries4JCas, FelisCatus Zero-Libraries4UIMA and JCas generated from [FelisCatusZero's type system](src/main/resources/desc/ts) by [JCasGen](jcasgen).
FelisCatus Zero-Libraries4JCas is dependent on FelisCatus Zero-Libraries and FelisCatus Zero-Libraries4UIMA.
The three libraries' dependencies are shown in the following table.

LIBRARY | DEPENDENT on UIMA | DEPENDENT on JCas generated from FelisCatusZero's type system
---|---|---
FelisCatus Zero-Libraries | NO | NO
FelisCatus Zero-Libraries4UIMA | YES | NO
FelisCatus Zero-Libraries4JCas | YES | YES

See [build.sbt](build.sbt) if you want to look in more detail.
{'fraction_non_alphanumeric': 0.11826967326277037, 'fraction_numerical': 0.007132995858260469, 'mean_word_length': 5.410766961651918, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 34, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6826304', 'n_tokens_mistral': 3005, 'n_tokens_neox': 2834, 'n_words': 730}