<commit_before>nihongoparserd
===========

A service to convert japanese languages into romaji, kana etc.

for the moment it's based on libmecab but it may be replaced by homebrew parser in the future

### Running under Docker

Since getting this code to compile under macOS (the use of `strdupa` from the
GNU C++ extensions may be hard or impossible under clang++), this repo contains
a Dockerfile:

```
$ docker build -t nihongoparserd .
$ docker run -p 8842:8842 nihongoparserd
```

You can then hit the service at
[localhost:8842/parse](http://localhost:8842/parse). However, at the present
time, it looks like adding the require `str` query param results in an immediate
segfault.

### Requirement ###

  * libexpat
  * libevent
  * libmecab2
  * cmake

a C++11 compatible compiler

### Usage ###

    ./nihonggoparserd -p PORT [-h HOSTNAME]

That will launch an HTTP server listening on port PORT, eventually HOSTNAME.
It provides the following API calls, that will return an XML answer.
Note that contents are wrapped into `<![CDATA[...]]>`, which is removed from the following examples for readability.

#### Parse ####<commit_msg>make it interactive to give a chance of killing it<commit_after>nihongoparserd
===========

A service to convert japanese languages into romaji, kana etc.

for the moment it's based on libmecab but it may be replaced by homebrew parser in the future

### Running under Docker

Since getting this code to compile under macOS (the use of `strdupa` from the
GNU C++ extensions may be hard or impossible under clang++), this repo contains
a Dockerfile:

```
$ docker build -t nihongoparserd .
$ docker run -t -i -p 8842:8842 nihongoparserd
```

You can then hit the service at
[localhost:8842/parse](http://localhost:8842/parse). However, at the present
time, it looks like adding the require `str` query param results in an immediate
segfault.

### Requirement ###

  * libexpat
  * libevent
  * libmecab2
  * cmake

a C++11 compatible compiler

### Usage ###

    ./nihonggoparserd -p PORT [-h HOSTNAME]

That will launch an HTTP server listening on port PORT, eventually HOSTNAME.
It provides the following API calls, that will return an XML answer.
Note that contents are wrapped into `<![CDATA[...]]>`, which is removed from the following examples for readability.

#### Parse ####
{'fraction_non_alphanumeric': 0.0947867298578199, 'fraction_numerical': 0.016372253339077984, 'mean_word_length': 4.253393665158371, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1903702', 'n_tokens_mistral': 720, 'n_tokens_neox': 665, 'n_words': 317}