<commit_before>libkrisp Database management
===

KRISP database used two external databases. One is ***[KRNIC(KISA)](http://kisa.or.kr)*** isp database and another is GeoLiteCountry database in [Maxmind](http://maxmind.com), available from http://maxmind.com.

## Database update cycle

update database every month, between 7th and 14th.


## Database download

Get ***krisp.csv.gz*** or ***krisp-geoip.csv.gz*** at follow url:

     http://mirror.oops.org/pub/oops/libkrisp/data/v2/

Each database has features as follows:

 * ***krisp.csv.gz***
   * only Korea allocated information
   * enable ISP inforamtion
   * small size (< 1MB)
 * ***krisp-geoip.csv.gz***
   * world wide Country information
   * only Korean ISP information eanbled
   * big size (>20MB)

## Database building

First, download ***krisp*** database

```bash
    [root@host ~]$ # blocking wget user agent, be carefull.
    [root@host ~]$ wget -U mozilla 'http://mirror.oops.org/pub/oops/libkrisp/data/v2/krisp-geoip.csv.gz`
```

And, uncompress download database.

```bash
    [root@host ~]$ gzip -d krisp-geoip.csv.gz
```

And, build database with libkrisp scheme in libkrisp source

The scheme file is located in current directory by named 'table'

```bash
    [root@host ~]$ # make scheme
    [root@host ~]$ cat <<EOF > scheme
    CREATE TABLE krisp
    (
        start unsigned integer NOT NULL DEFAULT '0',
        end unsigned integer NOT NULL DEFAULT '0',
        data varchar,
        PRIMARY KEY (start DESC),
        UNIQUE (end)
    );

    .sep "\t"
    .imp krisp-geoisp.csv krisp
    EOF
    [root@host ~]$ sqlite3 krisp-geoip.dat < ./table
```

Last, move data file to libkrisp data directory. ($prefix/share/krisp)<commit_msg>fixed typo<commit_after>libkrisp Database management
===

KRISP database used two external databases. One is ***[KRNIC(KISA)](http://kisa.or.kr)*** isp database and another is GeoLiteCountry database in [Maxmind](http://maxmind.com), available from http://maxmind.com.

## Database update cycle

update database every month, between 7th and 14th.


## Database download

Get ***krisp.csv.gz*** or ***krisp-geoip.csv.gz*** at follow url:

     http://mirror.oops.org/pub/oops/libkrisp/data/v2/

Each database has features as follows:

 * ***krisp.csv.gz***
   * only Korea allocated information
   * enable ISP inforamtion
   * small size (< 1MB)
 * ***krisp-geoip.csv.gz***
   * world wide Country information
   * only Korean ISP information eanbled
   * big size (>20MB)

## Database building

First, download ***krisp*** database

```bash
    [root@host ~]$ # blocking wget user agent, be carefull.
    [root@host ~]$ wget -U mozilla 'http://mirror.oops.org/pub/oops/libkrisp/data/v2/krisp-geoip.csv.gz`
```

And, uncompress download database.

```bash
    [root@host ~]$ gzip -d krisp-geoip.csv.gz
```

And, build database with libkrisp scheme in libkrisp source

The scheme file is located in current directory by named 'table'

```bash
    [root@host ~]$ # make scheme
    [root@host ~]$ cat <<EOF > scheme
    CREATE TABLE krisp
    (
        start unsigned integer NOT NULL DEFAULT '0',
        end unsigned integer NOT NULL DEFAULT '0',
        data varchar,
        PRIMARY KEY (start DESC),
        UNIQUE (end)
    );

    .sep "\t"
    .imp krisp-geoisp.csv krisp
    EOF
    [root@host ~]$ # build krisp-geoip.dat
    [root@host ~]$ sqlite3 krisp-geoip.dat < ./scheme
```

Last, move data file to libkrisp data directory. ($prefix/share/krisp)
{'fraction_non_alphanumeric': 0.1341779441405125, 'fraction_numerical': 0.006334581053843939, 'mean_word_length': 3.283600493218249, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6905412', 'n_tokens_mistral': 1223, 'n_tokens_neox': 1149, 'n_words': 353}