<commit_before>ifndef::imagesdir[:imagesdir: doc]
Zamia AI Installation and First Steps
=====================================

These are generic installation instructions aimed at technical users and developers
who want to run Zamia AI on GNU/Linux (and similar) systems.

We assume that you want to install Zamia AI as a non-root user in a directory in your
home directory: `~/zamia-ai`. It should be straightforward to adapt these instructions to install
Zamia AI in a different directory inside or outside of your home where you have write permissions
or in a system-wide location like `/usr/local/zamia-ai` or `/opt/zamia-ai` (which then may require
root permissions at some point).

[source,bash]
----
mkdir ~/zamia-ai
cd ~/zamia-ai
----

Requirements
------------

We assume the following packages are installed on your OS:

* Python 2.7 with pip, cython, swig
* tensorflow
* SQLite, python-sqlite 
* libatlas, zlib, libpulse 
* pkg-config, git, gcc, automake, autoconf, libtoolize, subversion, eSpeak NG 
* XSB dependencies: libcurl, libpcre, makedepend, dos2unix

If you want to install all on these on Debian, run (as root):

[source,bash]
----
pip install tensorflow
apt-get install cython swig python-sqlite libatlas-dev libatlas3-base pkg-config git zlib1g-dev automake autoconf libtool subversion libpulse-dev libcurl4-openssl-dev libpcre3-dev xutils-dev dos2unix espeak-ng
----

XSB Prolog with Python Bindings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Download the latest XSB tarball from https://sourceforge.net/projects/xsb/

`py-xsb-prolog` needs the XSB dynamic library to work: First, follow the standard XSB build instructions:

[source,bash]
----
tar xfvz XSB38.tar.gz 
cd XSB/build
./configure
./makexsb 
----

in order to build `libxsb.so`, execute these commands:

[source,bash]
----
./makexsb dynmodule
cat >>~/.bashrc <<EOF
export LD_LIBRARY_PATH="\${HOME}/zamia-ai/XSB/config/x86_64-unknown-linux-gnu/bin:\$LD_LIBRARY_PATH"
EOF
source ~/.bashrc
----

now, we can install the prolog bindings:

[source,bash]
----
pip install --user py-xsb-prolog
----

Kaldi ASR with Python Bindings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Clone the latest Kaldi ASR sources from gibthub, follow the regular build instructions:
[source,bash]
----
cd ~/zamia-ai/
git clone https://github.com/kaldi-asr/kaldi.git
cd kaldi/tools
make
cd ../src
./configure --shared
make depend
make
----

next, we need to create a package-config file which describes where Kaldi ASR is installed:
[source,bash]
----
cd ~/zamia-ai/kaldi
mkdir pkgconfig
cat >pkgconfig/kaldi-asr.pc <<EOF
kaldi_root=/home/bofh/zamia-ai/kaldi

Name: kaldi-asr
Description: kaldi-asr speech recognition toolkit
Version: 5.2
Libs: -L\${kaldi_root}/tools/openfst/lib -L\${kaldi_root}/src/lib -lkaldi-decoder -lkaldi-lat -lkaldi-fstext -lkaldi-hmm -lkaldi-feat -lkaldi-transform -lkaldi-gmm -lkaldi-tree -lkaldi-util -lkaldi-matrix -lkaldi-base -lkaldi-nnet3 -lkaldi-online2<commit_msg>fix typo<commit_after>ifndef::imagesdir[:imagesdir: doc]
Zamia AI Installation and First Steps
=====================================

These are generic installation instructions aimed at technical users and developers
who want to run Zamia AI on GNU/Linux (and similar) systems.

We assume that you want to install Zamia AI as a non-root user in a directory in your
home directory: `~/zamia-ai`. It should be straightforward to adapt these instructions to install
Zamia AI in a different directory inside or outside of your home where you have write permissions
or in a system-wide location like `/usr/local/zamia-ai` or `/opt/zamia-ai` (which then may require
root permissions at some point).

[source,bash]
----
mkdir ~/zamia-ai
cd ~/zamia-ai
----

Requirements
------------

We assume the following packages are installed on your OS:

* Python 2.7 with pip, cython, swig
* tensorflow
* SQLite, python-sqlite 
* libatlas, zlib, libpulse 
* pkg-config, git, gcc, automake, autoconf, libtoolize, subversion, eSpeak NG 
* XSB dependencies: libcurl, libpcre, makedepend, dos2unix

If you want to install all on these on Debian, run (as root):

[source,bash]
----
pip install tensorflow
apt-get install cython swig python-sqlite libatlas-dev libatlas3-base pkg-config git zlib1g-dev automake autoconf libtool subversion libpulse-dev libcurl4-openssl-dev libpcre3-dev xutils-dev dos2unix espeak-ng
----

XSB Prolog with Python Bindings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Download the latest XSB tarball from https://sourceforge.net/projects/xsb/

`py-xsb-prolog` needs the XSB dynamic library to work: First, follow the standard XSB build instructions:

[source,bash]
----
tar xfvz XSB38.tar.gz 
cd XSB/build
./configure
./makexsb 
----

in order to build `libxsb.so`, execute these commands:

[source,bash]
----
./makexsb dynmodule
cat >>~/.bashrc <<EOF
export LD_LIBRARY_PATH="\${HOME}/zamia-ai/XSB/config/x86_64-unknown-linux-gnu/bin:\$LD_LIBRARY_PATH"
EOF
source ~/.bashrc
----

now, we can install the prolog bindings:

[source,bash]
----
pip install --user py-xsb-prolog
----

Kaldi ASR with Python Bindings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Clone the latest Kaldi ASR sources from github, follow the regular build instructions:
[source,bash]
----
cd ~/zamia-ai/
git clone https://github.com/kaldi-asr/kaldi.git
cd kaldi/tools
make
cd ../src
./configure --shared
make depend
make
----

next, we need to create a package-config file which describes where Kaldi ASR is installed:
[source,bash]
----
cd ~/zamia-ai/kaldi
mkdir pkgconfig
cat >pkgconfig/kaldi-asr.pc <<EOF
kaldi_root=/home/bofh/zamia-ai/kaldi

Name: kaldi-asr
Description: kaldi-asr speech recognition toolkit
Version: 5.2
Libs: -L\${kaldi_root}/tools/openfst/lib -L\${kaldi_root}/src/lib -lkaldi-decoder -lkaldi-lat -lkaldi-fstext -lkaldi-hmm -lkaldi-feat -lkaldi-transform -lkaldi-gmm -lkaldi-tree -lkaldi-util -lkaldi-matrix -lkaldi-base -lkaldi-nnet3 -lkaldi-online2
{'fraction_non_alphanumeric': 0.14290586630286495, 'fraction_numerical': 0.006139154160982265, 'mean_word_length': 4.091145833333333, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5577970', 'n_tokens_mistral': 2115, 'n_tokens_neox': 1962, 'n_words': 650}