<commit_before>.. Getting Started describes how to build and install FogLAMP

.. |br| raw:: html

   <br />

.. Images
.. |foglamp_all_round| image:: images/foglamp_all_round_solution.jpg

.. Links
.. _FogLAMP project on GitHub: https://github.com/foglamp/FogLAMP/issues

.. Links in new tabs
.. |FogLAMP Repo| raw:: html

   <a href="https://github.com/foglamp/FogLAMP" target="_blank">https://github.com/foglamp/FogLAMP</a>

.. |GCC Bug| raw:: html

   <a href="https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66425" target="_blank">here</a>

.. =============================================


***************
Getting Started
***************

Let's get started! In this chapter we will see where to find and how to build, install and run FogLAMP for the first time.


FogLAMP Platforms
=================

Due to the use of standard libraries, FogLAMP can run on a large number of platforms and operating environments, but its primary target is Linux distributions. |br| Our testing environment includes Ubuntu LTS 16.04 and Ubuntu Core 16.04, but we have installed and tested FogLAMP on other Linux distributions such as CentOS and Raspbian. In addition to the native support, FogLAMP can also run on Virtual Machines, Docker and LXC containers. |br| FogLAMP development Ubuntu 16.04 and later additions.


General Requirements
--------------------

This version of FogLAMP requires the following software to be installed in the same environment:
- Python 3.5+
- PostgreSQL 9.5+

The requirements largely depend on the plugins that run in FogLAMP, but Python and PostgreSQL are essential for the Core and Storage microservices and tasks.


Building FogLAMP
================

In this section we will describe how to build FogLAMP on Ubuntu 16.04.3 LTS (Server or Desktop).  Other Linux distributions, Debian or Red-Hat based, or even other versions of Ubuntu may differ. If you are not familiar with Linux and you do not want to build FogLAMP from the source code, you can download a snap package from Snappy.

Build Pre-Requisites
--------------------

FogLAMP is currently based on C/C++ and Python code. The packages needed to build and run FogLAMP are:

- cmake, g++, make
- liboost-dev, liboost-system-dev, liboost-thread-dev, libpq-dev
- python3-pip
- postgresql

.. code-block:: console

  $ sudo apt update
  Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
  ...
  All packages are up-to-date.
  $
  $ sudo apt install cmake g++ make
  Reading package lists... Done
  Building dependency tree
  ...
  $
  $ sudo apt-get install libboost-dev libboost-system-dev libboost-thread-dev libpq-dev
  Reading package lists... Done
  Building dependency tree
  ...
  $
  $ sudo apt-get install python3-pip
  Reading package lists... Done
  Building dependency tree
  ...
  $
  $ sudo apt install postgresql
  Reading package lists... Done
  Building dependency tree
  $


Setting the PostgreSQL Database
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In this version of FogLAMP the PostgreSQL database is the default storage engine, used by the Storage microservice. Make sure that PostgreSQL is installed and running correctly:

.. code-block:: console

  $ sudo systemctl status postgresql
  ● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2017-12-08 15:56:07 GMT; 15min ago
   Main PID: 14572 (code=exited, status=0/SUCCESS)
     CGroup: /system.slice/postgresql.service

  Dec 08 15:56:07 ubuntu systemd[1]: Starting PostgreSQL RDBMS...
  Dec 08 15:56:07 ubuntu systemd[1]: Started PostgreSQL RDBMS.
  Dec 08 15:56:11 ubuntu systemd[1]: Started PostgreSQL RDBMS.
  $
  $ ps -ef | grep postgres
  postgres 14806     1  0 15:56 ?        00:00:00 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
  postgres 14808 14806  0 15:56 ?        00:00:00 postgres: checkpointer process
  postgres 14809 14806  0 15:56 ?        00:00:00 postgres: writer process
  postgres 14810 14806  0 15:56 ?        00:00:00 postgres: wal writer process
  postgres 14811 14806  0 15:56 ?        00:00:00 postgres: autovacuum launcher process
  postgres 14812 14806  0 15:56 ?        00:00:00 postgres: stats collector process
  ubuntu   15198  1225  0 17:22 pts/0    00:00:00 grep --color=auto postgres
  $

When you install the Ubuntu package, PostreSQL is set for a *peer authentication*, i.e. the database user must match with the Linux user. Other packages may differ. You may quickly check the authentication mode set in the *pg_hba.conf* file. The file is in the same directory of the *postgresql.conf* file you may see as output from the *ps* command shown above, in our case */etc/postgresql/9.5/main*:

.. code-block:: console

  $ sudo grep '^local' /etc/postgresql/9.5/main/pg_hba.conf
  local   all             postgres                                peer
  local   all             all                                     peer
  $

The installation procedure also creates a Linux *postgres* user. In order to check if everything is set correctly, execute the *psql* utility as sudo user:

.. code-block:: console

  $ sudo -u postgres psql -l
                                    List of databases
     Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
  -----------+----------+----------+-------------+-------------+-----------------------
   postgres  | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 |
   template0 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
   template1 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
  (3 rows)
  $

Encoding and collations may differ, depending on the choices made when you installed your operating system. |br| Before you proceed, you must create a PostgreSQL user that matches your Linux user. Supposing that your user is *ubuntu*, type:

.. code-block:: console

  $ sudo -u postgres createuser -d ubuntu
 
The *-d* argument is important because the user will need to create the FogLAMP database.

Finally, you should now be able to see the list of the available databases from your current user:

.. code-block:: console

  $ psql -l
                                    List of databases
     Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
  -----------+----------+----------+-------------+-------------+-----------------------
   postgres  | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 |
   template0 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
   template1 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
  (3 rows)
  $


Obtaining the Source Code
-------------------------

FogLAMP is available on GitHub. The link to the repository is |FogLAMP Repo|. In order to clone the code in the repository, type:

.. code-block:: console

  $ git clone https://github.com/foglamp/FogLAMP.git
  Cloning into 'FogLAMP'...
  remote: Counting objects: 15639, done.
  remote: Compressing objects: 100% (88/88), done.
  remote: Total 15639 (delta 32), reused 58 (delta 14), pack-reused 15531
  Receiving objects: 100% (15639/15639), 9.71 MiB | 2.11 MiB/s, done.
  Resolving deltas: 100% (10486/10486), done.
  Checking connectivity... done.
  $

The code should be now in your home directory. The name of the repository directory is *FogLAMP*:

.. code-block:: console

  $ ls -l FogLAMP/
  total 84
  drwxrwxr-x 5 ubuntu ubuntu  4096 Dec  8 18:00 C
  -rw-rw-r-- 1 ubuntu ubuntu   180 Dec  8 18:00 CMakeLists.txt
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 data
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 docs
  dtrwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 examples
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 extras
  -rw-rw-r-- 1 ubuntu ubuntu  5869 Dec  8 18:00 Jenkinsfile
  -rw-rw-r-- 1 ubuntu ubuntu 11342 Dec  8 18:00 LICENSE
  -rw-rw-r-- 1 ubuntu ubuntu 10654 Dec  8 18:00 Makefile
  -rw-rw-r-- 1 ubuntu ubuntu  5842 Dec  8 18:00 pr_tester.sh
  drwxrwxr-x 4 ubuntu ubuntu  4096 Dec  8 18:00 python
  -rw-rw-r-- 1 ubuntu ubuntu  5916 Dec  8 18:00 README.rst
  drwxrwxr-x 8 ubuntu ubuntu  4096 Dec  8 18:00 scripts
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 tests
  $


Building FogLAMP
----------------

Now that you have downloaded the code, you are ready to build your first FogLAMP project. Move to the *FogLAMP* project directory, type the ``make`` comand and let the magic happen.

.. code-block:: console

  $ cd FogLAMP/<commit_msg>Minor changes<commit_after>.. Getting Started describes how to build and install FogLAMP

.. |br| raw:: html

   <br />

.. Images
.. |foglamp_all_round| image:: images/foglamp_all_round_solution.jpg

.. Links
.. _FogLAMP project on GitHub: https://github.com/foglamp/FogLAMP/issues

.. Links in new tabs
.. |FogLAMP Repo| raw:: html

   <a href="https://github.com/foglamp/FogLAMP" target="_blank">https://github.com/foglamp/FogLAMP</a>

.. |GCC Bug| raw:: html

   <a href="https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66425" target="_blank">here</a>

.. =============================================


***************
Getting Started
***************

Let's get started! In this chapter we will see where to find and how to build, install and run FogLAMP for the first time.


FogLAMP Platforms
=================

Due to the use of standard libraries, FogLAMP can run on a large number of platforms and operating environments, but its primary target is Linux distributions. |br| Our testing environment includes Ubuntu LTS 16.04 and Ubuntu Core 16.04, but we have installed and tested FogLAMP on other Linux distributions such as CentOS and Raspbian. In addition to the native support, FogLAMP can also run on Virtual Machines, Docker and LXC containers. |br| FogLAMP development Ubuntu 16.04 and later additions.


General Requirements
--------------------

This version of FogLAMP requires the following software to be installed in the same environment:
- Python 3.5+
- PostgreSQL 9.5+

The requirements largely depend on the plugins that run in FogLAMP, but Python and PostgreSQL are essential for the Core and Storage microservices and tasks.


Building FogLAMP
================

In this section we will describe how to build FogLAMP on Ubuntu 16.04.3 LTS (Server or Desktop).  Other Linux distributions, Debian or Red-Hat based, or even other versions of Ubuntu may differ. If you are not familiar with Linux and you do not want to build FogLAMP from the source code, you can download a snap package from Snappy.

Build Pre-Requisites
--------------------

FogLAMP is currently based on C/C++ and Python code. The packages needed to build and run FogLAMP are:

- cmake, g++, make
- liboost-dev, liboost-system-dev, liboost-thread-dev, libpq-dev
- python3-pip
- postgresql

.. code-block:: console

  $ sudo apt update
  Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
  ...
  All packages are up-to-date.
  $
  $ sudo apt install cmake g++ make
  Reading package lists... Done
  Building dependency tree
  ...
  $
  $ sudo apt-get install libboost-dev libboost-system-dev libboost-thread-dev libpq-dev
  Reading package lists... Done
  Building dependency tree
  ...
  $
  $ sudo apt-get install python3-pip
  Reading package lists... Done
  Building dependency tree
  ...
  $
  $ sudo apt install postgresql
  Reading package lists... Done
  Building dependency tree
  $


Setting the PostgreSQL Database
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In this version of FogLAMP the PostgreSQL database is the default storage engine, used by the Storage microservice. Make sure that PostgreSQL is installed and running correctly:

.. code-block:: console

  $ sudo systemctl status postgresql
  ● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2017-12-08 15:56:07 GMT; 15min ago
   Main PID: 14572 (code=exited, status=0/SUCCESS)
     CGroup: /system.slice/postgresql.service

  Dec 08 15:56:07 ubuntu systemd[1]: Starting PostgreSQL RDBMS...
  Dec 08 15:56:07 ubuntu systemd[1]: Started PostgreSQL RDBMS.
  Dec 08 15:56:11 ubuntu systemd[1]: Started PostgreSQL RDBMS.
  $
  $ ps -ef | grep postgres
  postgres 14806     1  0 15:56 ?        00:00:00 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
  postgres 14808 14806  0 15:56 ?        00:00:00 postgres: checkpointer process
  postgres 14809 14806  0 15:56 ?        00:00:00 postgres: writer process
  postgres 14810 14806  0 15:56 ?        00:00:00 postgres: wal writer process
  postgres 14811 14806  0 15:56 ?        00:00:00 postgres: autovacuum launcher process
  postgres 14812 14806  0 15:56 ?        00:00:00 postgres: stats collector process
  ubuntu   15198  1225  0 17:22 pts/0    00:00:00 grep --color=auto postgres
  $

When you install the Ubuntu package, PostreSQL is set for a *peer authentication*, i.e. the database user must match with the Linux user. Other packages may differ. You may quickly check the authentication mode set in the *pg_hba.conf* file. The file is in the same directory of the *postgresql.conf* file you may see as output from the *ps* command shown above, in our case */etc/postgresql/9.5/main*:

.. code-block:: console

  $ sudo grep '^local' /etc/postgresql/9.5/main/pg_hba.conf
  local   all             postgres                                peer
  local   all             all                                     peer
  $

The installation procedure also creates a Linux *postgres* user. In order to check if everything is set correctly, execute the *psql* utility as sudo user:

.. code-block:: console

  $ sudo -u postgres psql -l
                                    List of databases
     Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
  -----------+----------+----------+-------------+-------------+-----------------------
   postgres  | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 |
   template0 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
   template1 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
  (3 rows)
  $

Encoding and collations may differ, depending on the choices made when you installed your operating system. |br| Before you proceed, you must create a PostgreSQL user that matches your Linux user. Supposing that your user is *ubuntu*, type:

.. code-block:: console

  $ sudo -u postgres createuser -d ubuntu
 
The *-d* argument is important because the user will need to create the FogLAMP database.

Finally, you should now be able to see the list of the available databases from your current user:

.. code-block:: console

  $ psql -l
                                    List of databases
     Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
  -----------+----------+----------+-------------+-------------+-----------------------
   postgres  | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 |
   template0 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
   template1 | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
  (3 rows)
  $


Obtaining the Source Code
-------------------------

FogLAMP is available on GitHub. The link to the repository is |FogLAMP Repo|. In order to clone the code in the repository, type:

.. code-block:: console

  $ git clone https://github.com/foglamp/FogLAMP.git
  Cloning into 'FogLAMP'...
  remote: Counting objects: 15639, done.
  remote: Compressing objects: 100% (88/88), done.
  remote: Total 15639 (delta 32), reused 58 (delta 14), pack-reused 15531
  Receiving objects: 100% (15639/15639), 9.71 MiB | 2.11 MiB/s, done.
  Resolving deltas: 100% (10486/10486), done.
  Checking connectivity... done.
  $

The code should be now in your home directory. The name of the repository directory is *FogLAMP*:

.. code-block:: console

  $ ls -l FogLAMP
  total 84
  drwxrwxr-x 5 ubuntu ubuntu  4096 Dec  8 18:00 C
  -rw-rw-r-- 1 ubuntu ubuntu   180 Dec  8 18:00 CMakeLists.txt
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 data
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 docs
  dtrwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 examples
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 extras
  -rw-rw-r-- 1 ubuntu ubuntu  5869 Dec  8 18:00 Jenkinsfile
  -rw-rw-r-- 1 ubuntu ubuntu 11342 Dec  8 18:00 LICENSE
  -rw-rw-r-- 1 ubuntu ubuntu 10654 Dec  8 18:00 Makefile
  -rw-rw-r-- 1 ubuntu ubuntu  5842 Dec  8 18:00 pr_tester.sh
  drwxrwxr-x 4 ubuntu ubuntu  4096 Dec  8 18:00 python
  -rw-rw-r-- 1 ubuntu ubuntu  5916 Dec  8 18:00 README.rst
  drwxrwxr-x 8 ubuntu ubuntu  4096 Dec  8 18:00 scripts
  drwxrwxr-x 3 ubuntu ubuntu  4096 Dec  8 18:00 tests
  $


Building FogLAMP
----------------

Now that you have downloaded the code, you are ready to build your first FogLAMP project. Move to the *FogLAMP* project directory, type the ``make`` comand and let the magic happen.

.. code-block:: console

  $ cd FogLAMP
{'fraction_non_alphanumeric': 0.12218613506551686, 'fraction_numerical': 0.05162952178295442, 'mean_word_length': 2.4570267131242742, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 32, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5600665', 'n_tokens_mistral': 6371, 'n_tokens_neox': 5422, 'n_words': 2056}