<commit_before>% Instructions to modify this document:
% * Remember to ALWAYS execute "git pull" BEFORE any commit you make!
% * Use the \ToDo{...} command to remark tasks which still need to be done. Add your name in the comment.
% * Use the \input{file.tex} command to split the document into several parts
% * Do not change the current LaTeX coding style to yours. The style and format should be homogeneous along sections.

% To convert .dia diagrams into PDF:
% 1) Create the diagram with dia
% 2) Export it as .eps
% 3) use epstopdf to convert to PDF
%
% Editing SVG files and exporting them to PDF with Inkscape is admitted too.
% Remember to keep a copy of the editable file (.dia or .svg files).

\documentclass[a4paper,12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{amsmath,graphicx}
\usepackage{bm}
\usepackage{amssymb}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{subfigure}
\usepackage{ifpdf}
\usepackage{url}
\usepackage{color}
\usepackage[hidelinks]{hyperref}
\usepackage{multirow}
\usepackage{datetime}
\usepackage{comment}
\usepackage{float} % To put figures in their exact place with \begin{figure}[H]
\usepackage{longtable}
\usepackage{tabularx}
\usepackage{listings}
\usepackage{xcolor}


\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}}
\newcolumntype{C}[1]{>{\centering\arraybackslash}p{#1}}
\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}}


% Definitions and commands
\def \np{\vskip 0.25 cm}
\def \ap{\vskip 0.15 cm}

% JSON listing (see 
%  http://tex.stackexchange.com/questions/83085/how-to-improve-listings-display 
% -of- json-files)
\colorlet{punct}{red!60!black}
\definecolor{background}{HTML}{EEEEEE}
\definecolor{delim}{RGB}{20,105,176}
\colorlet{numb}{magenta!60!black}

\lstdefinelanguage{json}{
    basicstyle=\footnotesize\ttfamily,
    numbers=left,
    numberstyle=\scriptsize,
    stepnumber=1,
    numbersep=8pt,
    showstringspaces=false,
    breaklines=true,
    frame=lines,
    backgroundcolor=\color{background},
    literate=
     *{0}{{{\color{numb}0}}}{1}
      {1}{{{\color{numb}1}}}{1}
      {2}{{{\color{numb}2}}}{1}
      {3}{{{\color{numb}3}}}{1}
      {4}{{{\color{numb}4}}}{1}
      {5}{{{\color{numb}5}}}{1}
      {6}{{{\color{numb}6}}}{1}
      {7}{{{\color{numb}7}}}{1}
      {8}{{{\color{numb}8}}}{1}
      {9}{{{\color{numb}9}}}{1}
      {:}{{{\color{punct}{:}}}}{1}
      {,}{{{\color{punct}{,}}}}{1}
      {\{}{{{\color{delim}{\{}}}}{1}
      {\}}{{{\color{delim}{\}}}}}{1}
      {[}{{{\color{delim}{[}}}}{1}
      {]}{{{\color{delim}{]}}}}{1},
}

\lstset{language=Bash, basicstyle=\color{gray}}

\newcommand{\ToDo}[1]{\textcolor{magenta}{\textbf{[ToDo]} \textbf{#1}}}
\newcommand{\miguel}[1]{\textcolor{magenta}{\textbf{[Miguel]} \textbf{#1}}}


\begin{document}


\begin{titlepage}

\begin{center}
\vspace*{-1in}

\vspace*{0.6in}
\begin{Large}
\textbf{The IPOL Demo System 2.0 \\Technical documentation} \\
\end{Large}

\vspace*{0.6in}

\small{Compiled on \today\ at \currenttime}

\vspace*{0.6in}
\rule{80mm}{0.1mm}\\
\vspace*{0.1in}
\end{center}

\end{titlepage}

This document contains technical documentation for the IPOL Demo System 2.0. Specifically, the architecture of the service-oriented platform, its modules, and the real-time template generation of demos from their textual description.
\vspace*{0.6in}

\textbf{Software engineers and external consultants, in alphabetical order:}

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

\vspace*{0.2in}

\textbf{Project direction and team coordination}

<NAME> - \url{http://mcolom.info}



%\maketitle
\newpage

\tableofcontents
\newpage
\listoffigures
\newpage

% Introduction
\input{introduction/introduction.tex}

% Project management and development methodology
\input{methodology/methodology.tex}

% The Core module
\input{core/core.tex}

% The Blobs module
\input{blobs/blobs.tex}

% The Archive module
\input{archive/archive.tex}

% The Demo Info module
\input{demo_info/demo_info.tex}

% The Demo Dispatcher module
\input{dispatcher/dispatcher.tex}

% The Demo Runner module
\input{demo_runner/demo_runner.tex}

% The Control Terminal 
\input{terminal/terminal.tex}

% The Access Control 
\input{access_control/access_control.tex}

% Tools
\input{tools/tools.tex}

% Installation
\input{installation/installation.tex}

% General notes
\section{General notes}
In this section we add general comments on the projects, parts which still neeed to be written or coded. In general, information which needs to be documented but has not still found its place in the document. It works as a reminder not to forget.

\begin{itemize}
  \item The magic module. We use PIP, not the python-magic package found in most distributions. \ToDo{Explain why}.
  \item List of packages needed and how to install them.
  \item Description of the general requirements needed for a module to be part of the system (start, ping, and shutdown services ; structure of the general launching script...
\end{itemize}

- List of packages used for installing the system. Packages to be installed with sudo apt-get install:

\begin{itemize}
\item python2.7
\item wget: to download files
\item cmake: to create a multiplatform Makefile easily
\item libtiff5-dev libtiff-tools libjpeg-dev libpng-dev: open and write TIFF, JPEG, PNG files
\item libfftw3-dev: fast DFT operations
\item libgsl0-dev libeigen3-dev liblapack-dev libblas-dev: linear algebra
\item libopenblas-base libopenblas-dev: allow numpy, scipy, and others to run some operations in parallel
\item python python3 python-dev python3-dev python-cherrypy3 python-mako python-jsonschema: Python language, Python headers, webserver, web templating, JSON
\item gnuplot gnuplot-nox: to draw figures
\item python-pip: to install python packages
\item python-yaml: for parsing
\item libpq-dev: PostgreSQL headers
\item texlive-full: compilation of the LaTeX documentation
\item libmagic-dev: needed to install python-magic with pip
\item libav-tools: audio/video tools for demos with these data types
\item libgsl2 libgsl-dev libgsl-dbg  libblas-dev libblas3 liblapack-dev liblapack liblapacke-dev liblapacke: numerical computations with GSL, BLAS, LAPACK
\item octave octave-general octave-dataframe octave-image octave-linear-algebra octave-splines octave-statistics: Octave and libraries
\item sshfs: to mount filesystem across different machines using SSH.
\item libgdal-dev
\item gdal-bin
\item python-gdal
\end{itemize}

Packages to be installed with pip install: \miguel{This should be put in a requirements.txt file and here document only how to use that file.}
\begin{itemize}
\item python-magic
\item pypng
\item numpy
\item libtiff
\item pillow
\item simplejson
\item matplotlib
\item validoot (\url{https://github.com/AstromechZA/validoot/tarball/1.3})
\item future 
\item pyfftw
\item sklearn
\item cython
\item sortedcontainers
\item tifffile
\item weightedstats
\item bs4
\item utm
\item mgrs
\item requests
\end{itemize}

- Example of another demo system: \url{http://places.csail.mit.edu/demo.html}

- Integrate Openseedragon, \url{https://openseadragon.github.io/}

- Document the format of the demo package, with desc.jon, images/, extra/, etc

- Most of our modules name the files according to the hash sum of their contents. This file names are OK for internal use, but the users should obtain the files with the proper names. This can be achieved with the ``download" attribute in HTML5. All website interfaces should provide the corresponding human-readable name. For example:

\begin{verbatim}
<a href="9021384984901238490128490123841222223312.png" download="denoised.png">
  Download the denoised image
</a>
\end{verbatim}<commit_msg>[doc/system] Removed packages list that are installed with pip<commit_after>% Instructions to modify this document:
% * Remember to ALWAYS execute "git pull" BEFORE any commit you make!
% * Use the \ToDo{...} command to remark tasks which still need to be done. Add your name in the comment.
% * Use the \input{file.tex} command to split the document into several parts
% * Do not change the current LaTeX coding style to yours. The style and format should be homogeneous along sections.

% To convert .dia diagrams into PDF:
% 1) Create the diagram with dia
% 2) Export it as .eps
% 3) use epstopdf to convert to PDF
%
% Editing SVG files and exporting them to PDF with Inkscape is admitted too.
% Remember to keep a copy of the editable file (.dia or .svg files).

\documentclass[a4paper,12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{amsmath,graphicx}
\usepackage{bm}
\usepackage{amssymb}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{subfigure}
\usepackage{ifpdf}
\usepackage{url}
\usepackage{color}
\usepackage[hidelinks]{hyperref}
\usepackage{multirow}
\usepackage{datetime}
\usepackage{comment}
\usepackage{float} % To put figures in their exact place with \begin{figure}[H]
\usepackage{longtable}
\usepackage{tabularx}
\usepackage{listings}
\usepackage{xcolor}


\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}}
\newcolumntype{C}[1]{>{\centering\arraybackslash}p{#1}}
\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}}


% Definitions and commands
\def \np{\vskip 0.25 cm}
\def \ap{\vskip 0.15 cm}

% JSON listing (see 
%  http://tex.stackexchange.com/questions/83085/how-to-improve-listings-display 
% -of- json-files)
\colorlet{punct}{red!60!black}
\definecolor{background}{HTML}{EEEEEE}
\definecolor{delim}{RGB}{20,105,176}
\colorlet{numb}{magenta!60!black}

\lstdefinelanguage{json}{
    basicstyle=\footnotesize\ttfamily,
    numbers=left,
    numberstyle=\scriptsize,
    stepnumber=1,
    numbersep=8pt,
    showstringspaces=false,
    breaklines=true,
    frame=lines,
    backgroundcolor=\color{background},
    literate=
     *{0}{{{\color{numb}0}}}{1}
      {1}{{{\color{numb}1}}}{1}
      {2}{{{\color{numb}2}}}{1}
      {3}{{{\color{numb}3}}}{1}
      {4}{{{\color{numb}4}}}{1}
      {5}{{{\color{numb}5}}}{1}
      {6}{{{\color{numb}6}}}{1}
      {7}{{{\color{numb}7}}}{1}
      {8}{{{\color{numb}8}}}{1}
      {9}{{{\color{numb}9}}}{1}
      {:}{{{\color{punct}{:}}}}{1}
      {,}{{{\color{punct}{,}}}}{1}
      {\{}{{{\color{delim}{\{}}}}{1}
      {\}}{{{\color{delim}{\}}}}}{1}
      {[}{{{\color{delim}{[}}}}{1}
      {]}{{{\color{delim}{]}}}}{1},
}

\lstset{language=Bash, basicstyle=\color{gray}}

\newcommand{\ToDo}[1]{\textcolor{magenta}{\textbf{[ToDo]} \textbf{#1}}}
\newcommand{\miguel}[1]{\textcolor{magenta}{\textbf{[Miguel]} \textbf{#1}}}


\begin{document}


\begin{titlepage}

\begin{center}
\vspace*{-1in}

\vspace*{0.6in}
\begin{Large}
\textbf{The IPOL Demo System 2.0 \\Technical documentation} \\
\end{Large}

\vspace*{0.6in}

\small{Compiled on \today\ at \currenttime}

\vspace*{0.6in}
\rule{80mm}{0.1mm}\\
\vspace*{0.1in}
\end{center}

\end{titlepage}

This document contains technical documentation for the IPOL Demo System 2.0. Specifically, the architecture of the service-oriented platform, its modules, and the real-time template generation of demos from their textual description.
\vspace*{0.6in}

\textbf{Software engineers and external consultants, in alphabetical order:}

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

<NAME>

\vspace*{0.2in}

\textbf{Project direction and team coordination}

<NAME> - \url{http://mcolom.info}



%\maketitle
\newpage

\tableofcontents
\newpage
\listoffigures
\newpage

% Introduction
\input{introduction/introduction.tex}

% Project management and development methodology
\input{methodology/methodology.tex}

% The Core module
\input{core/core.tex}

% The Blobs module
\input{blobs/blobs.tex}

% The Archive module
\input{archive/archive.tex}

% The Demo Info module
\input{demo_info/demo_info.tex}

% The Demo Dispatcher module
\input{dispatcher/dispatcher.tex}

% The Demo Runner module
\input{demo_runner/demo_runner.tex}

% The Control Terminal 
\input{terminal/terminal.tex}

% The Access Control 
\input{access_control/access_control.tex}

% Tools
\input{tools/tools.tex}

% Installation
\input{installation/installation.tex}

% General notes
\section{General notes}
In this section we add general comments on the projects, parts which still neeed to be written or coded. In general, information which needs to be documented but has not still found its place in the document. It works as a reminder not to forget.

\begin{itemize}
  \item The magic module. We use PIP, not the python-magic package found in most distributions. \ToDo{Explain why}.
  \item List of packages needed and how to install them.
  \item Description of the general requirements needed for a module to be part of the system (start, ping, and shutdown services ; structure of the general launching script...
\end{itemize}

- List of packages used for installing the system. Packages to be installed with sudo apt-get install:

\begin{itemize}
\item python2.7
\item wget: to download files
\item cmake: to create a multiplatform Makefile easily
\item libtiff5-dev libtiff-tools libjpeg-dev libpng-dev: open and write TIFF, JPEG, PNG files
\item libfftw3-dev: fast DFT operations
\item libgsl0-dev libeigen3-dev liblapack-dev libblas-dev: linear algebra
\item libopenblas-base libopenblas-dev: allow numpy, scipy, and others to run some operations in parallel
\item python python3 python-dev python3-dev python-cherrypy3 python-mako python-jsonschema: Python language, Python headers, webserver, web templating, JSON
\item gnuplot gnuplot-nox: to draw figures
\item python-pip: to install python packages
\item python-yaml: for parsing
\item libpq-dev: PostgreSQL headers
\item texlive-full: compilation of the LaTeX documentation
\item libmagic-dev: needed to install python-magic with pip
\item libav-tools: audio/video tools for demos with these data types
\item libgsl2 libgsl-dev libgsl-dbg  libblas-dev libblas3 liblapack-dev liblapack liblapacke-dev liblapacke: numerical computations with GSL, BLAS, LAPACK
\item octave octave-general octave-dataframe octave-image octave-linear-algebra octave-splines octave-statistics: Octave and libraries
\item sshfs: to mount filesystem across different machines using SSH.
\item libgdal-dev
\item gdal-bin
\item python-gdal
\end{itemize}

- Example of another demo system: \url{http://places.csail.mit.edu/demo.html}

- Integrate Openseedragon, \url{https://openseadragon.github.io/}

- Document the format of the demo package, with desc.jon, images/, extra/, etc

- Most of our modules name the files according to the hash sum of their contents. This file names are OK for internal use, but the users should obtain the files with the proper names. This can be achieved with the ``download" attribute in HTML5. All website interfaces should provide the corresponding human-readable name. For example:

\begin{verbatim}
<a href="9021384984901238490128490123841222223312.png" download="denoised.png">
  Download the denoised image
</a>
\end{verbatim}
{'fraction_non_alphanumeric': 0.13096195069667738, 'fraction_numerical': 0.020431404072883172, 'mean_word_length': 4.622975517890772, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1504913', 'n_tokens_mistral': 5178, 'n_tokens_neox': 4880, 'n_words': 1482}