<commit_before>
\chapter{LeapGesture library dedicated for Leap Motion controller}\label{libraryChapter}

\lstset{ %
language=C++,                % choose the language of the code
basicstyle=\footnotesize,       % the size of the fonts that are used for the code
numbers=left,                   % where to put the line-numbers
numberstyle=\footnotesize,      % the size of the fonts that are used for the line-numbers
stepnumber=1,                   % the step between two line-numbers. If it is 1 each line will be numbered
numbersep=5pt,                  % how far the line-numbers are from the code
backgroundcolor=\color{white},  % choose the background color. You must add \usepackage{color}
showspaces=false,               % show spaces adding particular underscores
showstringspaces=false,         % underline spaces within strings
showtabs=false,                 % show tabs within strings adding particular underscores
frame=single,           % adds a frame around the code
tabsize=2,          % sets default tabsize to 2 spaces
captionpos=b,           % sets the caption-position to bottom
breaklines=true,        % sets automatic line breaking
breakatwhitespace=false,    % sets if automatic breaks should only happen at whitespace
escapeinside={\%*}{*)}          % if you want to add a comment within your code
}

A library containing previously described gesture recognition methods has been created. 
It is designed as an aid for developers who would like to use gesture input in their applications, but do not want to implement recognition methods manually.
Using it does not require deep knowledge about gesture recognition methods -- a library is like a black box: after initial learning process, it is feed with frames from Leap Motion, and responds if there is a match to previously learnt gesture. 
Code was written in C++, so it can be used on many platforms.
When there is a need to use the library in an application written in different programming language, the library can be compiled into dynamic-link library (DLL) or dynamic shared object (DSO) and linked in a project.
Thus, the library can be easily used in many projects.


\section{Architecture} \label{architectureSection}

The library is divided into three main modules:
\begin{itemize}
\item a static gesture recognition module,
\item a dynamic gesture recognition module,
\item a fingers recognition module.
\end{itemize}

Each of those modules has its own separate methods -- for learning new gestures, and for further operational work (recognition). They are implemented using the observer pattern. 
The main processing of the library is performed within a separate thread, that creates certain list of events.
The set of observers is defined to be called on the library's events.
The most important events that are fired in the key moments of recognition include: an event when particular gesture began to be recognized, an event when the particular gesture has being recognized, an event when a frame processing is finished.

The library can be divided into learning (offline) and recognition (online) parts.
	
\subsection{Learning}

Learning process is intended to be run only once and it has to be done before user decides to recognize gestures. Its goal is to create a model, which will be can be later used in recognition task. As an input, the learning needs a set of recordings. The recordings are files with gestures recorded using the Leap Motion and the Recorder application. Additionally, learning process requires a configuration file containing learning parameters. The output of learning process is a model file used in recognition process. 
The learning process informs about the what is the expected recognition measured on the training set.

Sample code illustrating the reading recorded data, including the configuration file and performing a process of learning two classes of static gestures:
\begin{lstlisting}
	vector<pair<const char*, const char*> > input;
	input.push_back(make_pair("gesture1.lmr", "gesture1"));
	input.push_back(make_pair("gesture2.lmr", "gesture2"));

	TrainingStaticRecConf conf;
	conf.configurationPath = "/home/user";
	conf.configurationName = "test";

	Learning::learnStatic(input, conf);
\end{lstlisting}


\subsection{Testing}

Testing (recognition) process is a process performed to compute if and what gesture is recognized. 
The testing process gathers data from the Leap Motion controller and processes them in a separate threads.

The general workflow can be defined in a following way:
\begin{enumerate}
  \item developer creates object of his own class, implementing RecognizedGestureListener,
  \item developer creates LeapProcess object, specifies which modules should be run and supplies RecognizedGestureListener object,
  \item developer creates configuration object and sets it as appropriate field in LeapProcess:
  \begin{enumerate}
  	\item class TestingStaticRecConf, field staticRecConfiguration -- for static gesture recognition,
  	\item class TestingDynamicRecConf, field dynamicRecConfiguration -- for dynamic gesture recognition,
  	\item class TestingFingerRecConf, field fingerRecConfiguration -- for fingers recognition,
  \end{enumerate}
  \item developer attaches LeapListener object to the LeapProcess object,
  \item developer starts the process,
  \item if a gesture recognition event had happened, an user supplied method is executed.
\end{enumerate}

\begin{figure}[htb]
\centering
 \includegraphics[width=0.8\columnwidth]{figures/timeline.eps}
 \caption[]{Timing diagram of processing static and dynamic gestures}
 \label{processingtimeline}
\end{figure}

Sample code showing testing (recognition) for model prepared in previous step:
\begin{lstlisting}
// specifies method which will be executed after successful recognition
class MyGestures: public RecognizedGestureListener {

	void onStaticRecognized(TestingResult *tr) {
		if(tr->testClassResults[0].classTrainRate > 0.5) {
			cout << "CLASS 0!" << endl;
		}
		else cout << "CLASS 1!" << endl;
	}

	void onDynamicRecognized() {}

};

int main(int argc, char **argv) {

	// create instance of recognized gesture actions
	MyGestures *gst = new MyGestures();

	// create new data processor (for static gestures only)
	LeapProcess *process = new LeapProcess(gst, true, false);

	// specify settings: path where model with given name is located
	TestingStaticRecConf settings("/home/user", "test");
	process->staticConf = &settings;
	
	// create a listener and attach it to a process
	LeapListener listener(3);	// 3 = radius of preprocessing windows
	listener.attachToProcess(process);
	
	// create a Leap Motion controller object and direct its data output to a listener
	Controller c(listener);

	// start processing
	process->start();
	
	// as processing works in a separate thread, application can now perform other tasks
	cin.get();
	
	// stop processing
	c.removeListener(listener);

}
\end{lstlisting}

\subsection{Elements of library}
\begin{itemize}
\item LeapProcess -- main class, acting as an interface between Leap Motion device, application and processing threads.
\item LeapListener -- class responsible for gathering data from Leap Motion and feeding them to processing threads (via LeapProcess).
\item FileListener -- as above, but data comes from a file, not controller. Used mainly for testing.
\item TestingResult -- class containing results of testing process:
\begin{itemize}
	\item bool recognized -- notifies if a gesture has been recognized
	\item string className -- name of class with highest recognition rate
	\item string genericClassName -- generic name of class with highest recognition rate
	\item vector\textless TestingClassResult \textgreater testClassResults -- vector of TestingClassResult
\end{itemize}
\item TestingClassResult -- contains testing results for a single class
\begin{itemize}
	\item double classTrainRate -- recognition rate for given class
	\item string className -- name of a classs
	\item string genericClassName -- generic name of class
\end{itemize}

\item RecognizedGesture -- abstract class containing methods which will be executed after successful recognition
\item GestureFinger, GestureHand, GestureFrame, Vertex - classes representing respectively: Frame, Hand and Finger, with additional Vertex class, which simpifies operations on 3D points
\item StorageDriver -- class for I/O operations on LMR files<commit_msg>Jezykowe<commit_after>
\chapter{LeapGesture library dedicated for Leap Motion controller}\label{libraryChapter}

\lstset{ %
language=C++,                % choose the language of the code
basicstyle=\footnotesize,       % the size of the fonts that are used for the code
numbers=left,                   % where to put the line-numbers
numberstyle=\footnotesize,      % the size of the fonts that are used for the line-numbers
stepnumber=1,                   % the step between two line-numbers. If it is 1 each line will be numbered
numbersep=5pt,                  % how far the line-numbers are from the code
backgroundcolor=\color{white},  % choose the background color. You must add \usepackage{color}
showspaces=false,               % show spaces adding particular underscores
showstringspaces=false,         % underline spaces within strings
showtabs=false,                 % show tabs within strings adding particular underscores
frame=single,           % adds a frame around the code
tabsize=2,          % sets default tabsize to 2 spaces
captionpos=b,           % sets the caption-position to bottom
breaklines=true,        % sets automatic line breaking
breakatwhitespace=false,    % sets if automatic breaks should only happen at whitespace
escapeinside={\%*}{*)}          % if you want to add a comment within your code
}

A library containing previously described gesture recognition methods has been created. 
It is designed as an aid for developers who would like to use gesture input in their applications, but do not want to implement recognition methods manually.
Using it does not require deep knowledge about gesture recognition methods -- a library is like a black box: after initial learning process, it is feed with frames from Leap Motion, and responds if there is a match to previously learnt gesture. 
Code was written in C++, so it can be used on many platforms.
When there is a need to use the library in an application written in different programming language, the library can be compiled into dynamic-link library (DLL) or dynamic shared object (DSO) and linked in a project.
Thus, the library can be easily used in many projects.


\section{Architecture} \label{architectureSection}

The library is divided into three main modules:
\begin{itemize}
\item a static gesture recognition module,
\item a dynamic gesture recognition module,
\item a fingers recognition module.
\end{itemize}

Each of those modules has its own separate methods -- for learning new gestures, and for further operational work (recognition). They are implemented using the observer pattern. 
The main processing of the library is performed within a separate thread, that creates certain list of events.
The set of observers is defined to be called on the library's events.
The most important events that are fired in the key moments of recognition include: an event when particular gesture began to be recognized, an event when the particular gesture has being recognized, an event when a frame processing is finished.

The library can be divided into learning (offline) and recognition (online) parts.
	
\subsection{Learning}

Learning process is intended to be run only once and it has to be done before user decides to recognize gestures. Its goal is to create a model, which will be later used in recognition task. As an input, the learning needs a set of recordings. The recordings are files with gestures recorded using the Leap Motion and the Recorder application. Additionally, learning process requires a configuration file containing learning parameters. The output of learning process is a model file used in recognition process. 
The learning process informs about the what is the expected recognition rate measured on the training set.

Sample code illustrating the reading recorded data, including the configuration file and performing a process of learning two classes of static gestures:
\begin{lstlisting}
	vector<pair<const char*, const char*> > input;
	input.push_back(make_pair("gesture1.lmr", "gesture1"));
	input.push_back(make_pair("gesture2.lmr", "gesture2"));

	TrainingStaticRecConf conf;
	conf.configurationPath = "/home/user";
	conf.configurationName = "test";

	Learning::learnStatic(input, conf);
\end{lstlisting}


\subsection{Testing}

Testing (recognition) process is a process performed to compute if and what gesture is recognized. 
The testing process gathers data from the Leap Motion controller and processes them in a separate threads.

The general workflow can be defined in a following way:
\begin{enumerate}
  \item developer creates object of his own class, implementing RecognizedGestureListener,
  \item developer creates LeapProcess object, specifies which modules should be run and supplies RecognizedGestureListener object,
  \item developer creates configuration object and sets it as appropriate field in LeapProcess:
  \begin{enumerate}
  	\item class TestingStaticRecConf, field staticRecConfiguration -- for static gesture recognition,
  	\item class TestingDynamicRecConf, field dynamicRecConfiguration -- for dynamic gesture recognition,
  	\item class TestingFingerRecConf, field fingerRecConfiguration -- for fingers recognition,
  \end{enumerate}
  \item developer attaches LeapListener object to the LeapProcess object,
  \item developer starts the process,
  \item if a gesture recognition event had happened, an user supplied method is executed.
\end{enumerate}

\begin{figure}[htb]
\centering
 \includegraphics[width=0.8\columnwidth]{figures/timeline.eps}
 \caption[]{Timing diagram of processing static and dynamic gestures}
 \label{processingtimeline}
\end{figure}

Sample code showing testing (recognition) for model prepared in previous step:
\begin{lstlisting}
// specifies method which will be executed after successful recognition
class MyGestures: public RecognizedGestureListener {

	void onStaticRecognized(TestingResult *tr) {
		if(tr->testClassResults[0].classTrainRate > 0.5) {
			cout << "CLASS 0!" << endl;
		}
		else cout << "CLASS 1!" << endl;
	}

	void onDynamicRecognized() {}

};

int main(int argc, char **argv) {

	// create instance of recognized gesture actions
	MyGestures *gst = new MyGestures();

	// create new data processor (for static gestures only)
	LeapProcess *process = new LeapProcess(gst, true, false);

	// specify settings: path where model with given name is located
	TestingStaticRecConf settings("/home/user", "test");
	process->staticConf = &settings;
	
	// create a listener and attach it to a process
	LeapListener listener(3);	// 3 = radius of preprocessing windows
	listener.attachToProcess(process);
	
	// create a Leap Motion controller object and direct its data output to a listener
	Controller c(listener);

	// start processing
	process->start();
	
	// as processing works in a separate thread, application can now perform other tasks
	cin.get();
	
	// stop processing
	c.removeListener(listener);

}
\end{lstlisting}

\subsection{Elements of library}



\begin{itemize}
\item LeapProcess -- main class, acting as an interface between Leap Motion device, application and processing threads.
\item LeapListener -- class responsible for gathering data from Leap Motion and feeding them to processing threads (via LeapProcess).
\item FileListener -- as above, but data comes from a file, not controller. Used mainly for testing.
\item TestingResult -- class containing results of testing process:
\begin{itemize}
	\item bool recognized -- notifies if a gesture has been recognized
	\item string className -- name of class with highest recognition rate
	\item string genericClassName -- generic name of class with highest recognition rate
	\item vector\textless TestingClassResult \textgreater testClassResults -- vector of TestingClassResult
\end{itemize}
\item TestingClassResult -- contains testing results for a single class
\begin{itemize}
	\item double classTrainRate -- recognition rate for given class
	\item string className -- name of a classs
	\item string genericClassName -- generic name of class
\end{itemize}

\item RecognizedGesture -- abstract class containing methods which will be executed after successful recognition
\item GestureFinger, GestureHand, GestureFrame, Vertex - classes representing respectively: Frame, Hand and Finger, with additional Vertex class, which simpifies operations on 3D points
\item StorageDriver -- class for I/O operations on LMR files
{'fraction_non_alphanumeric': 0.060714285714285714, 'fraction_numerical': 0.002261904761904762, 'mean_word_length': 4.366017246885979, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '913998', 'n_tokens_mistral': 4174, 'n_tokens_neox': 3974, 'n_words': 2068}