<commit_before>\subsection{Preprocessing of questions}
Given a question, the system will first of all convert the question
to a format which can be used to perform a query to the indexed
database. This is done by whitelisting all characters which
are not in our supported swedish alphabet. The question is also
processed through a linear classifier to determine the category of our 
answers.

\subsection{Training the Question Categorizer}
A linear classifier is used to classify a given question
such that the "class" of the answer can be determined.

For this system, the training data consists of 2300 questions 
from the swedish board game Kvitt eller Dubbelt transcribed 
by <NAME>, <NAME>, <NAME> \cite{QASYS}.

The training data consists of 2300 rows containing 9 columns each.
We will only be concerned with two columns of each row:
\begin{itemize}
\item Question Formulation - The original question taken from the transcribed
  card.
\item Answer Class - Which was manually determined during transcription. 
  Possible values: action, binary, description, entity, human, location, numeric.
\end{itemize}


\subsection{Performing the query}
\subsection{Post-processing}<commit_msg>remove unneccesary sections<commit_after>\subsection{Preprocessing of questions}
Given a question, the system will first of all convert the question
to a format which can be used to perform a query to the indexed
database. This is done by whitelisting all characters which
are not in our supported swedish alphabet. The question is also
processed through a linear classifier to determine the category of our 
answers.

\subsection{Training the Question Categorizer}
A linear classifier is used to classify a given question
such that the "class" of the answer can be determined.

For this system, the training data consists of 2300 questions 
from the swedish board game Kvitt eller Dubbelt transcribed 
by <NAME>, <NAME>, <NAME> \cite{QASYS}.

The training data consists of 2300 rows containing 9 columns each.
The system is only concerned with two columns of each row:
\begin{itemize}
\item Question Formulation - The original question taken from the transcribed
  card.
\item Answer Class - Which was manually determined during transcription. 
  Possible values: action, binary, description, entity, human, location, numeric.
\end{itemize}
{'fraction_non_alphanumeric': 0.046781115879828326, 'fraction_numerical': 0.007725321888412017, 'mean_word_length': 4.886363636363637, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7473104', 'n_tokens_mistral': 600, 'n_tokens_neox': 567, 'n_words': 329}