<commit_before>\chapter{Evaluation\label{chap:evaluation}}
  \section{Evaluation Metrics}
    H1: our tool produces more readable summaries than a simple, statistical, bag-of-words, baseline, extractive summarizer.
    H2: the layout \& explanation of summary content is valuable to the readability of summaries
    H3: for given points in the discussion, our tool selects good extracts to represent them.
    how measured in summary, vs baseline etc.
    ``A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization''
  \section{Evaluation Design}
    set up to test what's being measured, surveys, turk etc.
    space for talk / feedback from that.
  \section{Results}
    results.nsubj be.verb good.dobj
  \section{Discussion}<commit_msg>Complete draft of evaluation design<commit_after>\chapter{Evaluation\label{chap:evaluation}}
  This chapter will detail the evaluation metrics, how they were tested and what the results of the evaluation were. Two (or three) sets of evaluations were carried out making use of six debates from our corpus \cite{walker2012corpus}: abortion, creation, gay rights, the existence of god, gun ownership and healthcare.

  \section{Research Questions}
    We wanted to evaluate our tool as a tool for automated summarization. The result of a final analysis was a summary comprising of point extracts for a given debate, by assessing the quality of this we also evaluate the precursor analysis. The following research questions were set:

    \begin{itemize}
      \item{Are our summaries more readable and informative than those produced by existing tools?}
      \item{Does introducing summary sections with an explanation improve readability?}
      \item{What is the level of agreement between our extract selection module and humans given the same task?}
      \item{TBD: Which sections of generated summaries are most useful to readers?}
    \end{itemize}

    To address these questions we compared different versions of our summaries against equal-length summaries generated by an implementation of \cite{nenkova2006compositional} (`stock summaries'). We also gathered scores for groups of extracts and compared summaries generated with randomly selected extracts against one based on our bigram model for selection.

  \section{Design}
    In response to the our research questions we set the following null hypotheses for the evaluation:

    \begin{itemize}
      \item{The readability and informativeness of our summaries do not improve on those of existing tools. (\textbf{H1})}
      \item{Explanations introducing summary sections does not improve readability. (\textbf{H2})}
      \item{Our means of selecting extracts is not better than random selection. (\textbf{H3})}
      \item{\textcolor{red}{TBD: All summary sections are equally useful to readers. (\textbf{H4})}}
    \end{itemize}

    To test these hypotheses we prepared three comparative studies. These are described in the following sections.

    \subsection{Summary Styles}
      In the upcoming sections, reference is made to a range of summary styles.
      \begin{itemize}
        \item{\textbf{Stock}: A summary generated using an implementation of \cite{nenkova2006compositional}, used as a benchmark for summaries generated by our tool. There is no structure to these summaries.}
        \item{\textbf{Plain}: This is a collection of point extracts in the same styles as the stock summaries. The order of extracts is the same as it would be if the summary had sections annotated. This presentation is designed to be as close as possible to the stock summary presentation.}
        \item{\textbf{Layout}: A summary that has explanatory text that introduces sections. The extracts are the same as those in the plain summary.}
        \item{\textbf{Formatted}: A layout summary with explanation keywords in bold and topic words in green.}
      \end{itemize}

      Participants for Study 1 and 2 were recruited using Amazon Mechanical Turk. \textcolor{red}{In the third study\dots}

    \subsection{Study 1: Summary Comparison}
      In order to address \textbf{H1} and \textbf{H2} we designed a questionnaire that compared various summary styles against equal-length stock summaries. Each questionnaire was made up of three sections.

      The first section compared a plain summary against a stock summary, on the same topic. Users were instructed to read both summaries and rate them relatively on the following criteria: `Content Interest / Informativeness', `Readability', `Punctuation \& Presentation' and `Organization'. Finally they they were asked to give an overall rating and justify their response. The next section asked the same questions but instead compared a layout summary against a stock one. Layout summaries are longer because of the explanatory text, the stock summary was the same length as the layout summary and was longer than the stock summary used in the first section. The final section compared a layout summary with a formatted one. The layout summary is reused from the second section, the formatted summary is also the same content (only different formatting). There was only an overall rating and justification for the comparison in this section.

      There were six versions of the questionnaire setup in a Latin square to cover all six topics. Section 1 had a different topic from sections 2 so as to not repeat summary content. Section 3 addresses the question of formatting only and has the same content as the layout summary in section 2 to make the task faster.

      Responses from the first section that compare our plain summary against a stock one can be used in addressing \textbf{H1}. The difference in the responses gathered in section 2 will allow us to address \textbf{H2}. Section 3 extends on this, testing if the formatting is a useful addition to the layout summary.

    \subsection{Study 2: Extract Comparison}
      To investigate the performance of the tool in greater depth we ran a further study testing the quality of our extract selection mechanism. Responses from this questionnaire addressed \textbf{H3}.

      The task given to participants had two sections. First were a series of extracts for a number of point patterns, participants were asked to rate extracts accounting for their succinctness and the extent to which they made sense. This was the closest we could make the task for participants to that of the tool. The following section, similar to the first study compared two summaries. Both summaries were of the formatted style however their content differed. The extracts in one summary were selected using the bigram model, the extracts in the other were selected at random. Participants were asked to give these a relative rating and justify their response. Responses for these two tasks were both intended to address \textbf{H3}.

    \subsection{\textcolor{red}{TBD: Study 3: Section Comparison}}

  \section{Results}
  \section{Discussion}
{'fraction_non_alphanumeric': 0.045746962115797, 'fraction_numerical': 0.005146533238027162, 'mean_word_length': 4.2208955223880595, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7456050', 'n_tokens_mistral': 1610, 'n_tokens_neox': 1514, 'n_words': 962}