This section describes a range of different experiments to demonstrate the impact of linguistic and structural knowledge on ask/framing detection. It has been argued that the evaluation of dialogue systems is best achieved through comparison to a large set of human-generated responses [1]}. However, with the intermediate step of ask/framing detection—a well-defined sub-problem of the larger dialogue task in a social-engineering setting—it is possible to achieve a very informative evaluation using a single validation set with straight-forward labels.
