Policy Compliance Detection via Expression Tree Inference
Policy Compliance Detection (PCD) is a task we encounter when reasoning over
texts, e.g. legal frameworks. Previous work to address PCD relies heavily on
modeling the task as a special case of Recognizing Textual Entailment.
Entailment is applicable to the problem of PCD, however viewing the policy as a
single proposition, as opposed to multiple interlinked propositions, yields
poor performance and lacks explainability. To address this challenge, more
recent proposals for PCD have argued for decomposing policies into expression
trees consisting of questions connected with logic operators. Question
answering is used to obtain answers to these questions with respect to a
scenario. Finally, the expression tree is evaluated in order to arrive at an
overall solution. However, this work assumes expression trees are provided by
experts, thus limiting its applicability to new policies. In this work, we
learn how to infer expression trees automatically from policy texts. We ensure
the validity of the inferred trees by introducing constrained decoding using a
finite state automaton to ensure the generation of valid trees. We determine
through automatic evaluation that 63% of the expression trees generated by our
constrained generation model are logically equivalent to gold trees. Human
evaluation shows that 88% of trees generated by our model are correct.