Efforts have gone into the generation and communication of explanations for complex systems. Some of the approaches proposed are algorithm-based [1]}, that is, they investigate the underlying algorithm in detail in order to support debugging tasks [2]}. These types of explanations are particularly tailored for specific algorithms (algorithm dependent) and the explainer is usually intrinsic. This means that the algorithm to be explained has built-in capabilities to generate explanations. Other explanation approaches are algorithm-agnostic in that they are able to assess the properties of a solution independent of the algorithms used to develop the solution [3]}, [4]}. A solution in this context is the behaviour of the agent. These explainers aim to help the inferential process of the end-user and also enhance the user's knowledge of the agent. Instead of explaining the deep learning algorithm behind the agent, our approach aims to explain the actions of the agent in natural language by observing and learning its behaviour (in a transparent way) over time.
