Traditionally question answering systems relied heavily on knowledge base access.
This has been expanded by other NLP tasks for which models have been proposed (sec:system-description) that make use of knowledge bases.
Not only do they perform better on e.g. fact-aware language modelling, but they can also provide a degree of explainability (which fact was retrieved) and allow for the model and knowledge base components to be trained separately.
The latter is a strong prerequisite for efficient knowledge base manipulation and control, such as removing misinformation or biases [1]}, [2]} or adding new information.
This is problematic for models like BERT or T5 because their knowledge is tightly coupled with their generative ability, which is stored implicitly in the parameters.
Such models make use of parametric knowledge representations as opposed to non-parametric ones, which are typically used in retrieval-based approaches (separation of the memory and generation components).
Separating memory from the generative component allows for more flexible architectures, though at the cost of increased complexity.
