With growth of highly specialized architectures for a variety of use-cases and their superior performance, Deep Neural Networks (DNNs) are increasingly being used in sensitive and critical applications such as medical diagnosis, employment/recruiting, financial credit analysis, etc. However, widespread adoption of such models faces several challenges - primarily the black-box nature of their predictions. Many recent research works have proposed explanation methods which provide deeper understanding of model predictions by providing “explanations”. Explanations range from being local in nature where they assign importance scores to the features present in an input sample but can also be global in nature where the model identifies certain “concepts” present in the input sample. A concept can be thought of as an abstraction of features which are usually shared across multiple similar sample points. For example, in Figure REF , a concept can be entirely clinical “osteophyptes- femur”, “sclerosis-tibia”, etc. Usually DNNs are trained end-to-end, which makes it difficult to isolate concepts and even harder to make them human understandable. To alleviate this, concept-based approaches have been proposed [1]}, [2]} which map a sample from input space to a concept space and subsequently map the concept space to the prediction space. The concept space usually consists of high-level human understandable concepts. A model trained incorporating either manually curated or automatically learned concepts increases both interpretability and reliability of its predictions. One such example, as proposed in [1]}, Concept Bottleneck Models (CBMs), can help domain experts quickly identify any discrepancy and intervene when and where needed. CBMs also offer generalizability in the sense that any DNN can be easily converted into a CBM by resizing an intermediate layer to correspond to the size of any closed concept set pre-selected by domain experts. The training of such models uses standard training procedure with a loss function augmented with an extra term from the bottleneck layer.
<FIGURE>