Course | IDL

Introduction to Interpretability in Deep Learning

The course is available for Master and PhD Students with scholarship (Read here).

Course Links on the University Website:

INF-3605: https://uit.no/utdanning/emner/emne/802776/inf-3605
(master course - 4 ECTS)
INF-8605: https://uit.no/utdanning/emner/emne/802871/inf-8605
(PhD course - 5 ECTS)

Venue : TEKNOBYGGET 1,022AUD, Tromsø, Norway

Registration Tip: The course can be taken as a singular course. The course is organized for participants who are registered at NORA summer school, DLN transdisciplinary course participants, UiT students and during summer/winter school and/or otherwise. The url to the UiT Application Web is: https://fsweb.no/soknadsweb/login.jsf

First digital introductory lecture will be on 8th May 10.15am to 12.00 (noon). This digital lecture will include a course overview and project-related description, as well as what is anticipated in the necessary homework, home exam, and project report. There are only 60 seats available in the course.

Apply interpretability analysis to real application problems.

5 Modules

6 Hours per days

5 ECTS

Hands-On Experience

with deadlines

This course earns you a 5 ECTS credit of completion from UiT The Arctic University of Norway. (Course Link)

Syllabus

hours

3

Concepts Introduction

1. Unveiling the Black-box Problem

A visual history of AI landmarks since 1943
Challenges with black-box model
Trade-offs : completeness vs interpretability; efficacy vs privacy; human explanations vs accuracy

2. Tracing the Evolution of Interpretability and Explainability

Overview of explainable AI techniques
Demand for interpretability over time, relevance & necessity
Taxonomy, type of explanations & flowchart of interpretability
Societal impact and research/industry scope & directions
Applications of interpretability

3. Knowledge versus Performance, Need of Explainability

Challenges and limitations of traditional techniques
Quick introduction to neural network models
Forward propagation, backward propagation & gradient descent
Model validation, learning curve and evaluations

4. Overview of Course & Learning Resources

Quick example: role of interpretabilityy in time series, NLP & CV
References materials: book, online resources, survey papers.

hours

5

Explainable Approaches

1. History of Explaining Data

Fundamentals of machine learning interpretability
Graphical: heatmap, box-plot, radial chart, density plot
Non-Graphical: stat analysis
Traditional: regression, GLM, GAM, Naïve Bayes
Decision trees & k-nearest neighbours

2. Decoding Interpretability in Neural Network

Feature importance and feature interaction
Neural Network perturbation: role of neurons, LIME, occlusion

* Hands-on tutorial: Using image segmentation with LIME in Python

Theory and intuition behind CAM
Neural Network gradient: CAM, saliency, LRP, Grad-CAM

* Hands-on tutorial: Using Grad-CAM (Gradient-weighted Class Activation Mapping) and Saliency in Python

3. Fuzzy Theory, Rules and Fuzzy Learning with Deep Learning

Introduction to rule-based approach: Bayesian rules, rule induction, causal relationship
Quick review of fuzzy Theory, rules, & membership functions
Benefit of fuzzy learning blended with neural network
Reference for further reading

hours

5

Model-agnostic Techniques

1. Knowledge Abstraction & Encoding

Image encoding and example of pixel-wise distance
Perspective of an image
Word encoding and representation
Graph representation and embedding
Tagging mechanism for knowledge embedding

2. Interpretation and Visualization of Abstract Encoding

Concept and techniques, visual explanations
Case study: Image manipulation and semantic interpretations
Case study: Unmasking Clever Hans

3. Example-Based Interpretation

Contrastive explanations & confirmation-bais mind trap
Counterfactual reasoning and causal inference models
Influential instances
Prototypes / case-based reasoning
Introduction to adversarial attacks and defenses
References: adversarial robustness toolbox

hours

5

Neural Networks & Explainability

1. Network Visualization and Neuron Interaction

Introduction to CNNS and RNNs
Role of neurons, layer visualization and feature maps
Semantic understanding
Network understanding in its simplicity
Understanding the influence of different activation functions
Review of CNN visualization techniques

2. Depth of Network, and Abstraction

Rethinking image saliency
Case study: siamese network
Case Study: LightOCT network
Discussion on suitability of shallow learning

3. Unique Way of Explainability Classification

Segregating newer methods into the class and presentation
Harnessing the power of self-interpretable models: towards robust interpretability

4. Generative Adversarial Networks

Generative models
Adversarial training of models
Challenges with generative interpretability

hours

2

The coursework 1 practical lab report and one problem-solving assignment, all of them graded "Approved" / "Not Approved".

Practical lab exercise report and code: Students will undertake practical lab exercises. Students may perform the exercise as groups to support co-learning, but each student will solve the same exercise. One lab report for the mandatory lab exercises with description of the task done and the outcome will be made by each student, which will be collectively submitted as a single report. Codes of the mandatory lab exercises will be collectively submitted as a single code package.

Group problem solving assignment: The students will solve an assigned problem as a group (of 4 to 6 students per group) and submit the declaration about the group work. Each student independently submits the solution. If any student likes to work on this assignment alone, then that will be allowed with prior approval from the course lecturer/TAs.

Individual home assignment: The students will be given a home assignment which each student should solve individually and submit a report about approach and outcome.

The exam consist of two parts:

Home exam counting: 50%
Oral Exam counting: 50%

The home exam period is 8 weeks. Project will be given at the introductory class and student will build their project as the course progress. At the end student will submit a project report upto 10 pages along with the source code they have developed for their project work.

Oral examination includes a 15 min presentation on the self-reading of research article.

Learning Outcome / Objective of the Course:

A general educational aim of the course will be to equip students with knowledge and skills regarding explainable artificial intelligence, for considering explainable approaches for solving a neural learning problem, as well as for developing an explanation for an existing knowledge model developed by a black box approach. This will enable the students to understand, work with, and solve deep learning tasks with a balance of explainability and accuracy, as needed.

Knowledge – The student

understands the need for explainability and the basic concepts underlying the ability or disability of a black-box model being interpretable.
learns about the knowledge versus performance trade-off and examples to build an interpretable system.
learns the various model-agnostic approach for interpretability and the underlying concept of knowledge encoding and abstractions in neural networks, which differs from the traditional knowledge encoding in classical machine learning systems.
knows a variety of explainable approaches of artificial intelligence
knows tools and visualization techniques used for making black-box models explainable
comprehends the neural network and its complexity, learning about tools to explain the abstraction of deep networks.
learns about fuzzy learning in neural network and its basics in making a system more explainable.

Skills – The student can

apply explainable approaches for problem solving
compute feature importance and counterfactual explanation
visualize knowledge encoding in the model
compute class activation and saliency maps and popular visualization techniques.
compare different artificial intelligence approaches for explainability and performance

General competence – The student

has developed confidence of assessing the explainability of an artificial intelligence model
has developed the confidence and skills of making a knowledge model explainable
has developed confidence and skills of making choices and discussing solutions for explainable artificial intelligence
has developed enough background and confidence to discuss, assess, and potentially contribute to advances and newest developments in explainable artificial intelligence
has developed perspectives about the value, ethics, and compromises of explainable artificial intelligence

Learning Modules for Explainable AI with Python

GitHub

Here, we outline a learning structure for you, including some GitHub repositories to follow. This suggested order of curated list of resources, including video lectures, websites, and research papers, shall help students complete the lab exercise. These resources should provide you with a solid foundation for understanding the ethical considerations, explainability, and potential biases in AI systems. Since AI continues to advance, it's essential to address these issues to ensure responsible development and deployment.

Introduction to Explainable AI

Video: Opening the Black Box With Explainable A.I. by UC Berkeley’s Trevor Darrell, Krishna Gade of Fiddler Labs, and Karen Myers from SRI International.
Video: Machine Learning Interpretability by Patrick Hall in H2O.ai
Video: Explainable AI: From Prediction to Understanding by George’s presentation at ODSC 2018.
Video Playlist: Explainable AI Explained by DeepFindr
Website: Explainable AI: A Guide for Making Black Box Models Explainable by George Anadiotis.
Reading: Introduction to Interpretability

Feature Importance and Local Interpretable Model-agnostic Explanations (LIME)

Video Presentation: "Why Should I Trust You?": Explaining the Predictions of Any Classifier. in KDD 2016
Video: Explainable AI explained! | #3 LIME by DeepFindr
GitHub: LIME (Local Interpretable Model-agnostic Explanations)
GitHub: SHAP (SHapley Additive exPlanations)
Video: ML Interpretability: SHAP/LIME by Connor Tann and Dr. Tim Scarfe.

Layer-wise Relevance Propagation (LRP)

Video: Layer-wise Relevance Propagation | Lecture 21 (Part 2)
GitHub: iNNvestigate Neural Networks

Class Activation Mapping (CAM)

Method Paper: Learning Deep Features for Discriminative Localization by Zhou et al. (CVPR 2016)
Video: Deep Learning: Class Activation Maps Theory (Lazy Programmer).
Video: Explaining CNNs: Class Attribution Map Methods (NPTEL-NOC IITM)
GitHub: CAM

Grad-CAM: Visual Explanations from Deep Networks

Paper: Grad-cam: Visual explanations from deep networks via gradient-based localization by Selvaraju et al. (CVPR 2017)
Video: Grad-CAM | Lecture 28 (Part 2) by Maziar Raissi
GitHub: Grad-CAM (Gradient-weighted Class Activation Mapping)

Image Saliency

Paper: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps by Simonyan et al. (arXiv 2013)
Video: Lecture 12 | Visualizing and Understanding by Stanford University School of Engineering.
GitHub: Convolutional Neural Network Visualizations

Integrated Gradients

Video: Feature Attribution | Stanford CS224U Natural Language Understanding | Spring 2021 by Stanford University School of Engineering.
GitHub: Integrated Gradients

Network Dissection

Paper: Network Dissection: Quantifying Interpretability of Deep Visual Representations by Bau et al. (CVPR 2017)
Video: Network Dissection: Visualizing and Understanding Deep Visual Representations by David Bau in CVF Videos.
GitHub: Network Dissection

Counterfactual Explanations

Video: Counterfactual Explanations: The Future of Explainable AI by Aviv Ben Arie
Video: Counterfactual Explanations Can Be Manipulated in UCI NLP (NeurIPS 2021)
GitHub: Alibi: Algorithms for Monitoring and Explaining Machine Learning Models

Ethics in AI

Paper: Moral dilemmas for moral machines by Travis LaCroix (Springer 2022)
Paper: The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation by Brundage et al. (arXiv 2018).
Video: The three big ethical concerns with artificial intelligence by Frank Rudzicz in MaRS Discovery District.
Video: The Mythos of Model Interpretability by Zachary Lipton [The Human Use of Machine Learning [HUML16]]

Data Bias and Model Understanding

Paper: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings by Bolukbasi et al. (NeurIPS 2016)
Paper: Datasheets for Datasets by Gebru et al. (ACM 2021)
Paper: Fairness Definitions Explained by S. Verma and J. Rubin (ACM 2018)
Website: AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias

Additional Resources & Case Studies

Video: Machine Learning Explainability Workshop I Stanford by Professor Hima Lakkaraju
Video: Explainable AI Cheat Sheet - Five Key Categories by Jay Alammar
Website: Explainable AI (XAI) - DARPA by Dr. Matt Turek
Book: Interpretability in Deep Learning by Somani et al. (Springer 2023)
Book: Interpretable Machine Learning by Christoph Molnar

Introduction to Interpretability in Deep Learning

Apply interpretability analysis to real application problems.

5

Modules

6

Hours per days

5

ECTS

Hands-On Experience

with deadlines

Syllabus

hours

3

Concepts Introduction

The exam consist of two parts:

Home exam counting: 50%

Oral Exam counting: 50%

The home exam period is 8 weeks. Project will be given at the introductory class and student will build their project as the course progress. At the end student will submit a project report upto 10 pages along with the source code they have developed for their project work.

Oral examination includes a 15 min presentation on the self-reading of research article.

Learning Modules for Explainable AI with Python

Introduction to Explainable AI

Feature Importance and Local Interpretable Model-agnostic Explanations (LIME)

Layer-wise Relevance Propagation (LRP)

Class Activation Mapping (CAM)

Grad-CAM: Visual Explanations from Deep Networks

Image Saliency

Integrated Gradients

Network Dissection

Counterfactual Explanations

Ethics in AI

Data Bias and Model Understanding

Additional Resources & Case Studies