top of page
Screenshot 2023-04-29 at 18.03.05.png

Introduction to Interpretability in Deep Learning

The course is available for Master and PhD Students with scholarship (Read here). 

Course Links on the University Website:

Venue :  TEKNOBYGGET 1,022AUD, Tromsø, Norway 

Registration Tip: The course can be taken as a singular course. The course is organized for participants who are registered at NORA summer school, DLN transdisciplinary course participants, UiT students and during summer/winter school and/or otherwise. The url to the UiT Application Web is:  

First digital introductory lecture will be on 8th May 10.15am to 12.00 (noon). This digital lecture will include a course overview and project-related description, as well as what is anticipated in the necessary homework, home exam, and project report. There are only 60 seats available in the course.

Apply interpretability analysis to real application problems.




Hours per days



Hands-On Experience

with  deadlines

This course earns you a 5 ECTS credit of completion from UiT The Arctic University of Norway. (Course Link)



Concepts Introduction

1. Unveiling the Black-box Problem

  • A visual history of AI landmarks since 1943

  • Challenges with black-box model

  • Trade-offs : completeness vs interpretability; efficacy vs privacy; human explanations vs accuracy

2. Tracing the Evolution of Interpretability and Explainability

  • Overview of explainable AI techniques

  • Demand for interpretability over time, relevance & necessity

  • Taxonomy, type of explanations & flowchart of interpretability

  • Societal impact and research/industry scope & directions

  • Applications of interpretability

3. Knowledge versus Performance, Need of Explainability

  • Challenges and limitations of traditional techniques

  • Quick introduction to neural network models

  • Forward propagation, backward propagation & gradient descent

  • Model validation, learning curve and evaluations

4. Overview of Course & Learning Resources

  • Quick example: role of interpretabilityy in time series, NLP & CV

  • References materials: book, online resources, survey papers.

This seminar course will consider different topics of importance regarding explainable artificial intelligence, equipping the students with knowledge of approaches that can be used to explain artificial intelligence, and approaches that are more explainable than others. In addition, the students will receive practical skills of applying selected approaches for interpreting AI, which will equip the students with practical skills of adapting to the rapid pace of technology development in this field.

The coursework 1 practical lab report and one problem-solving assignment, all of them graded "Approved" / "Not Approved".
Practical lab exercise report and code: Students will undertake practical lab exercises. Students may perform the exercise as groups to support co-learning, but each student will solve the same exercise. One lab report for the mandatory lab exercises with description of the task done and the outcome will be made by each student, which will be collectively submitted as a single report. Codes of the mandatory lab exercises will be collectively submitted as a single code package.
Group problem solving assignment: The students will solve an assigned problem as a group (of 4 to 6 students per group) and submit the declaration about the group work. Each student independently submits the solution. If any student likes to work on this assignment alone, then that will be allowed with prior approval from the course lecturer/TAs.
Individual home assignment: The students will be given a home assignment which each student should solve individually and submit a report about approach and outcome. 


The exam consist of two parts:

  • Home exam counting: 50%

  • Oral Exam counting: 50%

The home exam period is 8 weeks. Project will be given at the introductory class and student will build their project as the course progress. At the end student will submit a project report upto 10 pages along with the source code they have developed for their project work.

Oral examination includes a 15 min presentation on the self-reading of research article.

Screenshot 2023-04-29 at 17.59.46.png

Learning Modules for Explainable AI with Python

Here, we outline a learning structure for you, including some GitHub repositories to follow. This suggested order of curated list of resources, including video lectures, websites, and research papers, shall help students complete the lab exercise. These resources should provide you with a solid foundation for understanding the ethical considerations, explainability, and potential biases in AI systems. Since AI continues to advance, it's essential to address these issues to ensure responsible development and deployment.

Introduction to Explainable AI

Feature Importance and Local Interpretable Model-agnostic Explanations (LIME)

  • Video Presentation: "Why Should I Trust You?": Explaining the Predictions of Any Classifier. in KDD 2016

  • Video: Explainable AI explained! | #3 LIME by DeepFindr

  • GitHub: LIME (Local Interpretable Model-agnostic Explanations)

  • GitHub: SHAP (SHapley Additive exPlanations)

  • Video: ML Interpretability: SHAP/LIME by Connor Tann and Dr. Tim Scarfe. 

Layer-wise Relevance Propagation (LRP)

  • Video: Layer-wise Relevance Propagation | Lecture 21 (Part 2)
  • GitHub: iNNvestigate Neural Networks

Class Activation Mapping (CAM)

  • Method Paper: Learning Deep Features for Discriminative Localization by Zhou et al. (CVPR 2016)

  • Video: Deep Learning: Class Activation Maps Theory (Lazy Programmer).

  • Video: Explaining CNNs: Class Attribution Map Methods (NPTEL-NOC IITM)

  • GitHub: CAM

Grad-CAM: Visual Explanations from Deep Networks

  • Paper: Grad-cam: Visual explanations from deep networks via gradient-based localization by Selvaraju et al.  (CVPR 2017)
  • Video: Grad-CAM | Lecture 28 (Part 2) by Maziar Raissi

  • GitHub: Grad-CAM (Gradient-weighted Class Activation Mapping)

Image Saliency

  • Paper: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps by Simonyan et al. (arXiv 2013)

  • Video: Lecture 12 | Visualizing and Understanding by Stanford University School of Engineering.

  • GitHub: Convolutional Neural Network Visualizations

Integrated Gradients

  • Video: Feature Attribution | Stanford CS224U Natural Language Understanding | Spring 2021 by Stanford University School of Engineering. 
  • GitHub: Integrated Gradients

Network Dissection

  • Paper: Network Dissection: Quantifying Interpretability of Deep Visual Representations by Bau et al. (CVPR 2017)

  • Video: Network Dissection: Visualizing and Understanding Deep Visual Representations by David Bau in CVF Videos.

  • GitHub: Network Dissection

Counterfactual Explanations

  • Video: Counterfactual Explanations: The Future of Explainable AI by Aviv Ben Arie
  • Video: Counterfactual Explanations Can Be Manipulated in UCI NLP (NeurIPS 2021)

  • GitHub: Alibi: Algorithms for Monitoring and Explaining Machine Learning Models

Ethics in AI

  • Paper: Moral dilemmas for moral machines by Travis LaCroix (Springer 2022)
  • Paper: The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation by Brundage et al. (arXiv 2018).

  • Video: The three big ethical concerns with artificial intelligence by Frank Rudzicz in MaRS Discovery District​.

  • Video: The Mythos of Model Interpretability by Zachary Lipton [The Human Use of Machine Learning [HUML16]]

Data Bias and Model Understanding

  • Paper: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings by Bolukbasi et al. (NeurIPS 2016)
  • Paper: Datasheets for Datasets by Gebru et al. (ACM 2021)

  • Paper: Fairness Definitions Explained by S. Verma and J. Rubin (ACM 2018)

  • Website: AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias

Additional Resources & Case Studies

  • Video: Machine Learning Explainability Workshop I Stanford by Professor Hima Lakkaraju

  • Video: Explainable AI Cheat Sheet - Five Key Categories by Jay Alammar

  • ​Website: Explainable AI (XAI) - DARPA​ by Dr. Matt Turek

  • Book: Interpretability in Deep Learning by Somani et al. (Springer 2023)

  • Book: Interpretable Machine Learning by Christoph Molnar

bottom of page