Citation :

, 2026. "A Sample-Based Study of Influence Functions for Assessing and Improving the Quality of Training in Deep Networks" ESP International Journal of Artificial Intelligence & Data Science [IJAIDS]  Volume 2, Issue 1: 15-29.

Abstract :

Deep neural networks have reached unprecedented performance in many applications, however their training dynamics are complex, and even when controlled well can often be incomprehensible. Although most research efforts have focused on model architectures and optimization algorithms, the influence of individual training samples on learning behaviour has recently received increased attention. Key points Sample influence modelling is a general framework that enables understanding of how individual training examples can affect model predictions and changes to parameters during training. Examining the contributions of individual samples can provide a better understanding of model behavior, difficult parts of data and help develop stronger learning strategies.This paper provides a systematized investigation of sample influence modelling to analysis and control training dynamics in deep neural networks. Authors - Mohammed Y. Saki, Matthew Kwiatkowski and Manias Ramesh This paper starts by providing the theoretical framework of influence functions that estimate the change in a model parameter or prediction as a result of removing or changing an observation in the training sample. The next examines some gradient-based influence estimation methods that enable scalable options for large deep learning systems. Such methods allow you to easily keep track of how the contribution of individual samples influences learning over time and provides real-world tools.It also explores sample reweighting with curriculum learning strategies to improve training efficiency and robustness. Models that pay variable importance to training samples in accordance with their influence, can give priority to informative data and down-weight the noise or harmful effect of samples. It also covers techniques for identifying mislabelled or adversarial points in the dataset, and shows how influence modelling can be adapted to get assistance with debugging and improving the quality of your dataset.Additionally, the paper explores the influence of samples on optimization dynamics — how they affect convergence and loss landscape properties. To achieve this, we introduce influence-based regularization methods (IBRMs), which obtain maximally data-driven constraints that guide the learning problem towards a solution with property. Explores how sample influence modelling can be applied in explainable artificial intelligence to enhance model transparency and trustworthiness by gaining insight into the decision-making process.We provide a set of experimental evaluations and case studies using benchmark datasets and performance metrics to validate the proposed techniques. Experimental results demonstrate that influence-based methods can substantially increase the robustness, training stability and Explainability of models. Nevertheless, one also discusses computational complexity and approximation accuracy as challenges in the context of efficiently or scalable solutions.Conclusions our in-depth investigation of influences on deep learning, reports that training can be improved considering influences at the level of each sample. This work contributes to the advancement of deep networks that are more reliable, interpretable and high-performing by tying theoretical insights with practical techniques.

References :

[1] Pang Wei Kohl and Percy Liang (2017) Understanding Black-box Predictions via Influence Functions. Proceedings of ICML.

[2] Been Kim, Martin Wattenberg and Justin Gilmer (2018) Interpretability beyond Feature Attribution. ICML.

[3] Marco Tulia Ribera, Sameer Singh and Carlos Gastrin (2016) “Why Should I Trust You?” Explaining the Predictions of Any Classifier. KDD.

[4] Scott Lundberg and Su-In Lee (2017) A Unified Approach to Interpreting Model Predictions. NIPS.

[5] Avanti Shrikumar, Peyton Greenside and Anshan Mundane (2017) Learning Important Features Through Propagating Activation Differences. ICML.

[6] Karen Simony an, Andrea Vivaldi and Andrew Fisherman (2014) Deep Inside Convolutional Networks. ICLR Workshop.

[7] Dmitri Ethan et al. (2009) Visualizing Higher-Layer Features of a Deep Network.

[8] Ian Good fellow et al. (2015) Explaining and Harnessing Adversarial Examples. ICLR.

[9] Moritz Hard, Benjamin Resht and Yore Singer (2016) Train Faster, Generalize Better. ICML.

[10] Port Indy et al. (2019) Data Shapley: Equitable Valuation of Data for Machine Learning. ICML.

[11] Amerada Ghorbanifar and James Zoo (2019) Data Shapley: Equitable Valuation of Data for Machine Learning. ICML.

[12] Vitally Feldman and Chan Goo (2021) Does Learning Require Memorization? ICLR.

[13] Chi Yuan Zhang et al. (2017) Understanding Deep Learning Requires Rethinking Generalization. ICLR.

[14] Alexey Dosovitskiy et al. (2020) An Image is Worth 16x16 Words: Transformers for Image Recognition. ICLR.

[15] Hashish Aswan et al. (2017) Attention is All You Need. NIPS.

[16] Sergey Offer and Christian Szeged (2015) Batch Normalization. ICML.

[17] Died Erik Kingman and Jimmy Ba (2015) Adam: A Method for Stochastic Optimization. ICLR.

[18] Geoffrey Hinton et al. (2012) Deep Neural Networks for Acoustic Modelling. IEEE Signal Processing Magazine.

[19] Joshua Bagnio et al. (2007) Greedy Layer-Wise Training of Deep Networks. NIPS.

[20] Corinne Cortes and Vladimir Vapid (1995) Support-Vector Networks. Machine Learning.

[21] Leo Bremen (2001) Random Forests. Machine Learning.

[22] Trevor Hastie, Robert Tibshirani and Jerome Friedman (2009) the Elements of Statistical Learning. Springer.

[23] Christopher Bishop (2006) Pattern Recognition and Machine Learning. Springer.

[24] Kevin Murphy (2012) Machine Learning: A Probabilistic Perspective. MIT Press.

[25] Zorbing Ghahramani (2015) Probabilistic Machine Learning and AI. Nature.

[26] Chelsea Finn et al. (2017) Model-Agnostic Meta-Learning. ICML.

[27] Alex Krizhevsky, Ilia Sutskever and Geoffrey Hinton (2012) Image Net Classification with Deep CNNs. NIPS.

[28] Sergey Zagoruyko and Nikos Komodakis (2016) Wide Residual Networks. BMVC.

[29] Aiming He et al. (2016) Deep Residual Learning for Image Recognition. CVPR.

[30] Tianjin Chen and Carlos Gastrin (2016) Boost: A Scalable Tree Boosting System. KDD.

[31] Rich Carina et al. (2015) Intelligible Models for Healthcare. KDD.

[32] Been Kim et al. (2018) Concept Activation Vectors. ICML.

[33] Raved Shwartz-Ziv and Naphtali Fishy (2017) Opening the Black Box of Deep Neural Networks.

[34] Saied Sharif et al. (2020) On the Stability of Influence Functions.

[35] Henie Siddhi et al. (2019) The Singular Values of Convolutional Layers. ICLR.

Keywords :

Sample Influence Modelling, Deep Neural Networks, Training Dynamics, Influence Functions, Gradient-Based Influence, Importance Of Data By Samples Re-Weighting, Curriculum Learning. Noisy Data Detection, Optimization Stability, Lost Landscape Explainable AI (XAI), Data Debugging, Reliable Model Robust Learning. Model Interpretability.