Information Theory for Modern Machine Learning: From Theory to Python Practice

Information Theory for Modern Machine Learning: From Theory to Python Practice bridges the classical foundations of information theory with the algorithms that power modern AI. It presents a unified view of machine learning as the transformation, compression, and allocation of information, connecting core quantities such as entropy, mutual information, KL divergence, and Fisher information to practical methods in supervised learning, representation learning, generative modeling, gradient descent optimization, and deep architectures.Unlike texts that treat information theory as a self-contained topic, this book shows how information-theoretic ideas directly explain why familiar objectives and algorithms work and how they generalize to today’s large-scale models.What you’ll learnHow entropy, cross-entropy, KL divergence, and mutual information organize classical ML objectives and generalization behaviorHow optimization dynamics relate to information geometry and Fisher informationHow neural networks reshape uncertainty via nonlinear transformations and JacobiansHow representation learning can be understood through compression–prediction trade-offs (Information Bottleneck)How modern generative modeling methods shape distributions: VAEs, GANs, normalizing flows, and diffusion modelsHow self-supervised and contrastive learning relate to mutual-information objectives and boundsAn information-theoretic perspective on Transformers and attention as adaptive information routingAn integrated framework for reinforcement learning, maximum-entropy control, intrinsic motivation, and RLHFHow information-theoretic principles support lifelong and continual learning, addressing stability–plasticity trade-offs, representational interference, and knowledge accumulation as prerequisites for artificial general intelligence (AGI)Designed for learning and teachingClear conceptual narrative with rigorous mathematical developmentConceptual, mathematical, and proof-based exercises in every chapterComputational exercises and notebooks in Python to help you experiment, visualize, and internalize the ideasWho this book is forAdvanced undergraduates and graduate students in machine learning, statistics, applied math, or engineeringResearchers and practitioners who want a principled framework that connects modern ML to first principlesReaders with basic background in probability and linear algebra (no prior information theory required)Companion codePython implementations of major machine learning algorithm from scratch are available on GitHub: github.com/TangWPI/book_ml_python Read more