Information and Decision-Making Theories

Statistical modeling is based on two fundamental concepts: information (which is to be extracted from data) and decision-making (which is to be carried out based on that data). This course provides an introduction to the theoretical formalization of these two concepts. It is therefore logically placed at the beginning of the program, as many other courses subsequently draw on these concepts and results.

Introduce the two concepts that form the foundation of mathematical statistics: information (quantification of information and coding) and decision-making (quantification and management of risk).

Probability Theory Course.

Recommended prerequisites: a solid understanding of probability theory, differentiation, and integration.

Introduction

Uncertain Environment: The Challenge of Reducing Uncertainty and Associated Risks.

I - Information Theory

  1. Entropy of a distribution

     a) Locating an element in a probabilistic set. Optimal coding and entropy of a discrete variable. b) Entropy of a continuous variable. c) Entropy of a vector. d) General properties of entropy: affine transformation, independence.

   2. Mutual Information

     a) Mutual information and conditional entropy of two events. b) Mutual information and conditional entropy of two variables. c) Kullback-Leibler contrast. Chi-squared approximation. Application to the mutual information of two variables.

  3. An Introduction to Statistical Applications

    a) Selection of predictive variables. b) Regression and classification trees (CART). c) Classification using segmentation trees. d) Distribution selection in a parametric model. Pseudo-true distribution. e) Optimal recoding of a variable: unsupervised vs. supervised cases.

II - Decision Theory

  1. Background and Research Question.

    a) Random experiment, natural state, decision, loss, pure/mixed decision rule, risk. b) Pre-order on decision rules.

  2. Some common problems:

   a) Point estimate. b) Set-theoretic estimate. c) Hypothesis testing. d) Diagnosis (classification).

  3. The pre-order on decision rules.

    a) General lack of an optimal rule: no optimal estimator, no optimal test.

     b) Admissible rules. c) Essentially complete class of rules. Convexity theorem.

  4. Statistical Principles & Rule Selection

     a) Principles for transforming a partial preorder into a total preorder: the minimax principle, Bayes' principle.

    b) Selection principles: unbiased rules, rules based on information theory.

  5. An In-Depth Look at the Bayesian Framework

    a) Prior density of the parameter. Joint density of the parameter and the observations. Posterior density of the parameter.

    b) What priors? Conjugate priors. Uninformative priors: uniform prior, Jeffrey prior.

    c) Bayesian risk. Bayes' rule: Bayesian estimator and Bayesian test. Credibility interval.

Hourly volumes:

            CM: 9

            TD: 9

            Practical Work: 

            Lot: