Information and decision theories

Statistical modeling is based on the two fundamental notions of information (which must be extracted from the data) and decision (which must be taken in view of these data). This course introduces the theoretical formalization of these two notions. It is therefore logically placed at the beginning of the curriculum, as many other courses use its notions and results later on.

Introduce the two notions that are at the foundation of mathematical statistics: information (quantification of information and coding) and decision (quantification and risk management).

Course in probability theory.

Recommended prerequisites: a good command of probability calculus, derivation and integration.

Introduction

Random environment: the problem of reducing uncertainty and associated risks.

I - Information theory

  1. Entropy of a distribution

     a) Locating an element in a probabilized set. Optimal coding and entropy of a discrete variable. b) Entropy of a continuous variable. c) Entropy of a vector. d) General properties of entropy: affine variable change, independence.

   2. Mutual information

     a) Mutual information and conditional entropy of two events. b) Mutual information and conditional entropy of two variables. c) Kullback-Leibler contrast. Approximation of chi2. Application to the mutual information of two variables.

  3. Outline of statistical applications

    a) Selection of predictor variables. b) Regression trees and classification (CART). c) Classification by segmentation trees. d) Law selection in a parametric model. Pseudo-true law. e) Optimal recoding of a variable: unsupervised / supervised case.

II - Decision theory

  1. Framework and problematic.

    a) Random experiment, state of nature, decision, loss, pure/mixed decision rule, risk. b) Pre-order on decision rules.

  2. Some classical problems:

   a) Point estimation. b) Ensemblistic estimation. c) Hypothesis testing. d) Diagnosis (ranking).

  3. The pre-order on decision rules.

    a) General absence of optimal rule: absence of optimal estimator, absence of optimal test.

     b) Admissible rules. c) An essentially complete class of rules. Convexity theorem.

  4. Statistical principles & rule selection

     a) Principles allowing to transform the partial preorder into a total preorder: minimax principle, Bayes' principle.

    b) Selection principles: unbiased rules, rules based on information theory.

  5. Further development of the Bayesian framework

    a) A priori density of the parameter. Joint density of parameter and observations. Density a posteriori of the parameter.

    b) Which Priors? Conjugate priors. Non-informative priors: uniform prior, Jeffrey's prior.

    c) Bayes risk. Bayes' rule: Bayes' estimator and Bayes' test. Credibility interval.

Hourly volumes:

            CM : 9

            TD : 9

            TP: 

            Terrain: