Information and decision theories

Statistical modeling is based on two fundamental concepts: information (which must be extracted from data) and decision-making (which must be based on this data). This course introduces the theoretical formalization of these two concepts. It is therefore logically placed at the beginning of the curriculum, as many other courses use these concepts and results later on.

Introduce the two concepts that form the basis of mathematical statistics: information (quantification of information and coding) and decision-making (quantification and management of risk).

Probability theory course.

Recommended prerequisites: a good command of probability calculations, differentiation, and integration.

Introduction

Random environment: the issue of reducing uncertainty and associated risks.

I - Information theory

  1. Entropy of a distribution

     a) Locating an element in a probability space. Optimal coding and entropy of a discrete variable. b) Entropy of a continuous variable. c) Entropy of a vector. d) General properties of entropy: affine variable change, independence.

   2. Mutual information

     a) Mutual information and conditional entropy of two events. b) Mutual information and conditional entropy of two variables. c) Kullback-Leibler contrast. Chi2 approximation. Application to the mutual information of two variables.

  3. Outlines of statistical applications

    a) Selection of predictive variables. b) Regression and classification trees (CART). c) Classification using segmentation trees. d) Selection of distribution in a parametric model. Pseudo-true distribution. e) Optimal recoding of a variable: unsupervised/supervised cases.

II - Decision theory

  1. Context and issues.

    a) Random experiment, state of nature, decision, loss, pure/mixed decision rule, risk. b) Preorder on decision rules.

  2. Some common problems:

   a) Point estimation. b) Ensemble estimation. c) Hypothesis testing. d) Diagnosis (classification).

  3. Pre-order on decision rules.

    a) General absence of optimal rules: absence of optimal estimators, absence of optimal tests.

     b) Admissible rules. c) Essentially complete class of rules. Convexity theorem.

  4. Statistical principles & rule selection

     a) Principles for transforming partial preorder into total preorder: minimax principle, Bayes' principle.

    b) Selection principles: unbiased rules, rules based on information theory.

  5. Deepening the Bayesian framework

    a) Prior density of the parameter. Joint density of the parameter and observations. A posteriori density of the parameter.

    b) Which Priors? Conjugate Priors. Non-informative Priors: Uniform Prior, Jeffrey's Prior.

    c) Bayesian risk. Bayes' rule: Bayesian estimator and Bayesian test. Credibility interval.

Hourly volumes:

            CM: 9

            TD: 9

            TP: 

            Land: