Optimization

This course is a continuation of the optimization course offered in the second semester of the third year of the bachelor’s program.

After reviewing the results and numerical methods for first- and second-order optimization problems—both unconstrained and subject to equality and inequality constraints—the course focuses on topics currently of interest in industrial optimization, particularly robust, multi-criteria optimization under uncertainty.

The course then illustrates the role of optimization in the main machine learning algorithms. These topics are illustrated with examples of classification and regression problems in supervised learning. These examples provide an opportunity to discuss metrics and procedures for evaluating learning, validation, and inference (cross-validation, overfitting, etc.).

The course introduces the different types of learning: unsupervised, supervised, transfer learning, reinforcement learning, incremental learning, etc.

Topics related to database management are covered: generation, imputation, visualization, and segmentation.

The course explores the connections between transfer learning in mathematics and numerical simulation to address issues such as the generation of synthetic datasets, imputation, non-intrusive prediction, fast inference, and more.

The course includes a significant number of ongoing computer projects. All sessions take place in a computer-based environment and allow students to immediately apply the theoretical concepts they have learned.

Establish a connection between numerical optimization and mathematics education. Explore machine learning through real-world examples.

Foundations of analysis, numerical solutions to ordinary differential equations, numerical linear algebra, programming exercises in an interpreted language.

Recommended prerequisites: L3, second-semester optimization course. Python programming.

-Constraint-free optimization

-First-order methods: Gradient Descent, Conjugate Gradients

-Separable functions, stochastic gradient descent, coordinate descent

-Second-order methods: Newton, Quasi-Newton (BFGS, L-BFGS).

-Techniques for evaluating the gradient of a functional (finite differences, complex variables, the adjoint, automatic differentiation).

-How to Do Without Hessian

-Optimization Subject to Equality and Inequality Constraints

-Lagrangian

-Interpretation of Lagrange multipliers

-Primal-Dual Problem-

-Quadratic minimization subject to linear constraints

-Lagrangian Saddle Point-

-Uzawa's algorithm

-Comparison of Penalty Methods, Primal-Dual Methods, and Uzawa Methods, Augmented Lagrangian

-KKT Conditions

-Complementarity Requirements

-Projected gradient algorithm, projected Uzawa, external penalization

-Global optimization, moment methods, ADAM, RMSprop

-Multicriteria optimization, Pareto front

-Robust interval optimization under uncertainty

-Adjustment techniques (L1, L2, …)

-Illustrations in Python

-Machine learning optimization: linear models, logistic regression, wide-margin classifiers, random trees and forests, neural networks.

-Dimensionality reduction: principal component analysis, singular value decomposition

-Databases and Imputation

-Python Scikit-Learn library.

Hourly volumes:

            CM: 21

            TD: 21

            TP: 0

            Land: 0