Course focus

This course focuses on tractable optimization problems, that is, problems that can be reliably and efficiently solved on a computer. Specifically, we study convex optimization, with emphasis on problems involving convex quadratic constraints and objectives.

Philosophy and the main message

We follow the philosophy that has guided the development of linear algebra. While nearly every practical engineering problem is non-linear, the impact of linear algebra algorithms on engineering has been immense so far and continues to be. The reason is that there is a strong interplay, in engineering modeling, between what we’d like to do (model systems as accurately as possible) and what we can do (analyze or design simple models). Thus, while non-linear, complex models may be attractive, when concerned with practical analysis or design problems, engineers often rely on simpler (linear) models to perform computations or find approximate solutions.

Linear algebra has provided a nice trade-off between computer tractability and model accuracy. We view convex optimization as an extension of linear algebra, as it holds the promise of providing as reliable and efficient solutions, with a much wider application range.

Course goals

Our goal is to help develop a sense of what convex optimization is, and how it can be used in practical contexts. We would like to empower students to use new convex optimization software such as CVX, which allows them to quickly prototype convex optimization models.

We review some theories, including weak duality, which allows using convex optimization to develop approximations to hard, non-convex problems.

Optimization models by nature try to fit the solution to the available data. In general, the solution might exhibit extreme sensitivity to changes in the problem data. This appears to make any attempt at optimization a dangerous proposition. One of the main messages of this course is that there exist remedies to this sensitivity issue, from simple to sophisticated, which allow computing very robust solutions at a moderate drop in performance.

This course does not provide details on algorithms, although we briefly discuss important classes of algorithms along our way. Students are encouraged to use the wonderful modeling software CVX to get used to convex optimization.

Lectures outline

The lectures are divided into five parts.

Part I

In this introduction, we discuss optimization models and some basic definitions., analytics{UA-35362163-1}

Part II

Part II is entirely devoted to linear algebra, from basic concepts such as vectors and matrices to more advanced ideas such as singular value decomposition and principal component analysis. Least-squares and some variants are included in this part, as it can be solved with the traditional tools of linear algebra.

Part III

We are then equipped to introduce convex optimization problems. We describe briefly what convex optimization is, and how convex problems are solved in practice. We emphasize the practical importance of the notion of ‘‘disciplined convex programming’’ on which convex modeling software such as CVX rests. We then review a number of “standard” convex models, and applications. One topic in this part explores difficulties and some solution approaches for optimization problems with uncertain data.

Part IV

Part IV is a gentle introduction to the key notion of duality. We show how the approach of weak duality allows using convex optimization to approximately solve non-convex problems. The notion of strong duality, which applies to most convex problems, allows deriving rigorous optimality conditions, algorithms. It has many other applications, ranging from decentralized algorithms for large-scale convex problems to dimensionality reduction for problems with a large number of variables and a small number of constraints, or vice-versa.

Part V

Part V collects a few practical case studies that are evoked in the previous parts and more. The applications range from image processing to circuit design, antenna array design, and portfolio optimization.