Lecture 8: Linear Optimization

This notebook introduces linear optimization (LO), also called linear programming. It is a widely used optimization framework in practice: efficient solvers exist that are guaranteed to find the optimum even for problems with thousands of variables and constraints. We introduce it through an example, model it, solve it in Python with pulp, look at what the solver is doing both graphically and algebraically, and see what can go wrong.

Learning outcomes

On completion of this notebook, you will be able to:

• formulate a real-life decision problem as an LO model (decision variables, objective, constraints);

• implement an LO model in pulp, both directly and with the data kept separate from the model;

• understand the intuition behind the simplex method;

• interpret an LO solution both graphically and algebraically;

• recognize infeasibility and unboundedness in pulp’s solve status.

A Motivating Problem: Optimal Product Mix¶

A good way to introduce LO is through a concrete example. Consider a workshop that makes two products, bookcases and desks, from two limited resources, oak panels and assembly hours. The following information is known per unit of product:

For example, creating 2 bookcases creates a profit of 6 and requires 2 oak panels and 4 assembly hours. There are 12 oak panels and 10 assembly hours available this week. The optimization problem is to find the product mix that maximizes profit this week. For simplicity, we assume continuous amounts of bookcases and desks can be made; Integer Optimization revisits this example with the added requirement that only whole numbers are allowed.

The Modeling Approach¶

We follow the same four steps for every problem in this course:

1. study the problem in detail (done above);

2. define the decision variables: let $x$ and $y$ be the number of bookcases and desks produced, respectively;

3. define the objective: maximize profit, $3x + 5y$;

4. define the constraints: the oak-panel usage, $x + 3y$, cannot exceed 12; the assembly-hour usage, $2x + y$, cannot exceed 10; and $x, y \ge 0$.

In mathematical form:

The objective and all constraints are linear functions of the decision variables $(x, y)$, hence the name linear optimization.

Solving the Model Directly in pulp¶

For a small, one-off model like this, the fastest way from model to answer is to write it in pulp exactly as it appears above, with the numbers typed directly into the variable bounds and constraints. We walk through it step by step; each step maps onto one of the four modeling steps above.

Import pulp¶

pulp is the Python package we use to build and solve LO (and, later, ILO) models. Let’s import it.

Introduce the LP Problem¶

Python class LpProblem is pulp’s container for a model: give it a name and a sense (LpMaximize or LpMinimize) upon initialization.

Specify the Decision Variables (step 2)¶

Each decision variable becomes an LpVariable object; lowBound=0 encodes $x, y \ge 0$. Optionally, we can set cat by giving a string that specifies the variable type, for example "Continuous", "Integer", or "Binary" (resulting in LpContinuous, LpInteger, or LpBinary, respectively). By default, LpVariable is continuous; we will see integer variables in Integer Optimization. Optionally, an upBound can be specified.

Specify the Objective Function (step 3)¶

Add the objective to the problem with +=; the optional string names it for pulp’s output.

Specify the Constraints (step 4)¶

Constraints are added the same way, one += call per constraint. The comparison operators <=, >=, and == signify that a constraint is added (and not an objective).

Printing the Model¶

For small models, it is often useful to print the model to check that it was built correctly. pulp’s print() method produces a human-readable version of the model.

Solve and Check the Results¶

We are ready to solve the LO problem. We first load a solver and then use it in .solve(); LpStatus reports whether it found an optimum, and .value() reads off the variables and the objective.

Separating Data from the Model¶

The direct version above hardcodes every number into the model itself, which is fine for a (small) problem you solve once. It breaks down as soon as you want to re-solve for different data (more products, different resource limits) or reuse the same model code elsewhere: every number would have to be found and edited inside the model-building code, which is slow and error-prone. The fix is to keep the instance data in plain Python structures, separate from the model-building code, and build the model from that data:

The model-building code (the for loop and the pulp.lpSum calls) no longer mentions 3, 5, 12, or 10 anywhere: it works unchanged for any number of products and resources, as long as profit, resource_use, and available describe them. This is the pattern we use for every larger model from here on, and it is also the discipline behind the algebraic modeling languages (AMLs) that were used for optimization models before Python libraries like pulp.

A Larger Example in pulp¶

The same three ingredients, and the same data/model split, scale to any size. Suppose the workshop adds a third product, chairs, that uses 2 oak panels and 1 assembly hour per unit and earns a profit of 4 per unit, and that both resources become a bit more plentiful: 15 oak panels and 18 assembly hours. Only the data changes; the model-building code stays exactly the same as in the previous section:

The model-building code is a line-for-line copy of the two-product version; only the data changed.

A Graphical View¶

With only two decision variables we can draw the product-mix problem. Each constraint is a line; the feasible region is the set of points satisfying all of them at once. Because both the constraints and the objective are linear, the objective-value contour lines are straight and parallel. Sliding one in the direction of increasing profit, the last feasible point it touches is an optimal solution. Indeed, this may be a line segment if the objective function is parallel to a constraint. However, there is always a corner point of the feasible region that is optimal.

Use the figure below to check your hand drawing, which you do need to master. Hover over a corner point of the feasible region to see its coordinates and profit.

The dotted grey line (profit 15) still has feasible points but we can do better; sliding it out until it just leaves the feasible region gives the solid line, which touches the region only at the corner where the two resource constraints meet: the optimum pulp reported above.

An Algebraic View¶

The same optimum can be understood without a picture. Turn each “$\le$” constraint into an equality by adding a non-negative slack variable, the amount of the resource left unused:

$s_1$ is the number of oak panels left unused, and $s_2$ is the number of assembly hours left unused. There are now 2 equations and 4 variables. As a rule, a system of $k$ independent linear equations in $k$ unknowns has generally one solution, so if we set any 2 of the 4 variables to zero the other 2 are determined (where the two lines corresponding to the equations cross). Each such choice corresponds to a corner of the feasible region (some combinations give a negative value and are infeasible; those are corners of the lines’ intersections that lie outside the feasible region). The corners here are:

• $(x, y, s_1, s_2) = (0, 0, 12, 10)$: the origin;

• $(5, 0, 7, 0)$: assembly hours fully used;

• $(0, 4, 0, 6)$: oak panels fully used;

• $(3.6, 2.8, 0, 0)$: both resource constraints tight, the optimum.

The simplex algorithm (G.B. Dantzig, 1947), which pulp’s default solver uses for LO, hops from corner to neighboring corner, each time to one with a better objective value, and stops when no neighboring corner is better. Why is that enough to guarantee the global optimum? Because the feasible region of an LO problem is a convex polyhedron: a shape with flat faces and no hidden “hills”. If every neighbor of your current corner is worse, there is nowhere higher to go, since reaching a higher point would require the boundary to curve, and linear constraints never curve. So for LO, a local optimum is automatically a global optimum.

Three Possible Outcomes¶

Not every LO problem has an optimal solution. There are exactly three possibilities:

1. an optimal solution exists (the case above);

2. unbounded: solutions of arbitrarily large objective value exist. This cannot happen for the product-mix problem above, since every product uses some of both resources, but it can happen in general whenever some variable is left unconstrained, for example if the workshop also sold a service that used neither oak panels nor assembly hours;

3. infeasible: no point satisfies all constraints. This happens if, for example, a customer contract forces $y \ge 5$ while the oak-panel constraint allows at most $y = 4$.

pulp reports these as the LpStatus values "Unbounded" and "Infeasible" instead of "Optimal". Figure 2 shows both situations schematically.

In the following we illustrate the unbounded and infeasible errors in pulp for the product-mix problem. For unboundedness, we add a third variable $s$ that uses no resources at all; for infeasibility, we add a customer contract that requires $y \ge 5$:

Another Application: Project Planning¶

A graph is a network of nodes (also called vertices) connected by edges; a directed edge is called an arc. Many problems are naturally expressed on graphs. One is project planning: a project has activities (nodes), each with a duration, and precedence relations (arcs). An activity can only start once its predecessors are finished (you cannot roof a house before the walls are up).

Let $x_i$ be the finish time of activity $i$. If $i$ precedes $j$ then $x_i + d_j \le x_j$, where $d_j$ is the duration of $j$; also $x_i \ge d_i$. We want the time when all activities are finished, $\max_i x_i$. That maximum is not linear (question to yourself: why not?), but we can introduce a variable $z \ge x_i$ for all $i$ and minimize $z$: the optimization will push $z$ down to exactly the largest finish time. This “min-max” trick reappears throughout the course.

The finish time can also be found by a direct algorithm without LO, and that algorithm extends to random durations, which matters because durations are often hard to predict, one reason IT projects overrun.

References¶

• Koole, G. (2019). An Introduction to Business Analytics. §6.1 “Problem Formulation”, §6.3 “Example LO Problems” (project planning).