Often, in scientific computing, we are required to find the value of x for which a function f(x) will attain a maximum or minimum value. In other words, we want to maximize or minimize f(x). This process is termed as numerical optimization and can be summarized as follows:

In the preceding formula, x represents a vector of variables also known as the unknowns or parameters, f is the function of x we want to maximize or minimize known as the objective function, z_i is the constraint functions that x must fulfill, and N and M are sets of indices. Optimization problems are used in mathematics, finance, and computer science to find the best solution from all feasible solutions. We can simplify maximization optimization problems to minimization problems by remembering that the maximum of f(x) is essentially the minimum of -f(x). Therefore, to maximize the function f, we simply need to minimize f. In this chapter, you will learn different methods and functions used to perform...