Optimization involves finding extreme values of functions or their roots. We have already seen the power of optimization in the curve-fitting arena, but it does not stop there. There are applications to virtually every single branch of engineering, and robust algorithms to perform these tasks are a must in every scientist's toolbox.

The curve_fit routine is actually syntactic sugar for the general algorithm that performs least-squares minimization, leastsq, with the imposing syntax:

leastsq(func, x0, args=(), Dfun=None, full_output=0,
        col_deriv=0, ftol=1.49012e-8, xtol=1.49012e-8,
        gtol=0.0, maxfev=0, epsfcn=0.0, factor=100, diag=None):

For instance, the curve_fit routine could have been called with a leastsq call instead:

leastsq(error_function,p0,args=(x,y))

Here, error_function is equal to lambda p,x,y: target_function(x,p[0],p[1],p[2])-y

The implementation is given in the corresponding section on the IPython Notebook of this chapter. Most of the optimization routines...