Tukukkk

I was training my multiclass classifier when I came across two optimization functions in Python, one is scipy.optimize.fmin_cg and other is scipy.optimize.minimize. I read the documentation but couldn't figure out much. What are the basic differences between these two and in which specific cases should I use them?

If you go to the documentation of the minimize function (doc), after the argument list, the methods are specified with references to articles describing the details.

You can actually find the following line there: "Method CG uses a nonlinear conjugate gradient algorithm", which is exactly what fmin_cg does. So minimize is a more general function, it can use different functions, by selecting the cg it will do the same as the more specific fmin_cg function. (In the documentation of fmin_cg you can read exactly this: "common interface to all scipy.optimize algorithms for unconstrained and constrained minimization of multivariate functions. It provides an alternative way to call fmin_cg, by specifying method='CG'.")

So, when should you use fmin_cg? It is also answered in the documentation: "Conjugate gradient methods tend to work better when:

• f has a unique global minimizing point, and no local minima or other stationary points,


• f is, at least locally, reasonably well approximated by a quadratic function of the variables,


• f is continuous and has a continuous gradient,


• f' is not too large, e.g., has a norm less than 1000,


• The initial guess, x0, is reasonably close to f ‘s global minimizing point, xopt."

And when should you use minimize? Minimize is just a collection of optimization algorithms. So the real question is when should you use something else. Well, read the description of the methods in the docs and try to figure out. Possibly find the specific methods which will be called by minimize (like in this case fmin_cg) for some more detailed description.