Should linesearch enforce that the objective monotonically decreases?

[ahwillia]

I am finding that running GradientDescent(...) with stepsize=0.0 blows up, presumably because the initial stepsize used by the linesearch is too large. When I use stepsize=1e-3, the problem I'm working on runs fine. Is there a way to specify the initial stepsize for the linesearch? Or should I open an issue to request that the linesearch functions be revamped to be more robust?

If others do not encounter this problem I can try to reproduce a minimal working example. But from skimming the linesearch code it doesn't seem like there is anything that forces the objective to monotonically decrease at each iteration.

[mblondel]

Line search can become numerically instable in float32 precision unfortunately. Something I would like us to do is to rewrite GradientDescent to use our line search implementations instead of subclassing ProximalGradient as we currently do. In the mean time, I would recommend to use LBFGS or NonlinearCG.