Numerics for algorithm comparisons

[mattkingros]

Initial pull request during meeting

[mattkingros]

Added a local version of the Optim package to numerics folder.
Added piggybacking of tracer.jl for LBFGS for tracking of evaluations of f and grad(f).

To-do: add cg itns tracker to PDCO and plot results on shared axis

[mattkingros]

Added plot_results.jl and saved plots in /output folder, comparing by computational cost (matrix multiplications)

• PDCO not plotted, number of CG iterations is too large (as suspected)
• Comparison by matrix multiplications looks favorable for KLLS, particularily for small lambda

To-do:

• test for more values of lambda (log spaced, wider endpoints), plot results on new axis'.
• split small / large lambda into seperate plots so you can see results clearly.

[mpf]

Added a local version of the Optim package to numerics folder.

Added piggybacking of tracer.jl for LBFGS for tracking of evaluations of f and grad(f).

To-do: add cg itns tracker to PDCO and plot results on shared axis

Why copy the Optim.jl source code instead of just adding it as a package dependency?

[mattkingros]

The Optim.jl package didnt have the tracing options we needed!

Optim's LBFGS only reports functions/gradient values in its trace, even using extended tracer options.
Notably, it doesn't allow the user to trace function/gradient evaluation counts per iteration - and only reports these at termination.

So, ive changed Optim's tracer.jl file to report these per iteration.
(Frustratingly, it was already computing this and much more, like PDCO, but not allowing the user access)

If there is a cleaner way of replacing the default Optim tracer, please let me know!

[mpf]

@mattkingros Instead of copying in the entire Optim.jl package, I believe it's possible to modify the particular method after doing "using Optim".

For example, here I'm replacing a function inside of a module:

julia> module CoolMod
         hello()="Hello"
         call(x)=hello()
         end
Main.CoolMod

julia> CoolMod.call(1)
"Hello"

julia> CoolMod.hello() = "GoodBye"

julia> CoolMod.call(1)
"GoodBye"

You should be able to do something analogous for the tracer function you wish to modify.

[mattkingros]

@mpf Thanks for the help and MWE!

I'll take a look at this approach first thing tomorrow. If it is a simple override like your example, that would be a great fix.

[mattkingros]

@mpf A direct overwrite as you suggested is proving to be more difficult than expected.

Optim uses private types under-the-hood, and so the associated methods are resistant to being overwritten (and are throwing me a mix of "invalid redefinition of constant" and "type not defined" errors which have been unhelpful in finding a solution).

Do you have any time later today/this week for a quick chat to point me in the right direction?
In the meantime; I can clean up the plots with the previous version of the code!

[mattkingros]

• Repo cleaned up (Optim.jl Source code removed, replaced with forked repo as suggested)
• Scratch work and plots from Informs meeting saved, but moved to scratch folder.