Add caching to optimization to avoid duplicate work?

[pavelkomarov]

This would need to involve memory that's accessible across processes, which is nontrivial. From what I'm seeing, duplicate calls are pretty rare, so it's possible the overhead would be worse than the occasional method re-solve. But without characterizing, we can't be totally sure.

[pavelkomarov]

Here are a couple of histograms of optimization runs on robustdiff, which has 3 hyperparameters: order, qr_ratio, and huberM. Seems like there is indeed a lot of extra work being done.

Code:

s = """big printout of many `print(f"order = {order}, huberM = {huberM}, qr_ratio = {qr_ratio:.3e}` inside `robustdiff`"""

from collections import defaultdict
from matplotlib import pyplot

d = defaultdict(int)
for line in s.splitlines():
	d[line] += 1

pyplot.hist([v for k,v in d.items()], bins=100)
pyplot.show()

[pavelkomarov]

It turns out making a global dictionary that all processes can access isn't too hard, can done conveniently with multiprocessing.manager.dict. But hashing numpy arrays and dicts is pretty annoying, and doing it consistently across processes is extra annoying. I think I finally found a way:

key = sha1((''.join(f"{v:.3e}" for v in point) + # This hash is stable across processes. Takes bytes
               ''.join(str(v) for k,v in sorted(categorical_params.items()))).encode()).digest()

The .:3e puts things in scientific notation with 3 significant figures, my way of making nearby points have the same hash. Categoricals are a bit easier in that regard, because they shouldn't be floats, but I'm making sure to be extra cautious and sort the dictionary in case insertion order (which becomes iteration order in python 3.7+) varies across processes.

This histogram's 1-bucket has more entries in it because between last update and this one, I realized huberM was getting cast to integer but should really be float. Making it float produces a whole lot more unique calls, but there are still 162 repeats identified out of about 1200 calls. Saves about 100 seconds out of 450ish on my machine.

Here's another, with triangles instead of pi_cruise_control:

Found and averted 189 dupes. Tried with the sine example too and averted 110 dupes. Both of these are also out of around 1200 calls.

It's not nothing, so I'm going to fold this in as part of #161.

[pavelkomarov]

I wasn't properly passing through the huberM parameter, and doing so caused a solver failure, which I'm now accounting for by returning NaNs occasionally. Doesn't hurt the optimization loop. New total shortcircuits for the sims with these changes:

• pi_cruise_control: 123/1261
• sine: 129/1278
• triangle: 133/1268
• pop_dyn: 96/1276
• linear_autonomous: 184/1292
• lorenz_x: 119/1238

And no failures besides! I'll clean up some print statements and consider this g2g.