Every Combination of Settings, Best First

Tuning a model means trying combinations of settings. A first version has one nested loop per setting, so adding a setting means rewriting the function. Write a general best_settings(score, grid, *, top=1):

• grid is a dictionary from a setting's name to the list of values to try, for example {"rate": [0.1, 0.01], "depth": [2, 4, 8]}.
• Every combination is tried, in the order that takes the first setting's values slowest and the last setting's values fastest (for the example: (0.1, 2), (0.1, 4), (0.1, 8), (0.01, 2), ...). A combination is passed to score as keyword arguments, score(rate=0.1, depth=2), and score is called exactly once per combination.
• Return the top best combinations as a list of pairs (score value, settings dict), highest score first. Among equal scores, the combination tried earlier comes first. If there are fewer than top combinations, return them all.
• top is keyword-only; top < 1 raises ValueError. An empty grid {} is one combination with no settings; a setting with an empty list of values means there are no combinations at all.

Setup helpers, available with Run: search(name, grid, top=1) runs your function with one of the scores "forest", "ridge" or "flat" (a score that is the same everywhere) and returns (your result, number of score calls); raises(fn, *args, **kwargs) returns the name of the exception a call raises (or None).

Examples

Input:  best_settings(lambda a, b: -(a - 2) ** 2 - b, {"a": [1, 2, 3], "b": [0, 1]}, top=3)
Output: [(0, {"a": 2, "b": 0}), (-1, {"a": 1, "b": 0}), (-1, {"a": 2, "b": 1})]

Input:  search("flat", {"x": [1, 2], "y": ["p", "q"]}, top=2)
Output: ([(0.5, {"x": 1, "y": "p"}), (0.5, {"x": 1, "y": "q"})], 4)

Constraints

• At most 10**5 combinations. Scores are numbers.

Goals

• Generate every combination of options with `itertools.product` instead of nested loops
• Keep the best few results with `heapq.nlargest` and a stable tie-break
• Pass a combination to a function as keyword arguments with `**`