Problem 660404 · hard · Level 06 Heuristics & Optimization

Which Model Explains the Claims?

model comparison · AIC · BIC · zero-inflated Poisson · negative binomial · numerical maximisation

An insurer records how many claims each customer made last year. A Poisson model is the simplest choice, but claims data often has more zeros than a Poisson model allows (many customers never claim) or more spread (some customers are much riskier than others). Compare three models, each fitted by maximum likelihood:

  • Poisson with rate lam (1 parameter): P(x) = lam**x * exp(-lam) / x!.
  • Zero-inflated Poisson (2 parameters pi, lam, with 0 <= pi < 1): with probability pi a customer never claims, otherwise their count is Poisson(lam). So P(0) = pi + (1 - pi) * exp(-lam) and P(x) = (1 - pi) * lam**x * exp(-lam) / x! for x >= 1.
  • Negative binomial (2 parameters r > 0, 0 < p < 1): P(x) = Γ(x + r) / (Γ(r) * x!) * p**r * (1 - p)**x.

A richer model can always get at least as close as the Poisson model: when the data have no excess zeros, the best zero-inflated fit has pi = 0, and when their variance (dividing by n) is not larger than their mean, the negative binomial's likelihood only approaches the Poisson maximum as r grows without limit. In those cases report the Poisson maximum as that model's log-likelihood.

Write compare_count_models(counts) that returns a dict:

  • "poisson", "zero_inflated", "negative_binomial": each a tuple (loglik, aic, bic) with the maximised log-likelihood, AIC = 2k - 2 * loglik and BIC = k * ln(n) - 2 * loglik, where k is the model's number of parameters (1, 2 and 2) and n = len(counts);
  • "best_aic" and "best_bic": the name of the model with the smallest AIC and the smallest BIC (on a tie, the first in the order poisson, zero_inflated, negative_binomial).

The log-likelihoods must be accurate to within 1e-6 relative error. The setup provides claim_counts(n, kind, seed), which simulates n customers under the model kind ("poisson", "zero_inflated" or "negative_binomial").

Examples

Input:  counts = [0, 1, 0, 2, 5, 0, 0, 3, 0, 1]
Output: {"poisson": (-17.08453971104259, 36.16907942208518, 36.471664515079226),
         "zero_inflated": (-15.147407324156179, 34.294814648312354, 34.89998483430045),
         "negative_binomial": (-15.121058428130631, 34.24211685626126, 34.847287042249356),
         "best_aic": "negative_binomial", "best_bic": "negative_binomial"}

Input:  counts = [1, 2, 1, 0, 2, 1]
Output: {"poisson": (-7.307239602329081, 16.61447920465816, 16.406238673886218),
         "zero_inflated": (-7.307239602329081, 18.61447920465816, 18.197998143114273),
         "negative_binomial": (-7.307239602329081, 18.61447920465816, 18.197998143114273),
         "best_aic": "poisson", "best_bic": "poisson"}
Explanation: fewer zeros and less spread than a Poisson model expects, so both
richer models fall back to it and pay the penalty for their extra parameter.

Constraints

  • 1 <= len(counts) <= 20000, counts are whole numbers from 0 to 200
  • floats are compared with a tolerance of 1e-6

Goals

  • Fit three competing count models by maximum likelihood, two of them numerically
  • Recognise when a richer model's best fit falls back to the simpler one
  • Compare models of different sizes by log-likelihood with a penalty for extra parameters
Starting Python…