ArviZ LOO cross-validation

[AlfredoJF]

Hi @ulfaslak,

Do you have any examples regarding ArviZ LOO cross-validation you mentioned in the Q&A of your Time-Varying Coefficients in PyMC-Marketing talk?

You mentioned 10 samples and an in-sample test but I'm not quite sure/familiar with how I can set it up and iterate through some parameters without being a computationally expensive task.

Thanks!

[wd60622]

After fitting each MMM, you can using the arviz and pymc model comparison functions

import pymc as pm
import arviz as az

from pymc_marketing.mmm import MMM

first_model: MMM
second_model: MMM
models: list[MMM] = [first_model, second_model]

for model in models:
    pymc_model: pm.Model = model.model
    with pymc_model:
        pm.compute_log_likelihood(model.idata)

df_compare = az.compare({
    f"model_{i}": model.idata
    for i, model
    in enumerate(models)
})

references:

• az.compare: https://python.arviz.org/en/v0.17.0/api/generated/arviz.compare.html
• PyMC model comparison: https://www.pymc.io/projects/docs/en/stable/learn/core_notebooks/model_comparison.html

Related to #352

[AlfredoJF]

Thanks @wd60622, I'll try this one and let you know how it goes.

@ulfaslak still curious about the 10 samples you mentioned in the webinar, not sure if something was shared in the chat. Sorry, I missed it because it was like 2 am for me and couldn't find a chat history only the YouTube video.

Thanks both!

[AlfredoJF]

@wd60622 Thanks for the sample code, it worked well. Just one question, how can this method be compared to Out-of-sample R^2 or MAPE?

My question is because the best model based on elpd_loo, does not get the best R^2 and MAPE and I'm confused about which model to use in production.

In my case, I have the luxury of being able to split my data into training and testing but at the end of the day, I need to use the model with the best performance using the most up-to-date data.

Thanks!

[wd60622]

Hi @AlfredoJF
Are you saying that the LOO on the training set suggests a different model than the R^2 or MAPE on the test set?

If you just care about out of sample predictions and would like to pick a model based on that criteria, then I would just make use of the sample_posterior_predictive method of the MMM class. Ref: https://www.pymc-marketing.io/en/stable/api/generated/pymc_marketing.mmm.delayed_saturated_mmm.MMM.sample_posterior_predictive.html

[AlfredoJF]

Correct, that's why I'm a bit confused.

If you just care about out of sample predictions and would like to pick a model based on that criteria, then I would just make use of the sample_posterior_predictive method of the MMM class. Ref: https://www.pymc-marketing.io/en/stable/api/generated/pymc_marketing.mmm.delayed_saturated_mmm.MMM.sample_posterior_predictive.html

In that case, should I use in-sample R^2 and MAPE to do the model selection from a pool of models and then the out-of-sample from the top 3-5?