Module 25 ยท How do you compare and inspect models?
Choosing well, and looking under the hood
- Weigh models on simplicity, consistency, coverage, and plausibility.
- Use condTbl() and condition() to inspect a formula.
- Resist choosing a model on numbers alone.
When CNA hands you several candidate models, weigh them on four criteria, three statistical and one stubbornly human:
- Simplicity. Fewer conditions are easier to explain and easier to act on.
- Consistency. How reliably the pattern holds.
- Coverage. How much of the outcome it accounts for.
- Program plausibility. Does it actually make public-health sense?
To inspect any formula's scores directly, use condTbl(), which builds a tidy summary table of consistency and coverage for whatever conditions you pass it. To probe a formula facility by facility, use condition().
library(cna) # score a specific formula against the data condTbl("NoCold + NoSuper <-> LowCov", immunization) # see exactly which facilities each road covers condition("NoCold + NoSuper", immunization)
LowCov 1.000 1.000
Never pick a model just because the numbers are the prettiest. A statistically flawless pattern that no program officer can explain or act on is, in practical terms, useless. The model has to make public-health sense first, and impress the statistician second.
Consistency and coverage are the defaults, not the only options
This course uses standard consistency and coverage throughout, because they are the defaults and the easiest to learn. From version 4.0.0 onward, the cna package also offers several additional sufficiency and necessity measures (including prevalence-adjusted ones) through a measures argument in cna(). You do not need them to complete this course, but once you are comfortable, they are the natural next step for handling tricky or imbalanced data.
Imagine two models with similar scores.
- Which would you trust: the tidier one, or the one that makes more program sense?
- What would make a statistically strong model still unusable in practice?