Part 10 · How do you evaluate a model?
Module 23 · What is consistency?
Consistency: does the pattern keep its promise?
- Define consistency in plain language.
- Read a consistency score.
- Set the con threshold when data is noisy.
CNA gives every solution two scores. The first is consistency, and it answers a simple question: when this combination of gaps is present, does low coverage actually show up too?
Consistency is "how often the pattern holds"
A consistency of 1.00 means every facility with the combination also had the outcome, so the pattern never let you down. A consistency of 0.80 means it held 80% of the time. Higher is better. CNA's default threshold is 1, and you lower it when real-world data gets noisy, which it always does.
Consistency is "every time I leave the milk out of the fridge, does it go off?" If it goes off every single time, that's perfect consistency. If it sometimes survives, the consistency drops, and you start to wonder what else is going on.
Consistency can read deceptively high when the outcome itself is very common in your data. If, say, 90% of your facilities already have low coverage, almost any combination will look "consistent" with it, because the outcome is everywhere. So always read a consistency score next to how common the outcome is, and lean on the robustness checks in the next part rather than trusting a single high number. Recent work by De Souter and Baumgartner (2025) adds prevalence-adjusted measures precisely to handle this.
In code, you set the bar with the con argument:
# accept patterns that hold at least 90% of the time cna(immunization, outcome = "LowCov", con = 0.9)
Think about reliability.
- If a recipe works 8 times in 10, is that good enough to act on in your setting?
- Where would you set the bar, and what would you risk by lowering it?