"Our data are nested — do we need a multilevel model?" is one of the questions I get asked most often, and the honest answer is: not always, but ignoring nesting when it matters can quietly invalidate your standard errors, and by extension your p-values, without anything in your output flagging that something's wrong. Here's the short version of how I think through it.
Step 1: Is there meaningful clustering in your outcome?
Nesting alone — students in classrooms, patients in clinics, repeated measures within people — doesn't automatically require a multilevel model. What matters is whether observations within the same cluster are more similar to each other than to observations in other clusters, on the outcome you care about. This is the intraclass correlation (ICC). If the ICC is close to zero, a single-level model may be a reasonable approximation. As a rule of thumb, I get concerned once the ICC is above roughly 0.05, and I treat anything above 0.10 as a clear signal that ignoring clustering will meaningfully bias your standard errors — almost always downward, which inflates your Type I error rate.
Step 2: Does your research question live at more than one level?
Even with a low ICC, if your hypotheses are explicitly about a higher-level effect — does a school-level policy predict student outcomes, does a clinic-level intervention affect patient outcomes — you need a model that represents both levels explicitly, both to test the effect correctly and to avoid the ecological fallacy of drawing individual-level conclusions from aggregated data (or vice versa).
Step 3: What's your cluster count, not just your total N?
This is the constraint that catches people off guard. A multilevel model's precision for estimating level-2 effects (like a school- or clinic-level predictor) depends primarily on the number of clusters, not the total sample size. Thirty students in each of five schools is a very different design problem than five students in each of thirty schools, even though both give you N=150. As a general guideline, fewer than 20–30 clusters starts to strain the assumptions behind standard maximum-likelihood estimation, and you may need small-cluster corrections or a Bayesian approach with informative priors on the variance components.
Step 4: Are you actually interested in the variance, or just the fixed effects?
If your only goal is a design-based correction to standard errors — you don't care about between-cluster variance itself — a simpler approach like cluster-robust standard errors or GEE may get you what you need with less complexity and fewer distributional assumptions than a full multilevel model. Multilevel models earn their complexity when you actually want to characterize how much outcomes vary between clusters, test cross-level interactions, or allow effects to vary randomly across clusters.
A short checklist
- Estimate the ICC before deciding — don't assume nesting is negligible just because it's inconvenient.
- Match the model to where your research question actually lives, not just to the structure of the data.
- Count your clusters, not just your total sample size, before trusting level-2 estimates.
- Ask whether you need the variance components, or just correct standard errors — the answer changes which model is worth the added complexity.
If you're mid-analysis and not sure which of these applies to your data, I'm glad to take a look. Get in touch.