Designing Short Forms with IRT and Constrained Test Assembly

1. Internal Consistency and the Risks of Scale Reduction

In Classical Test Theory (CTT), an observed composite score $X = \sum_{i=1}^K X_i$ is modeled as the sum of a true score $T$ and measurement error $E$, where Cronbach’s alpha summarizes internal consistency. Its interpretation as reliability requires assumptions about item structure and errors:

When a research team reduces an instrument from $K_{\text{full}}$ items to a short form of length $K_{\text{short}} \ll K_{\text{full}}$ by greedily selecting items that maximize $\alpha$ or single-factor loadings, three risks require investigation:

1. Narrow Content Coverage and Redundancy: For a fixed number of items, alpha rewards higher average inter-item covariance relative to item variance. This can favor redundant content and omit substantive facets. High semantic similarity warrants review, but local dependence must be assessed conditional on an appropriate measurement model; alpha alone does not diagnose it.
2. Threshold Clustering and Information Collapse: Item-total correlations depend on the calibration sample and item properties. They do not directly target precision at a clinical cutoff or elsewhere on the trait scale. Threshold clustering and weak tail information are possibilities to inspect in fitted information curves, rather than universal consequences of classical selection.
3. Measurement Error and Study Planning: Because the Conditional Standard Error of Measurement is inversely proportional to the square root of test information,
   $$\text{CSEM}(\theta) = \frac{1}{\sqrt{I(\theta)}}$$
   lower information implies greater conditional uncertainty under the fitted model. Measurement error can reduce sensitivity to effects, but its impact on power depends on the outcome, design, scoring, and error assumptions. The StatPower Short-Form Precision Simulator explores illustrative planning scenarios; it does not fit IRT models, run MILP assembly, or estimate actual study power from response data.

2. Psychometric Parameterization: Samejima’s Graded Response Model (GRM)

Let $i \in \{1, 2, \dots, I\}$ index candidate items in a calibrated source bank, where each item is scored on $m_i + 1$ ordered polytomous categories $k \in \{0, 1, \dots, m_i\}$ (e.g., $m_i = 4$ for a 5-point Likert scale). Under Samejima’s (1969) Graded Response Model, the cumulative probability $P_{ik}^*(\theta)$ that a respondent with latent trait $\theta$ endorses category $k$ or higher is given by the two-parameter logistic ogive:

where $a_i > 0$ is the item discrimination parameter and $b_{i1} < b_{i2} < \dots < b_{im_i}$ are strictly ordered category threshold parameters. By boundary definition, $P_{i0}^*(\theta) = 1$ and $P_{i, m_i+1}^*(\theta) = 0$. The probability of responding in exact category $k$ is:

2.1 Item and Test Fisher Information

The Fisher information contributed by category $k$ of item $i$ at trait level $\theta$, denoted $I_{ik}(\theta)$, and the total item information $I_i(\theta)$ are derived from the first derivatives of the category response functions:

Under local independence—where item responses are conditionally independent given $\theta$—the Test Information Function (TIF) of any subset of items is strictly additive. Letting $x_i \in \{0, 1\}$ denote a binary decision variable indicating whether item $i$ is selected into the short form, the Test Information Function at point $\theta_q$ is a linear function of $\mathbf{x}$:

This linearity with respect to $x_i$ is the foundational property that allows IRT scale reduction to be optimized via Mixed-Integer Linear Programming, conditional on fixed item parameters, the chosen grid and constraints, and a solver-certified optimum within numerical tolerances.

3. Mathematical Formulation of Constrained MILP Test Assembly

Rather than relying on heuristic step-wise item deletion or stochastic genetic algorithms, we formulate short-form assembly using van der Linden’s (2005) Maximin / Target-Interval Information MILP framework.

Let $\Theta_{\text{target}} = \{\theta_1, \theta_2, \dots, \theta_Q\}$ represent a discrete quadrature grid spanning the study’s primary decision region (for example, $\Theta_{\text{target}} = \{0.5, 1.0, 1.5, 2.0, 2.5\}$ for a clinical pathology screener, or $\{-1.5, -0.75, 0.0, 0.75, 1.5\}$ for a broad educational survey), with relative information requirements $w_q > 0$. We introduce a continuous auxiliary floor variable $y \ge 0$ representing the minimum ratio of information to requirement across all quadrature points in $\Theta_{\text{target}}$.

3.1 Objective Function (Maximin Target Information)

First maximize $y$, giving $y^*$. Then maximize total grid information $\sum_q\sum_i I_i(\theta_q)x_i$ subject to the same constraints and $y \ge y^* - \delta$, where $\delta$ is a small stated absolute tolerance. This two-stage procedure preserves the primary optimum within tolerance. The floor constraints imply $y = \min_q I(\theta_q)/w_q$ at the first optimum; a larger $w_q$ requests proportionally more information at that point. These are not probability or quadrature-integration weights.

3.2 Structural and Psychometric Constraints

4. Implementation and runnable examples

The assembly solvers accept a supplied information matrix and reviewed metadata. They use two-stage maximin optimization, content bounds, enemy-pair exclusions and optional DIF exclusions, with explicit failure handling. They do not calibrate a model or establish validity.

Download the R solver or Python solver. Each includes a runnable six-item synthetic check. The companion report provides the complete code, setup commands and expected output.

For calibration, held-out scoring and the simulation extensions, use the complete reproducibility package. The code and results distinguish fitted-model information from empirical scoring accuracy.

5. Score Linking & Raw-to-Scale Crosswalk Generation

Selecting the optimal item subset $\mathcal{S} = \{i : x_i^* = 1\}$ is only half of a production psychometric deliverable. Applied clinical and educational teams frequently score short forms via simple summed scores $S = \sum_{i \in \mathcal{S}} X_i$ in REDCap, Qualtrics, or EHR pipelines. Because summed scores on a 6-item measure are not directly comparable to summed scores on the 30-item source bank, parameters can be retained on a defensible common bank metric and summed-score distributions computed using a polytomous extension of the Lord–Wingersky (1984) recursive algorithm.

Using the calibrated item parameters $\{a_i, \mathbf{b}_i\}_{i \in \mathcal{S}}$, the conditional distribution of summed score $S = s$ given latent trait $\theta$, denoted $P(S = s \mid \theta)$, is computed recursively across items. Assuming a prior population density $g(\theta)$ (typically $\mathcal{N}(0, 1)$), the expected a posteriori (EAP) scale score $\hat{\theta}_s$ and Standardized T-score $T_s$ associated with raw summed score $s$ are:

Providing this lookup crosswalk table alongside the MILP-assembled short form can simplify scoring on the calibrated metric. This is EAP conditional on a summed score, not necessarily response-pattern EAP under the GRM. Report posterior uncertainty and the prior used. Longitudinal comparability and links to national norms require separate evidence about calibration, population, invariance and administration; a lookup table alone does not establish equivalence. The T-score transformation assumes the intended reference metric has mean 0 and standard deviation 1.