Short-Form Selection and Scoring: Companion Simulation Report

7. Fitted-data demonstration: information gains and scoring tradeoffs

Simulated responses · Fitted training model · Independent held-out people · Not a clinical instrument

We generated 2,400 responses to 30 hypothetical five-category items under one unidimensional graded response model (seed 20261021). The first 1,600 responses calibrated the GRM; the remaining 800 were reserved for evaluation. Item selection and Q₃ screening used training data only. No item wording, real respondents or clinical labels are used.

Three six-item candidates are compared with the full 30-item bank: top corrected item-total correlations; top two correlations within each illustrative content group A/B/C; and constrained maximin assembly with exactly two items per group. The last method targets fitted information on θ = 0.5–2.0 in 0.1 increments. These comparators rank item-total correlations; they do not exhaustively maximize alpha. Content groups are blueprint metadata on one latent dimension.

The information objective improves; held-out scoring does not. The constrained form selects items 8, 10, 17, 20, 29, 30. Its minimum target information is 5.595, versus 2.924 for item-total selection. Yet target-region EAP RMSE is 0.456 versus 0.406. The paired RMSE difference is 0.050 (95% percentile bootstrap interval 0.003 to 0.097; 2,000 resamples of held-out people). This interval excludes calibration and selection uncertainty.

Information optimization is conditional on a model and objective. It does not guarantee lower finite-sample EAP error or better decisions. Prior shrinkage, scoring bias and threshold placement are possible contributors; this single experiment does not isolate why the ranking differs. The candidate should therefore be evaluated against the intended scoring and decision criteria before use.

Model checks and boundaries

The training fit converged. The C₂ test is 453.99 on 405 degrees of freedom (p = 0.0465); RMSEA is 0.0087 and SRMSR is 0.0181. The p-value is below .05 despite small approximate-fit indices; these diagnostics are reported together, not treated as automatic proof of fit. Mean training Q₃ is -0.029; zero pairs exceed the descriptive mean-centered Q₃ threshold of .20. Thus this run does not demonstrate an active enemy-pair exclusion. No DIF or readability assessment is claimed.

This initial result uses one seed from a correctly specified generating family and excludes parameter uncertainty and real-world validation. Section 7.2 below repeats sampling, calibration and selection for the same bank. Classification metrics in the downloads use the arbitrary cutoff θ ≥ 1 against simulated true θ. They are not diagnostic sensitivity or specificity for a clinical condition. Fitted information, EAP RMSE and posterior SD answer different questions.

Reproduce and inspect

Download the complete fitted-demo package (ZIP) · Methods and setup · Results JSON · Independent numerical checks · Held-out scores CSV

Extract the ZIP and run from its root. Install R packages mirt, lpSolve and jsonlite, plus Python packages NumPy, SciPy and Matplotlib; the README gives commands and recorded versions.

Rscript scripts/fitted-short-form-demo/run_demo.R
python scripts/fitted-short-form-demo/check_and_publish.py

Python independently checks information and EAP scoring from the frozen R calibration to an absolute error below 10⁻⁹ and selects the same constrained items with SciPy. It does not independently calibrate the GRM. See the mirt calibration documentation and scoring documentation.

7.1 Why the scoring rule matters: bias and variance

To go beyond one holdout sample, we enumerated all 15,625 possible response patterns for each frozen six-item form. Response probabilities use the known generating parameters; scores use the fitted parameters. At each θ, this gives the conditional expected score, bias, variance and mean squared error by direct summation. MSE = bias² + variance. The calculation is conditional on this one fitted bank and its selected forms, rather than a repeated-calibration simulation.

Both components contribute in this fitted bank. Under normal-prior EAP, the constrained form has higher integrated squared bias (0.0453 versus 0.0371) and variance (0.1502 versus 0.1410) than item-total selection. Its target mean bias is −0.208 versus −0.173. The population-weighted conditional RMSE is 0.442 versus 0.422; these are model-based expectations over responses and θ, not the finite holdout values of 0.456 and 0.406.

Prior sensitivity changes the ranking, without improving this form’s scoring error. Keeping the same fitted coefficients, item selections, response patterns and scoring grid, replacing the N(0,1) prior with equal grid weights on [−6,6] reduces constrained-form squared bias to 0.0055 but raises variance to 0.2421. Its RMSE increases from 0.442 to 0.498. With this flat-grid posterior mean it performs better than the two comparators, whose RMSEs also worsen. Thus prior weighting affects the comparison, but removing normal-prior shrinkage is not a demonstrated solution. The flat-grid score is a diagnostic perturbation, not maximum-likelihood scoring or an endorsed replacement.

The downloads also include empirical held-out errors in half-unit θ bins, with bin counts. Their error variance includes within-bin variation in θ and bias, so it must not be read as conditional score variance at a fixed θ. This analysis explains the frozen-bank comparison; the repeated-response study below examines stability across calibration samples, while Section 7.3 examines a heuristic selection objective aligned with scoring loss.

Conditional decomposition CSV · Held-out error bins CSV · Prior-sensitivity results JSON

7.2 Repeating calibration and selection across response samples

We fixed the generating bank from the original example and specified seeds 20261101–20261130 before running this extension. Each seed generates a fresh sample of 1,600 training and 800 held-out respondents, recalibrates the GRM, repeats training-only screening and item selection, then scores the held-out respondents with the fitted bank and the same N(0,1) EAP prior. These are 30 response-sample replications under one bank, not 30 different instruments.

30 of 30 attempts completed successfully; 0 failed. Every attempt is retained in the JSON log, including warnings and failure messages. Numerical summaries below use successful attempts. The selection objective, target grid, content requirements, scoring prior and evaluation definitions were unchanged.

Mean target-region RMSE was 0.260 for the 30-item bank, 0.423 for item-total selection, 0.434 for content-matched selection, and 0.432 for constrained assembly. The constrained form had lower target-region RMSE than item-total selection in 11 of 30 successful replications. The full bank is a length reference, not a burden-matched comparator.

The constrained form increased minimum target information relative to item-total selection in 30 of 30 successful runs. Both target-RMSE mean-difference intervals include zero in this run; they do not establish equivalence or a consistent average scoring advantage. Across the full population, constrained EAP RMSE was worse than item-total selection in every successful run (mean paired difference 0.074), showing why performance outside the target region also needs evaluation.

This study reports the distribution rather than selecting a favorable seed. Thirty attempts are an illustrative extension, not a precision-powered simulation study; the downloadable summary includes Monte Carlo standard errors and Wilson intervals for win fractions. The generating bank, population and model family remain fixed. These results do not establish clinical performance, generalize across item banks or generalize to all scoring-loss objectives; Section 7.3 tests one such heuristic. Parameter estimation varies across replications, but the original conditional analysis remains tied to its one frozen calibration.

Per-seed metrics CSV · Summary JSON · All attempts, warnings and selected items

# After reproducing the original fitted bank:
Rscript scripts/fitted-short-form-demo/repeat_demo.R
python scripts/fitted-short-form-demo/check_and_publish.py

7.3 Selecting for expected scoring error

We next changed the selection objective while keeping the generating bank, six-item length, two-items-per-domain blueprint, training-only enemy-pair screening, θ = 0.5–2 target region and N(0,1) EAP scoring prior fixed. Seeds 20261201–20261230 supply fresh training and held-out responses, with 1,600/800 people per attempt. They were chosen before this comparison and are separate from the earlier replication seeds.

The new criterion is expected EAP squared error, averaged over a target-truncated normal trait distribution under the fitted training model. It uses no generating item parameters, held-out responses or held-out true θ for selection. Held-out outcomes are read only after the selected candidate is saved.

This is a bounded heuristic search, not a globally optimal solution. Common random numbers generate 1,024 fitted-model design cases. Starting from maximin information and a feasible content-matched form, the search makes up to three best within-domain swaps per start. The five best approximate candidates plus feasible starts are evaluated by summing all 15,625 response patterns, with a 0.05 trapezoidal target-θ grid and the same 61-point EAP scoring grid. Exact response-pattern summation is not exact θ integration. Including maximin among finalists guarantees no worse fitted-model loss than that candidate, without guaranteeing better held-out error or global optimality.

30 of 30 comparisons completed successfully; 0 failed. Failure records are retained. Selection effort differs from MILP assembly and this is not a comparison of solver speed.

Mean target-region RMSE was 0.402 for scoring-loss selection, 0.433 for maximin information, 0.421 for item-total selection and 0.434 for content-matched selection. The paired mean target-RMSE interval favors scoring-loss selection in this fixed-bank experiment. Fitted-model target MSE decreased by 0.0243 on average relative to maximin; this model-based objective change is distinct from the held-out RMSE comparison.

The gain is specific to the target and scorer. Relative to maximin, scoring-loss selection changed minimum target information by -0.649 on average. Relative to item-total selection, its overall-population RMSE difference was +0.016 on average (positive is worse). Lower target-region scoring error therefore does not mean better performance for every trait level or every study purpose.

These results address alignment between one scoring rule and selection objective. The change includes both the error criterion and aggregation (worst-case information versus target-weighted mean squared error), as well as the search procedure; it does not isolate the contribution of each component. They do not identify a universally best short form. Search is heuristic, the target integral is numerical, calibration uncertainty is not modeled in selection, and only one generating item bank is used. No clinical validity, DIF, readability or external-population claim is added. The next generality check would vary generating banks and trait distributions.

Scoring-loss summary JSON · Per-seed metrics CSV · All selection/evaluation attempts · Updated complete reproducibility package

# After generating the original bank:
Rscript scripts/fitted-short-form-demo/prepare_scoring_loss.R
python scripts/fitted-short-form-demo/evaluate_scoring_loss.py
python scripts/fitted-short-form-demo/check_and_publish.py

7.4 Sensitivity to the heuristic search settings

We chose ten evenly spaced frozen calibrations (seeds 20261201, 20261204, …, 20261228) from the fresh-seed study before this check. No held-out outcomes were used to choose settings or evaluate this sensitivity analysis. The inputs are fitted training parameters, training-derived item subsets and enemy-pair metadata. These are ten calibrations from one generating bank, not ten different generating banks.

The nine settings include the published default; 256 and 4,096 design cases; one and six swap rounds; either original search start alone; another design draw; and two extra feasible random starts. Design sample sizes are nested around the original 1,024-case draw. Apart from the factor changed, settings retain the default 1,024 cases, three rounds and two starts. Every setting keeps the maximin and feasible content-matched anchors among finalists. Extra random starts are also finalists. Thus the start check concerns search paths, with shared anchor safeguards, rather than removing all influence of those anchor forms.

90 of 90 search attempts completed successfully; 0 failed. Paired summaries use 10 calibrations for which every setting completed. All default replays matched the published items and fitted-model loss. Failure details, if any, remain in the attempt log.

The published default’s fitted-model loss exceeded the best observed setting by 0.55% on average, with a worst-calibration gap of 1.72%. Changing an item set does not by itself imply a large scoring difference; conversely, similar average losses can hide an important individual calibration gap.

Six rounds and two extra starts each recovered the default item set in 10/10 and 10/10 calibrations, respectively. The 4,096-case design matched default items in 4/10 and attained the best observed loss in every calibration in this run. The 256-case design’s largest excess loss was 7.81%. These patterns indicate greater sensitivity to the simulation design than to the tested increases in rounds or starts; they do not establish that a larger design improves held-out performance.

This is a bounded sensitivity check. The best-observed reference is selected from the same tested settings and is descriptive, not an unbiased estimate of global regret. We neither promote the best setting using held-out outcomes nor replace the published default or its earlier performance claims. The check varies one factor at a time, omits interactions and examines one alternative design draw. It does not prove search convergence or robustness across generating banks, populations or instruments.

Stability summary JSON · Per-calibration settings CSV · All attempts and configurations

# Uses the existing fitted training banks; no recalibration is required:
python scripts/fitted-short-form-demo/search_stability.py
python scripts/fitted-short-form-demo/check_and_publish.py

Further work

The following extensions could test the limits of these findings. They have not been completed in this demonstration.

• Different item banks: vary threshold coverage and discrimination to test whether the findings extend beyond the current bank.
• Population and calibration conditions: vary trait distributions and calibration sample sizes, and account for item-parameter uncertainty.
• Model departures: examine multidimensionality, local dependence and missing responses.
• Broader search checks: test interactions among search settings, additional random draws and wider candidate searches.
• Other decision goals: compare target-region scoring error, overall accuracy and classification losses under explicit study priorities.
• Real-data validation: use appropriately permitted instruments and independent data to assess content coverage, subgroup performance and score comparability.