# Fitted short-form demonstration — simulated data

This package fits a graded response model to generated responses. It is not a real instrument, clinical validation, or evidence that IRT assembly always outperforms another method. Item IDs have no licensed item wording attached.

## Run from the extracted package root

Requirements: R with mirt, lpSolve and jsonlite; Python 3.10+ with numpy, scipy and matplotlib. Recorded versions are in session-info.txt and python-versions.txt. Install dependencies once:

```sh
Rscript -e 'install.packages(c("mirt", "lpSolve", "jsonlite"), repos="https://cloud.r-project.org")'
python -m pip install numpy "scipy>=1.9" matplotlib
```

Then run:

```sh
Rscript scripts/fitted-short-form-demo/run_demo.R
python scripts/fitted-short-form-demo/check_and_publish.py
```

The R script simulates 2,400 responses to 30 five-category items with seed 20261021; fits one unidimensional GRM to the first 1,600 cases; checks convergence, C2 fit and training residual Q3; selects six items; and scores the untouched 800 cases using fixed bank parameters and an N(0,1) EAP prior. The generating population is N(0,1), with item slopes uniform on [0.9,2.1], centers spanning [-1.7,1.7] within each blueprint group, and thresholds offset by [-0.9,-0.3,0.3,0.9].

The target information grid is 0.5–2.0 in 0.1 increments. The maximin form uses exactly two items per illustrative domain A/B/C and two-stage MILP (floor tolerance 1e-7). Domains are content labels on a single factor, not three latent dimensions. Centered Q3 above 0.20 defines descriptive enemy pairs; no pairs are flagged in this run, so this run does not demonstrate active exclusion. No DIF, readability or real content-validity assessment is performed. Zero reading grades disable that constraint.

Comparators are the top six training corrected item-total correlations and top two correlations within each domain. Neither is an exhaustive alpha-maximizing selection. All forms use the same calibrated parameters and EAP scoring rule, rather than a new calibration on the test sample. Full-bank scores are a length reference, not a burden-matched comparator.

Python independently reconstructs GRM information and EAP scores from the frozen R coefficients, re-solves the constrained assembly with SciPy, checks reported RMSE, and draws the figure. Python does not independently fit the GRM. It also computes a paired percentile bootstrap interval for the target-region RMSE difference using 2,000 resamples of held-out people. That interval excludes calibration and item-selection uncertainty.

## Interpretation

The constrained form improves the targeted minimum information but has worse EAP RMSE in this run, including among the 234 held-out people whose true theta is in the target region. This is a useful distinction between an optimization criterion and a scoring-validation outcome. Do not infer a general ranking or a causal explanation from one simulation. Prior shrinkage, bias, item thresholds, and the chosen objective may matter; this experiment does not isolate their contributions.

The arbitrary theta >= 1 classification is against simulated true theta, not a clinical diagnosis. The original example supplies no power estimate, national norms, score crosswalk, parameter uncertainty, instrument permissions review or clinical equivalence. A repeated-response extension is documented below. The known true theta is available only because responses are simulated. The fitted information and 1/sqrt(information) curves are model-based, not observed holdout standard errors or EAP posterior SDs.

Files under whitepapers/fitted-demo include responses and split/true theta, generating and fitted parameters, the saved training model, item information, Q3 residual correlations, selection and fit results, holdout EAP scores/posterior SDs, plotted comparisons, validation checks and environment versions. CHECKSUMS.json hashes all packaged source and result files other than itself and the ZIP. Do not delete or alter the saved results if you want to compare your rerun with the recorded run. Different package versions may change numerical results.

Primary implementation documentation: https://philchalmers.github.io/mirt/reference/mirt.html ; https://philchalmers.github.io/mirt/reference/fscores.html ; https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.milp.html


## Conditional bias–variance and prior sensitivity

The Python check script also runs bias_variance.py. It enumerates all 5^6 = 15,625 response patterns per fixed short form. Conditional pattern probabilities use generating parameters, while scoring uses fitted coefficients. The scoring grid has 61 points on [-6,6]. Normal-prior EAP reproduces the published scoring rule. Flat-grid posterior means use equal prior weights on the same grid as a diagnostic perturbation, not ML scoring or a recommended replacement.

Conditional mean score, bias, variance and MSE are evaluated at theta -3 to 3 in 0.05 increments. Each theta's response-pattern probabilities sum to 1; numerical checks enforce MSE = bias squared + variance. Target integrals use trapezoidal quadrature with normal-density weights conditional on theta in [0.5,2]. These are conditional on one frozen calibration and selection and exclude parameter uncertainty. Integrated expected RMSE differs from the finite holdout RMSE. The full 30-item bank is excluded from enumeration because 5^30 patterns are intractable here.

conditional-bias-variance.csv contains pointwise results for both scoring priors. bias-variance.json contains target aggregates. holdout-error-bins.csv includes empirical half-unit bins and counts; its error variance is not pointwise conditional score variance. No new form is selected during this diagnostic. Run it alone with `python scripts/fitted-short-form-demo/bias_variance.py`, or regenerate all validation, HTML and package outputs using check_and_publish.py. This does not perform repeated-seed calibration or test a new assembly objective.


## Response-sample replications

To extend the original analysis, run `Rscript scripts/fitted-short-form-demo/repeat_demo.R` after run_demo.R, then run check_and_publish.py. The generating slopes are frozen from the original parameters.csv; thresholds, population, 1,600/800 split sizes, 6-item selection rules and EAP scoring prior remain fixed. Seeds 20261101–20261130 were specified before running the extension. These are independent response samples within one generating bank, not draws of different item banks. Each replicate includes a new GRM calibration and new training-only Q3 screening and item selection. No failed seed is replaced; results are checkpointed after each attempt. All fit/assembly/scoring failures and warnings are retained in replications.json. Complete-success numerical summaries must be interpreted alongside the failure count.

The replication script writes intermediate per-seed data/model outputs to R temporary directories; the distributable retains the seed, design, full result metrics, fit diagnostics, warnings and selected items for every attempt, from which the response data and fits can be regenerated. Original frozen-bank outputs are not overwritten. Elapsed seconds are machine-dependent; CSV/numerical results should be reproducible with recorded package versions, but log timings will differ.

summarize_replications.py requires all 30 attempts and produces per-seed metrics, paired differences, means, standard deviations, Monte Carlo standard errors, t intervals for mean differences, Wilson intervals for win fractions and a plot. Numerical comparisons use successful attempts only. Differences are constrained minus comparator. Intervals quantify Monte Carlo uncertainty under this fixed-bank data-generating design; they are not clinical confidence intervals. Thirty seeds were chosen as a bounded illustrative run, without a simulation precision calculation. This extension does not evaluate a scoring-loss selection objective or generalize across banks.


## Training-only scoring-loss selection on fresh seeds

Run these from the package root after the original run_demo.R:

```sh
Rscript scripts/fitted-short-form-demo/prepare_scoring_loss.R
python scripts/fitted-short-form-demo/evaluate_scoring_loss.py
python scripts/fitted-short-form-demo/check_and_publish.py
```

The fresh seeds 20261201–20261230 use the same generating bank and 1,600 training/800 test split sizes. They differ from the earlier 30 replication seeds. prepare_scoring_loss.R recalibrates and performs the original training-only selection for each seed, retaining fitted parameters, baseline results and untouched held-out responses/scores under scoring-loss/<seed>/. Responses are simulated, not clinical records. Intermediate original-run files remain in R temporary directories; the seed and frozen bank reproduce them.

scoring_loss.py selects using fitted slopes/thresholds, content labels, reviewed enemy pairs and training-derived starting subsets only. The selection function never receives the held-out data or generating parameters. Its criterion is expected EAP squared error integrated over N(0,1) conditional on theta in [0.5,2], while scoring retains an N(0,1) prior on a 61-point [-6,6] grid.

A common 1,024-case fitted-model simulation (seed = replication seed + 1,000,000) ranks feasible subsets. Starting from the maximin and a feasible content-matched form, up to three best within-domain swaps are made per start. The top five visited candidates plus starts are scored using all 5^6 response patterns and 31 target-theta points (0.05 grid, trapezoidal normal weights). Final selection minimizes this fitted-model risk among finalists. This is a bounded heuristic, not an exhaustive or globally optimal loss-based solution. The included maximin finalist guarantees no worse fitted-model loss, but does not guarantee improved test error. All methods use six items and the new method respects the same two-per-domain and enemy-pair constraints as maximin. The unstratified item-total comparator does not impose the domain blueprint.

evaluate_scoring_loss.py saves selection.json before reading held-out responses. It then scores test cases, checks maximin EAP scores and information against R, records per-seed metrics, and summarizes paired differences. Failures are retained without replacement. summary.json fingerprints input files and selection code; packaging refuses stale inputs. The per-seed scoring-loss-test-scores.csv files support independent RMSE checks. Target-model loss and empirical held-out RMSE are distinct quantities.

The experiment holds one generating bank fixed. Thirty fresh response samples do not establish generality over banks, trait distributions or real instruments. Selection does not account for calibration uncertainty. No claim of clinical validity, DIF, readability or global optimality is added. The existing bias-variance and earlier 30-seed outputs retain their own original designs and seeds.

Run `python scripts/fitted-short-form-demo/verify_loss_selection.py` for a deterministic selector replay and an independent 32,768-case fitted-model simulation check of the enumerated target loss. This check uses no held-out outcomes.

The new objective also changes aggregation: worst-case information is replaced by target-weighted average squared error. Combined with the different search procedure, this means the comparison tests an alternative selection procedure, not an isolated causal effect of swapping the information criterion alone.


## Article and companion report

The main methodological guide is whitepapers/constrained-automated-test-assembly-irt-milp.html. Full solver implementations and simulation studies are collected in whitepapers/short-form-selection-simulation-report.html. Existing section anchors on the main guide retain concise summaries and link to matching companion sections. render_consolidated.py rebuilds both pages from the calculated results during check_and_publish.py; the extracted package includes both pages and that renderer. Shared website styling is served by the live site; the ZIP is an analysis reproduction package, not a complete offline website.


## Bounded search-stability check

Run `python scripts/fitted-short-form-demo/search_stability.py`, then check_and_publish.py. The check uses ten evenly spaced calibrations from the fresh-seed study: 20261201, 20261204, ..., 20261228. This cohort and the nine settings were fixed before assessing their stability results. It uses fitted training coefficients and training-derived anchor subsets/enemy metadata; no held-out outcome is used for tuning or evaluation.

Settings: published default (1,024 design cases, three swap rounds, two starts); 256 and 4,096 design cases; one and six rounds; either original search start alone; another design draw (offset 17); and two extra feasible starts. The first 1,024 cases exactly reproduce the published design; the smaller design is a prefix and larger design appends cases. Each setting retains the original feasible anchor subsets among exact-loss finalists. The extra-start setting also retains its extra starts as finalists. Thus this is search-path sensitivity under common finalist safeguards, not removal of all anchor influence. Tests are one factor at a time; interactions and additional alternative random draws are not examined.

All 90 attempts are retained. Paired summaries use calibrations with every setting completed successfully. The default must reproduce the published selected items and fitted-model loss. Reported loss is the same response-pattern summation with target-theta quadrature as before. Shared items and identical subsets are compared with the published default for each calibration. Excess loss is relative to the best observed among these nine settings within that calibration, not a proven optimum or unbiased estimate of global regret. A low mean gap does not rule out sensitivity in an individual calibration. Neither the original default nor its held-out performance claims are replaced based on this check.

search_stability.py writes attempts.json, metrics.csv and summary.json under scoring-loss/stability. Input/code hashes protect against stale results; stability_report.py renders the plot and report from those files. The study holds the generating bank fixed and does not establish robustness across banks, populations or real instruments.