Validation Test Suite
Every NAVAL-SEM release is gated by a full pytest + Playwright suite run against published benchmark datasets. All 168 tests must pass before a build ships. Results are compared to peer-reviewed anchor values — if the numbers don't match the literature, the release is blocked. This page documents all test cases including v1.0 fsQCA/reporting tests and the v1.1 literature-anchored parity suite (S1-S7): tests that assert model fit indices, FIML estimates, and plain-English annotations against numbers traceable to a specific published source (Rosseel 2012, Bollen 1989, Arbuckle 1996, Hu & Bentler 1999) rather than just checking that a run doesn't crash. Sub-features without an available original reference dataset are left as explicit skips rather than backed by an invented benchmark — see the "Skipped" tag in the registry below.
Release gate · pytest + Playwright · 168/168 required to pass
—
Total tests (pytest + Playwright)
—
Playwright UI tests
20
Benchmark datasets
—
Categories
21
API endpoints
Test Case Registry
| TC-ID | Category | Description | Dataset | Timeout (s) | Tags |
|---|
Published Benchmark Anchors
| TC-ID | Dataset | Metric | Published Value | Source |
|---|---|---|---|---|
| TC-31 | HS1939 | CB-SEM CFI | ≈ 0.931 | semopy ML · 3-factor CFA |
| TC-31 | HS1939 | CB-SEM RMSEA | ≈ 0.092 | Holzinger & Swineford 1939 |
| TC-41 | CorporateRep | Avg loading | ≈ 0.80 | Hair et al. 2011/2013 |
| TC-42 | CorporateRep | Max HTMT | ≈ 0.86 | Rönkkö & Cho 2022 |
| TC-52 | PoliticalDem | CB-SEM CFI | ≥ 0.997 | Bollen 1989 |
| TC-106 | FsQCA_Wagemann | Consistency (complex) | ≥ 0.80 | Wagemann & Schneider 2010 |
| TC-107 | FsQCA_Wagemann | Coverage (parsimonious) | ≥ 0.60 | Ragin 2008 |
| PAR-01 | HS1939 | CB-SEM TLI | ≈ 0.896 | Rosseel 2012 · lavaan tutorial |
| PAR-01 | HS1939 | CB-SEM SRMR | ≈ 0.065 | Rosseel 2012 · lavaan tutorial |
| PAR-02 | PoliticalDem | CB-SEM CFI | ≈ 0.995 | Bollen 1989 · lavaan tutorial |
| PAR-02 | PoliticalDem | CB-SEM RMSEA | ≈ 0.035 | Bollen 1989 · lavaan tutorial |
| PAR-04 | FIML_GroundTruth | FIML loading recovery (n=3000, 15% MCAR) | ±0.12 of true λ | Arbuckle 1996 · Little & Rubin 2019 |
| PAR-09 | Mardia_GroundTruth | Classification (n=500, normal vs. χ²-skewed) | correct verdict both ways | Mardia 1970 (method) — ground-truth simulation, not a textbook's reported statistic |
| PAR-10 | Bayes_GroundTruth | Posterior mean recovery (n=400, 1-factor, 4 chains) | ±0.2 of true λ, R-hat ≤ 1.01 | Posterior consistency (general theory) — no specific published table available |
| PAR-11 | LCA_TwoClass | Class mean recovery (n=300, 2 classes, 50/50) | low mean <4, high mean >6 | Ground-truth simulation — no published class solution available |
| PAR-12 | MGA_PlantedEffect | Path-difference detection (β=0.6 vs 0.15) / null case (β=0.4 vs 0.4) | significant / not significant | Henseler et al. 2009 (method); ground-truth simulation, not the paper's own dataset |
| PAR-13 | CTA_GroundTruth | Tetrad verdict (n=400, single-factor vs. 2-factor-forced) | supports reflective / consider formative | Bollen & Ting 2000 (method) — ground-truth simulation, not the paper's own worked example |
| PAR-14 | Predict_GroundTruth | Verdict (n=300, real F→Y signal vs. pure noise) | ≠ lacks relevance / = lacks relevance | Shmueli et al. 2019 (method); ground-truth simulation, not the paper's benchmark table |
Benchmark Datasets
CorporateRep
Hair et al. (2011/2013) corporate reputation dataset used in PLS-SEM primer examples.
Used by: TC-02, TC-04, TC-06, TC-14–17, TC-22, TC-23, TC-25, TC-27, TC-29, TC-30, TC-41–45, TC-54, TC-58–60, TC-62–65, TC-68
HS1939
Holzinger & Swineford (1939) mental ability test scores. CB-SEM CFA benchmark. PAR-01 pulls the exact original data via semopy's bundled example (skips rather than substitutes a synthetic surrogate if unavailable).
Used by: TC-03, TC-07, TC-08, TC-31–36, PAR-01
PoliticalDem
Bollen (1989) political democracy dataset. CB-SEM CFI/RMSEA anchor. PAR-02/05/07 pull the exact original data via semopy's bundled example (skip rather than substitute a synthetic surrogate if unavailable).
Used by: TC-52, TC-53, TC-56, PAR-02, PAR-05, PAR-07
ECSI_Satisfy
European Customer Satisfaction Index (Tenenhaus 2005). PLS-SEM path coefficient benchmark.
Used by: TC-26, TC-28, TC-46–49, TC-95–97
ECSI_MGA
Subgroup split of ECSI for multi-group analysis (MGA) testing.
Used by: TC-20, TC-21, TC-98–102
EStress
Hayes (2018) work-related stress dataset. Moderation and conditional process benchmark.
Used by: TC-24, TC-40, TC-70–75
Disaster
Hayes (2018) disaster frame mediation dataset.
Used by: TC-37, TC-38
PMI
Hayes (2018) prosocial motivation and immobilisation dataset (a matched synthetic version is used for PAR-06's annotation-format check).
Used by: TC-39, PAR-06
FIML_GroundTruth
Simulated from known population loadings (λ = 1.0/.8/.7/.6) with 15% MCAR injected on two indicators, so FIML recovery is checked against ground truth rather than a noisy finite-sample comparison fit.
Used by: PAR-04
Mardia_GroundTruth
One draw from a normal common factor, one from a χ²(df=2) (heavily right-skewed) common factor. No textbook dataset with a published Mardia statistic was available, so PAR-09 checks that both are classified correctly instead of matching one reported number.
Used by: PAR-09
Bayes_GroundTruth
Simulated from known population loadings (λ = 1.0/.75/.65/.55). No verifiable published Bayesian posterior-mean reference was available for A18, so this dataset lets PAR-10 check posterior consistency (R-hat + recovery of the true loadings) instead of an invented benchmark number.
Used by: PAR-10
LCA_TwoClass
Two well-separated simulated classes (indicator means ~2 vs ~8, equal size). No published class-proportion/item-response dataset was available for engine_lca.py, so PAR-11 checks recovery of this known structure instead.
Used by: PAR-11
MGA_PlantedEffect
Two simulated groups sharing one measurement model; the F→Y structural path is deliberately set to 0.6 in group A vs. 0.15 in group B (plus a separate 0.4/0.4 null-case draw). Henseler et al.'s (2009) own demonstration dataset wasn't available to reproduce exactly, so PAR-12 checks MGA's power/Type-I behaviour on a known planted effect instead.
Used by: PAR-12
CTA_GroundTruth
One draw from a genuine single common factor (tetrads should vanish), one from two independent factors forced into a single reflective block (tetrads should not vanish). Bollen & Ting's (2000) own worked example wasn't available, so PAR-13 checks this known property instead.
Used by: PAR-13
Predict_GroundTruth
Same measurement blocks in both draws; one has a genuine F→Y structural relationship (β=0.75), the other has Y generated as pure noise independent of F. Shmueli et al.'s (2019) exact benchmark table wasn't available, so PAR-14 checks the decision rule distinguishes these instead.
Used by: PAR-14
HS1939_Missing
HS1939 with 8% MCAR introduced for missing data handling tests.
Used by: TC-07, TC-08, TC-35, TC-55
CMB_Sim
Simulated dataset with injected common method bias (inflated R²).
Used by: TC-19, TC-51, TC-69
Heywood
Designed to trigger Heywood cases (negative residual variance) in CB-SEM.
Used by: TC-50, TC-57
CVI_Expert
Simulated expert rating matrix for CVI testing. Polit & Beck (2006) thresholds applied.
Used by: TC-80–85, TC-84, TC-127
FsQCA_Wagemann
Wagemann & Schneider (2010) replication dataset for fsQCA Boolean minimisation benchmarking.
Used by: TC-105–114, TC-123, TC-125, TC-126
API Endpoints — Quick Reference
| Method | Route | Description | TCs |
|---|---|---|---|
| GET | /health | Server health check | TC-01 |
| GET | /check-updates | Version update check | TC-61 |
| GET | /logs/{run_id} | Retrieve run log by ID | TC-30, TC-59 |
| POST | /run | Run PLS-SEM / CB-SEM / WLS model | TC-02–04, TC-06–08 |
| POST | /bootstrap | Bootstrap CIs and p-values | TC-05, TC-13, TC-36, TC-43 |
| POST | /htmt | HTMT discriminant validity matrix | TC-62–63 |
| POST | /indirect | Indirect effects (mediation) | TC-37–39 |
| POST | /mga | Multi-Group Analysis | TC-20–21 |
| POST | /hoc | Higher-Order Constructs | TC-22–23, TC-68 |
| POST | /moderation | Moderation analysis | TC-24, TC-40, TC-71 |
| POST | /mod-mediation | Moderated Mediation (a-path / b-path) | TC-72–75 |
| POST | /ipma | Importance–Performance Map Analysis | TC-25, TC-45 |
| POST | /nca | Necessary Condition Analysis | TC-26, TC-49 |
| POST | /nca-esse | NCA Effect Size Sensitivity Extension | TC-95–97 |
| POST | /cvi | Content Validity Index | TC-80–85 |
| POST | /efa | Exploratory Factor Analysis | TC-86–90 |
| POST | /nomological | Nomological validity (bivariate correlations) | TC-91–93 |
| POST | /invariance | Measurement Invariance (MICOM extended) | TC-98–102 |
| POST | /fsqcav1.0 | fuzzy-set QCA — calibration, necessity, truth table, QMC | TC-105–114 |
| POST | /reportv1.0 | APA 7th edition Word (.docx) export | TC-115–120 |
| POST | /export | Export R / Python / lavaan code | TC-15–17 |