98

Research

98-CI IS UNDER DEVELOPMENT AND VALIDATION.

98 is an engineering build of a cognitive-assessment product. It is not, yet, a validated psychometric instrument. The document below is the actual plan this codebase enforces structurally — a validated credential literally cannot be issued until the steps below are done, tracked in code by CredentialPolicyService.

98 Validation Plan

Status: NOT YET MEASURED for every item below unless explicitly marked otherwise. This document describes what evidence is required before VALIDATED_MODE may be enabled, or before any percentile, IQ-equivalent, or "top X%" claim may be shown to a candidate or employer. It is a plan, not a report of results — no results exist yet.

The current system defaults to RESEARCH_MODE and the CredentialPolicyService (packages/credentials) technically enforces that a VALIDATED_MODE credential cannot be issued without an AssessmentRelease row carrying all of the approvals below. As of this build, no `AssessmentRelease` has been created or approved — validated claims are structurally blocked, not just policy-blocked.

Construct definition — NOT YET MEASURED

The Core Cognitive Profile (Fluid Reasoning, Quantitative Reasoning, Visual-Spatial Reasoning, Working Memory, Processing Efficiency) is modeled on well-established constructs from the cognitive ability literature (broadly consistent with CHC theory's major strata), but 98's own item pool has not been reviewed by an external psychometrician or mapped to that literature formally. The Frontier Profile (Adaptive Reasoning, Learning Transfer, Probabilistic Reasoning, Evidence Updating, Systems Reasoning, Cognitive Flexibility, Metacognitive Calibration) is explicitly experimental and has no established construct-validity literature to lean on — it requires original construct-definition work, likely academic collaboration, before it can be scored at all, let alone combined with the Core composite. packages/db's schema already keeps these as separate composite score rows (CompositeScore.code) so they can never be silently merged.

Content validity — NOT YET MEASURED

Current item families (packages/assessment/src/item-generation/generators/*) are original, hand-authored templates covering numeric sequences, visual-spatial rotation, transitive logical relations, and digit-span memory — chosen for engineering demonstrability, not for content-validity coverage of the target constructs. A real content-validity study requires: a blueprint reviewed by subject-matter experts, item-by-item construct mapping, and a bias/sensitivity review panel.

Sampling / normative design — NOT YET MEASURED

No normative sample exists. NormSet/NormSetVersion (schema) require, at minimum, before any norm is marked non-demo: documented sample size, sampling methodology (must not be a convenience sample of early adopters if percentile claims are to mean anything), weighting methodology, geographic and age scope, and explicit sampling limitations. NormSetVersion.maxSupportedPercentile must be set conservatively relative to sample size — a rule of thumb used elsewhere in this codebase (services/psychometrics-lab/src/psychometrics_lab/norm_generation.py) is that a normative sample cannot statistically support a percentile finer than roughly 100 - 100/n for a sample of size n without extrapolation beyond the data; e.g. a 200-person sample should not claim above roughly the 99.5th percentile without a documented extrapolation model (fitted tail distribution) that has itself been validated.

Reliability — NOT YET MEASURED

services/psychometrics-lab has tested, working implementations of Cronbach's alpha, split-half (Spearman-Brown corrected), and test-retest correlation — the math is ready. No response data exists yet to run it on. Target (industry convention, not yet met or measured): internal consistency α ≥ 0.80 for any composite used in a pass/fail decision.

Test-retest / practice effects — NOT YET MEASURED

Given the product allows repeated attempts (Ascension, 90-day cooldown), practice effects are a first-order concern for construct validity over repeated administrations. No test-retest study has been run. Procedurally generated items (fresh seed per assignment) reduce but do not eliminate practice effects from familiarity with item _format_.

Alternate forms / equating — NOT YET MEASURED

services/psychometrics_lab/src/psychometrics_lab/form_equating.py implements linear/mean/Tucker equating and is tested against synthetic data with known ground truth. No real alternate forms exist yet to equate.

Convergent / discriminant validity — NOT YET MEASURED

Requires correlating 98 scores against established, validated instruments (with appropriate licensing and IRB/ethics review for any human-subjects research) and against constructs 98 should NOT correlate strongly with. Not started.

Criterion / predictive validity — NOT YET MEASURED

Any future marketing claim connecting 98 performance to real-world outcomes (job performance, income, achievement) requires a longitudinal predictive-validity study with a defined criterion measure, a real sample, and appropriate statistical controls for confounds. This is explicitly the category of claim section 62's copy constraints exist to prevent 98 from making prematurely ("Do not use unsupported factual claims about high-IQ people being guaranteed to become rich").

Measurement invariance / DIF — NOT YET MEASURED

services/psychometrics-lab/src/psychometrics_lab/dif_analysis.py implements Mantel-Haenszel and logistic-regression DIF detection, tested against synthetic data with known injected DIF. No real group-comparison data exists yet. This must run before any item is trusted across demographic groups, and before any claim of fairness is made publicly (see the /standards page requirement to disclose fairness work honestly).

Accessibility — PARTIALLY ADDRESSED, NOT YET MEASURED

AccommodationPolicy (schema) exists as a structural placeholder (extended time multiplier + a JSON config bag) but no accommodation has been validated to leave the underlying construct unchanged (per section 58's own caution: "an accommodation must not assume every accommodation leaves every construct unchanged"). No accessibility review has been performed on assessment content specifically (general application accessibility — contrast, keyboard nav, screen readers — is a separate, more tractable concern also NOT YET independently audited in this pass).

Item exposure / security maintenance — PARTIALLY ADDRESSED

ItemExposureState tracks assignment/answer counts per item; ItemCompromiseReport exists for flagging leaked content; Item.status supports SUSPENDED/COMPROMISED. No exposure-rate policy threshold has been chosen or enforced yet (packages/assessment's ExposureRateController exists and is unit-tested but is not yet wired into the live item-selection path in apps/api's AssessmentService, which currently just walks the fixed blueprint order — see BUILD_STATUS.md).

Calibration maintenance / norm refresh policy — NOT YET MEASURED

No policy has been written for how often norms are refreshed or recalibrated. This is a governance decision requiring a real Cognitive Standards Board (section 49's GovernanceMember/GovernanceApproval schema exists to record this once one is convened) — not something engineering can decide unilaterally.

What would unblock `VALIDATED_MODE`

  1. Convene real governance (or at minimum a named, qualified psychometric consultant) — populate GovernanceMember rows honestly.
  2. Run a real pilot with a real (even if modest) sample through PILOT-status items.
  3. Run the services/psychometrics-lab analyses against that real data.
  4. Create a NormSet/NormSetVersion with honest sample metadata and a conservative maxSupportedPercentile.
  5. Create an AssessmentRelease, get every approval flag set truthfully, set governanceApprovedAt.
  6. Only then does CredentialPolicyService permit a VALIDATED_MODE credential — and even then, only up to maxSupportedPercentile.