Stage 1 heterogeneous synthetic evidence

This article renders the completed, frozen stage1-heterogeneous-v2 synthetic-evidence artifact. It does not rerun the benchmark, tune a candidate, or make a biological claim. The complete artifact is hash-verified before its contents are read.

Result

The calibration rule selected C2_block_scaled_svd, but the frozen holdout acceptance rule failed. Therefore no heterogeneous Stage 1 baseline is accepted and Issue #24 remains blocked. C1 cannot be substituted after this result: it was not the calibration-selected candidate, and the v2 holdout is consumed.

knitr::kable(data.frame(
  field = c("protocol", "protocol digest", "generator digest", "artifact verification",
            "completed tasks", "result rows", "source commit", "calibration selection",
            "holdout decision", "holdout thresholds passed"),
  value = c(
    evidence$manifest$protocol_id,
    evidence$selection$protocol_digest,
    evidence$selection$generator_digest,
    "complete payload and content-addressed name verified",
    completed_tasks,
    nrow(evidence$results),
    evidence$environment$commit,
    evidence$selection$selected_candidate,
    evidence$holdout$decision,
    evidence$holdout$thresholds_passed
  ),
  stringsAsFactors = FALSE
), col.names = c("Field", "Value"))

Field	Value
protocol	stage1-heterogeneous-v2
protocol digest	dcae519df4a6975e9715bdaa69a5649fa00f1b35ffb0feb71e3fe59c34cb9226
generator digest	47030b60f48f7d859f92fc05ce33f95fe6e04fa164eddbecc8009c64eab097dc
artifact verification	complete payload and content-addressed name verified
completed tasks	40960
result rows	81920
source commit	6f1f0614b2c4f0d539baa69a20df1ef43705ade6
calibration selection	C2_block_scaled_svd
holdout decision	failed
holdout thresholds passed	FALSE

Execution inventory

The verified table contains all 40,960 deterministic stratum-by-seed tasks and no failed contract-gate rows. Expected missing-ID controls remain successful when they emit their required typed failure.

knitr::kable(data.frame(
  field = c("completed deterministic tasks", "candidate-result rows", "failed contract-gate rows"),
  value = c(completed_tasks, nrow(evidence$results), sum(!evidence$results$gate_passed)),
  stringsAsFactors = FALSE
), col.names = c("Field", "Value"))

Field	Value
completed deterministic tasks	40960
candidate-result rows	81920
failed contract-gate rows	0

Holdout threshold readout

The protocol requires every exact-ID stratum with shared signal 24 and noise SD 1 to meet both thresholds. This table reports the worst median among those frozen strata; a value above its threshold is sufficient to reject the candidate.

summary <- evidence$holdout$summary
required <- summary[
  summary$projection_case == "exact_ids" &
    summary$shared_signal == 24 &
    summary$noise_sd == 1 &
    summary$metric %in% c("shared_recovery_error", "projection_error"),
  , drop = FALSE
]
thresholds <- c(shared_recovery_error = 0.25, projection_error = 0.30)
threshold_readout <- do.call(rbind, lapply(names(thresholds), function(metric) {
  values <- required$estimate[required$metric == metric]
  data.frame(
    metric = metric,
    exact_id_strata = length(unique(required$stratum_digest[required$metric == metric])),
    worst_median = max(values),
    threshold = thresholds[[metric]],
    all_strata_pass = all(values <= thresholds[[metric]]),
    stringsAsFactors = FALSE
  )
}))
knitr::kable(transform(threshold_readout, worst_median = round(worst_median, 3)),
  col.names = c("Metric", "Exact-ID strata", "Worst median", "Threshold", "All pass"))

Metric	Exact-ID strata	Worst median	Threshold	All pass
shared_recovery_error	128	0.390	0.25	FALSE
projection_error	128	0.343	0.30	FALSE

Holdout figures

sre_dat <- exact[exact$metric == "shared_recovery_error", ]
.draw_holdout_panel(sre_dat,
  metric_label = "Holdout shared-subspace recovery error (C2)",
  threshold = 0.25,
  ylab_text = "Shared recovery error (lower is better)"
)

Figure 1. C2 holdout shared-subspace recovery error across all exact-ID strata. Each point is the median shared-recovery error (lower is better) for one canonical stratum over 20 holdout seeds, with 95% bootstrap confidence intervals. Colour encodes the signal-to-noise regime: red = high signal (24) / low noise (1), the threshold-decisive conditions; orange = high signal / high noise; blue = low signal (12) / low noise; grey = low signal / high noise. Triangle symbols indicate 3-layer (K = 3) strata; circles indicate 2-layer (K = 2) strata. The dashed horizontal line is the frozen acceptance threshold (0.25). Points above the line are failing strata. The majority of strata fail even under high signal and low noise, indicating that C2 does not reliably recover the planted shared subspace in this heterogeneous-feature setting.

pe_dat <- exact[exact$metric == "projection_error", ]
.draw_holdout_panel(pe_dat,
  metric_label = "Holdout projection error (C2)",
  threshold = 0.30,
  ylab_text = "Projection error (lower is better)"
)

Figure 2. C2 holdout projection error across all exact-ID strata. Each point is the median projection error (lower is better) for one canonical stratum over 20 holdout seeds, with 95% bootstrap confidence intervals. Colour and symbol encoding as in Figure 1. The dashed horizontal line is the frozen acceptance threshold (0.30). The low band of points (projection errors 0.22-0.35) corresponds to high-signal strata where the method partially captures the shared direction; the upper band (0.6-1.0) indicates near-complete failure. Even in the best-case high-signal, low-noise strata, a substantial fraction of replicates exceeds the threshold.

Provenance and next step

The checked artifact coverage comprises all 40,960 synthetic tasks and 81,920 candidate-result rows. Full artifacts are published only once every task checkpoint is complete, so this result has no failed tasks. It also records the seed manifest, protocol and generator digests, the calibration decision, and the holdout report. Its source commit is 6f1f0614b2c4f0d539baa69a20df1ef43705ade6.

The negative confirmation result is retained as evidence. Any future attempt to improve a candidate requires a new ADR and a newly frozen protocol; it must not reuse or tune against this holdout.