Hypothesis
The hypothesis decomposes into three testable sub-claims:
logic_verdict = Inconclusive should have
nli_label = Neutral and
neutral_classification = Neutral.
logic_verdict ∈ {True, False} should have
nli_label ∈ {Entailment, Contradiction} and
neutral_classification = Not Neutral.
Theoretical basis
The NLI label Neutral is defined as: the hypothesis is possible but not guaranteed under the premise set — neither entailed nor contradicted. The logical verdict Inconclusive is defined as: the KB is satisfiable under two interpretations that yield opposite truth values for the query, so no single truth value can be assigned.
Both conditions are triggered by the same KB structural pattern used in this dataset: a polysemous noun (bat, bank, wave…) with two senses defined in the KB, mutual-exclusion axioms that would disambiguate if the individual were typed, and a contextual observation predicate that is compatible with both senses. The prediction is that both formalisms will register this ambiguity as their respective "undecided" label.
Experimental Setup
Same input, two reasoning frameworks
Closed-World Assumption
claude-haiku
Monotonicity calculus
claude-haiku
Exception classification
Verdict
Dataset
| Property | Value |
|---|---|
| Source | results/eval_output.jsonl |
| KB entries | 8 polysemous nouns (bat, head, nail, paper, bank, mouse, pitcher, wave) |
| Query types per KB | 4 (inconclusive · conclusive · neutral_query · not_neutral) |
| Total items | 32 |
| Logic backbone | claude-haiku (agent_logic) |
| NLI backbone | claude-haiku (agent_NLI) |
| Synthesizer | claude-sonnet (agent_eval) |
Expected correspondence table
| Logic verdict | Expected NLI label | Expected Neutral class | Query types that target this |
|---|---|---|---|
| Inconclusive | Neutral | Neutral | inconclusive · neutral_query |
| True | Entailment | Not Neutral | conclusive · not_neutral |
| False | Contradiction | Not Neutral | conclusive · not_neutral |
| Paradox | flag for manual review — not expected in this dataset | ||
Verification Agent — agent_hypothesis_verifier
The agent reads the pre-computed results/eval_output.jsonl
(no re-running logic or NLI agents) and runs a 6-step analysis:
Reads papers, concepts, experiments, and reflection files to ground fallacy and failure-mode analysis in the documented KB design rationale.
Builds an internal table from eval_output.jsonl with logic_verdict, nli_label, neutral_classification, query_type, and dataset_quality_flag.
Precision, Recall, F1 for H₁(a) and H₁(b) in aggregate; then stratified by query type (4 groups × 8 items) and by KB key (8 keys × 4 items).
Every item where the expected mapping fails is classified into a documented failure mode:
FM1 Query-predicate disambiguation blindspot (logic agent) FM2 Universal KB scoping (logic returns True via undiscriminating universal) FM3 NLI exclusion-chain composition failure FM4 Dataset annotation error FM5 Novel failure mode
For the clearest GOLD matches: do both agents cite the same KB sentences as the ambiguity root cause? For the clearest exceptions: does the failure disprove the structural analogy or merely reveal an agent implementation gap?
Assigns SUPPORTED / PARTIALLY_SUPPORTED / REFUTED based on metric thresholds (see below), with a one-paragraph justification citing specific values.
What the result means for using NLI Neutral as a proxy for logical inconclusiveness, limitations of the test, and recommended next steps (larger dataset, cross-domain KBs, multi-model replication).
Verdict thresholds
precision_inc ≥ 0.90
recall_inc ≥ 0.80
All exceptions = FM1–FM3
OR precision ≥ 0.80 but
recall < 0.80
OR some FM4 exceptions
OR structural mismatch
on GOLD items
OR systematic FM5
How to Run
From the claude-code-agentic-simulation/ directory:
@agent_hypothesis_verifier
The agent reads results/eval_output.jsonl (already generated) and
writes its full report to results/hypothesis_test_report.md.
No additional inputs required — all pre-computed eval results are used as-is.
Output
The agent writes results/hypothesis_test_report.md containing:
- Aggregate and stratified precision / recall / F1 tables
- Per-item exception table with failure-mode classification
- Structural analogy assessment (qualitative)
- Formal SUPPORTED / PARTIALLY_SUPPORTED / REFUTED verdict
- Implications and recommended next steps
Prior Evidence from Evaluation
From the Evaluation Report, the pipeline achieved 71.9% MCQ accuracy (23/32). The correspondence between Inconclusive and Neutral is already visible in the per-key results:
| Query type | Total | Expected: Inc↔Neutral | Observed GOLD rate | Notes |
|---|---|---|---|---|
| inconclusive | 8 | H₁(a) test | 8/8 GOLD | All agents agree Inconclusive/Neutral across all 8 keys |
| neutral_query | 8 | H₁(a) meta-NLI | 7/8 GOLD | paper/neu both agents return Not Neutral — KB scoping flaw |
| conclusive | 8 | H₁(b) test | 2/8 GOLD | bat/con (NLI applies chain), paper/con (both agree); others fail FM1/FM2 |
| not_neutral | 8 | H₁(b) test | 3/8 GOLD | pitcher/not, paper/not, wave/not succeed; others FM1/FM2 |
The inconclusive/neutral_query rows show near-perfect H₁(a) support. The conclusive/not_neutral rows show partial H₁(b) support — failures are systematic (FM1/FM2), not random. The agent_hypothesis_verifier will compute the precise metrics and issue the formal verdict.