Logical Inconclusive Reasoning — Evaluation Report

Overview

This report presents the results of a three-agent evaluation pipeline applied to a toy dataset of lexically ambiguous knowledge-base reasoning problems. Each record introduces a homonymous entity (e.g., bat, bank, wave) whose referent is deliberately underspecified, creating cases where standard entailment reasoning is insufficient and logical inconclusiveness must be detected and classified.

The pipeline comprises:

• agent_logic — formal possible-worlds analysis under Closed-World Assumption; produces True / False / Inconclusive / Paradox.
• agent_NLI — Natural Language Inference + MacCartney Natural Logic; produces Entailment / Contradiction / Neutral and Neutral / Not-Neutral.
• agent_eval — synthesizes both outputs, detects fallacies, checks logic ↔ NLI correspondence, selects MCQ answer, assigns quality flag.

Aggregate Results

Quality flag distribution

Logic verdict distribution

NLI label distribution

Fallacy frequency

Per-Key Results

Cross-Key Observations

The inconclusive and neutral_query items are consistently handled well across all 8 keys: agent agreement is high (16/16 GOLD) because the contextual observation predicates ('flew', 'made decisions', 'eroded', 'moved quickly', 'was on the mound', 'approached') are genuinely non-discriminating, and both agents correctly identify referential underdetermination.

The most challenging query types are conclusive and not_neutral, which require agents to apply mutual-exclusion axioms as disambiguation triggers — a capability that only partially manifests: the NLI agent applies the chain for bat but both agents fail for mouse and pitcher/con.

Equivocation is the universal fallacy across all 32 items, reflecting the deliberate KB design. The KB universal property scoping flaw (properties asserted for all senses rather than per-sense) is the leading cause of CONFLICT flags, systematically affecting head, nail, paper, and wave conclusive/not_neutral items.

Logic ↔ NLI Correspondence Analysis

Logic and NLI agents agree in 23 of 32 items (71.9%), with systematic divergence patterns across two distinct failure modes.

• Mode 1 — NLI applies mutual-exclusion chain, logic does not (bat/con, bat/not): agent_NLI correctly uses the wings≠barrels axiom to derive Entailment, while agent_logic maintains Inconclusive because it does not treat the query predicate as a disambiguation signal under CWA.
• Mode 2 — Logic applies universal KB assertions, NLI maintains ambiguity (head/con, head/not, nail/con, wave/con): agent_logic returns True via an undiscriminating universal property assertion, while agent_NLI correctly flags residual referential ambiguity.

This asymmetry reflects a fundamental difference in how the two systems handle scope: formal logic under CWA propagates universal KB facts to specific instances immediately, while NLI's monotonicity reasoning is more sensitive to contextual cues.

Fallacy Analysis

Equivocation is the universal root cause across all 32 items, reflecting the deliberate introduction of polysemous entities. Hasty Generalization appears in 22 items as a latent risk — the contextual observation predicate could be, but is not, used as definitive evidence for one sense. Affirming the Consequent appears in 16 items in the same latent-risk capacity. Amphiboly appears in 5 items (head/con, head/not, nail/not, wave/inc, wave/con) where natural language predicates carry structural ambiguity beyond the equivocation on the main noun. False Premise appears in 2 nail items because the KB asserts 'Pointed tips are not keratin' as an axiom, embedding an empirical claim.

Dataset Quality Assessment

Of the 32 items, 18 receive GOLD flags (56.3%), 7 receive REVIEW (21.9%), and 7 receive CONFLICT (21.9%), with no ERROR flags. GOLD items concentrate in inconclusive and neutral_query types. REVIEW items arise where one agent reaches the correct verdict via a spurious path. CONFLICT items arise from two sources:

1. KB universal scoping flaws that cause agents to derive True/Entailment where the dataset annotates Inconclusive/Neutral (head/con, head/not, paper/inc, paper/neu).
2. Systematic agent failure to apply mutual-exclusion disambiguation in both agents simultaneously (mouse/con, mouse/not, bat/not).

Resolution requires either sense-scoped KB property assertions (source 1) or explicit query-predicate disambiguation hooks in the agent prompts (source 2). The dataset's conceptual design is sound; the KB implementation needs refinement for approximately 4 keys (head, paper, mouse, and wave).

Methodology Notes

• Closed-World Assumption applied throughout agent_logic: facts not stated in the KB are treated as false unless ambiguity is the cause, in which case they are flagged Undecidable rather than False.
• Natural Logic chains follow MacCartney & Manning (2008) relation symbols: ⊏ forward entailment, ⊐ reverse, ≡ equivalence, ^ negation, | alternation, ⌣ cover, # independence.
• MCQ correctness is evaluated by semantic consistency with the dataset's expected_reasoning field, not by string match.
• Models: agent_logic and agent_NLI use claude-haiku; agent_eval uses claude-sonnet.