Reflection: Eval Pipeline Results & Research Trajectory

Question

Based on the results and report, do you recommend progressing this work? Is this problem solved due to the evolving nature of LLMs?

Answer

What the Eval Reveals About the Research Gap

The eval achieved 71.9% MCQ accuracy (23/32) with claude-sonnet-4-6 as synthesizer. Three systematic failure modes emerged that are not reducible to general model capability:

Failure Mode 1 — Query-predicate disambiguation blindspot (logic agent)

The logic agent consistently fails to treat the query predicate itself as a disambiguation signal. In bat/not, mouse/con, mouse/not, and pitcher/con, the KB's mutual-exclusion axioms are designed to be triggered by the query predicate (e.g., "has wings" → wings≠barrels → baseball-bat excluded). But the logic agent evaluates world openness before applying the query, missing the intended resolution. This is an architectural issue — a stronger LLM using the same prompting strategy will not automatically fix it.

Failure Mode 2 — Universal KB scoping (4 of 8 KBs)

head, nail, paper, and wave all assert properties universally (e.g., "Heads contain a brain") rather than per-sense. This is intentional KB design — it introduces the ambiguity that humans detect quickly and use to arrive at inconclusiveness. The failure is not in the KB; it is in the LLM agents, which cannot reason like humans do: a human immediately recognises that "Heads contain a brain" AND "Heads have authority" under a single polysemous noun creates an incoherent universal, and flags the query as inconclusive without mechanically applying the assertion. Logic agents, operating under strict CWA, apply the universal directly and return True, bypassing the ambiguity entirely.

Failure Mode 3 — NLI exclusion-chain composition failure

For wave/con and bank/con, the NLI agent detects polysemy and returns Neutral without propagating the KB's own exclusion facts (e.g., water-molecules≠friendly-intent) to resolve the sense. The agent stops at polysemy detection rather than completing the disambiguation chain.

Is This Problem Solved by Evolving LLMs?

The mouse failure is the most diagnostic: both agents fail, agree with each other, and do so with high confidence — yet the annotation is wrong. This is not a capability gap fixable by scaling; it is a reasoning-scope gap where neither formal logic (CWA) nor NLI surface-form inference applies the query predicate as a disambiguation trigger before assessing world openness.

The inconclusiveness detection task — recognizing when a KB query is satisfiable under two mutually exclusive world models and therefore undecidable — is a property that evolving LLMs improve at in general reasoning benchmarks, but the structural distinction between satisfiable-inconclusive and conclusive remains non-trivial and is not reliably handled even by strong models without architectural support.

Specific Recommendations

Fix first (dataset quality)

1. Do not sense-scope the universal property assertions (head, nail, paper, wave) — the universals are intentional design that creates the ambiguity humans detect. The gap to address is in agent reasoning, not KB construction.
2. Re-annotate mouse/con and mouse/not — the expected reasoning commits Affirming the Consequent (treating has whiskers → rodent as a KB-licensed rule when only rodent → has whiskers is asserted). These items should be reclassified as inconclusive / neutral, not conclusive / not_neutral.

Then progress

Core Finding Worth Preserving

The fundamental research claim — that logical inconclusiveness requires detecting two simultaneously satisfiable but mutually exclusive worlds, and that this is systematically distinct from both True and False verdicts — is well-motivated and not yet solved. The eval pipeline built here (agent_logic + agent_NLI + agent_eval) is the right instrument to measure the gap, and the council-of-experts architecture with a Polysemy Expert and Arbitrator is the structural solution the results motivate.

The work is worth progressing. The dataset needs repair before the LLM capability question can be cleanly answered.

Elaboration: Failure Mode 2 — Universal KB Scoping

The Intended KB Design Pattern

All 8 KBs follow this structure:

1. X is [sense A]                    ← defines sense A
2. X is [sense B]                    ← defines sense B
3. Xs have [property of sense A]     ← universal, no sense restriction (intentional)
4. Xs have [property of sense B]     ← universal, no sense restriction (intentional)
5. [property A] is not [property B]  ← mutual exclusion
6. [property B] is not [property A]  ← symmetric
7. There is a [color] X              ← introduces individual
8. [color] X [observation predicate] ← non-discriminating context

Lines 3 and 4 are deliberately written as bare-plural universals — without sense restriction. This is how humans naturally write and speak. A human reading this KB immediately recognises that these two universals, combined with the mutual exclusion facts, create an incoherent assignment for any entity called "head" — and therefore flags any query about a specific head as inconclusive until the sense is resolved.

Why Humans Arrive at Inconclusiveness Quickly

A human reader processes the KB pragmatically:

1. Sees "Head is a body part" and "Head is the leader of organization" → registers polysemy
2. Sees "Heads contain a brain" → pragmatically restricts this to the anatomical sense
3. Sees "Heads have authority" → pragmatically restricts this to the organizational sense
4. Reads "White head made decisions" → compatible with both senses; does not resolve
5. Asked "White head is a body part" → immediately: which head? can't tell. Inconclusive.

Steps 2 and 3 are automatic pragmatic sense-restriction — humans do not apply the universals literally across both senses.

What LLM Agents Do Instead

agent_logic (strict CWA): Applies bare-plural universals as strict first-order universals: ∀x. head(x) → contains_brain(x). When asked "White head contains a brain?", it finds this universal, instantiates it for white_head, and returns True — without pausing to consider that both universals cannot coherently apply to the same entity. The logic agent does not possess pragmatic sense-restriction.

agent_NLI (surface-form monotonicity): Performs better because it applies world-knowledge modulation. It resists the universal inheritance and returns Neutral — closer to the correct human intuition — but it does so by detecting residual ambiguity rather than constructing the inconclusiveness argument from first principles.

The Agent Conflict This Produces

The Deeper Research Finding

The gap between human and LLM inconclusiveness detection is precisely located in pragmatic sense-restriction of universals.

Humans automatically scope bare-plural properties to the sense they belong to, making the polysemy and resulting inconclusiveness immediately visible. LLM agents either apply universals mechanically (logic) or stop at polysemy detection (NLI), neither of which replicates the human cognitive path.

This reframes the research question from "can LLMs detect logical inconclusiveness?" to "can LLMs apply pragmatic sense-restriction to universal KB assertions the way humans do?" — a sharper and more tractable target for the council-of-experts architecture, where a Polysemy Expert agent performs the sense-restriction step explicitly before agent_logic evaluates world openness.

What This Means for the Dataset

The universals should stay as-is. They are the test stimulus. The correct fix is on the agent side:

• Prompt the logic agent to identify which sense each KB property applies to before computing closure.
• Or add a Polysemy Expert step that produces sense-scoped property assignments for the logic agent to consume.
• The NLI agent is closer to the intended behaviour but needs to complete the inconclusiveness argument rather than stopping at Neutral.