Author raw notes & upload papers
Review & correct second-brain entries
Design PROMPT.md & validate schema
Ingest entries into second brain
Reflection & Q&A
Build mental map
Generate polysemy KB datasets
Parallel logic + NLI + eval
Produce eval report
Render interactive HTML pages
Author raw research notes
Human researcher writes paper notes, group-meeting logs (Jan–May 2023), and uploads PDFs. Source material covers SICCK/NLI compositionality and logical inconclusiveness abstention.
Design second-brain schema
Human writes CLAUDE.md specifying entry types (concept/paper/project/experiment/model/tool/ideation), YAML frontmatter format, directory structure, and research domain overview.
Ingest entries into second brain
Human writes Markdown entries from raw notes following CLAUDE.md schema; agent_mentalmap structures and cross-links them. Produces 4 concepts, 3 papers, 3 projects, 3 experiments, 1 model, 1 tool, 1 ideation entry.
Build mental map
agent_mentalmap reads all second-brain entries, extracts thematic clusters (NLI Compositionality, Logical Inconclusiveness, Agentic Evaluation, Pragmatic Sense-Restriction, Dataset Construction, LLM Reasoning Gaps), builds Mermaid graph and hierarchical outline.
Identify KB generation approach ⚠ Decision Point
Human reviews mental map, identifies gap in inconclusiveness detection research, decides on polysemy nouns + resolution refutation strategy. Designs the 18-step KB construction protocol covering two meanings, distinguishing features, equivalence exclusions, colored instance, and ambiguous action.
Generate polysemy KB datasets
agent_datagen follows the 18-step protocol for each polysemy noun (bat, head, nail, paper, bank, wave, mouse, pitcher + additional). Produces 4 JSON arrays: original_format, neutral_format, separated_inconclusive, separated_conclusive. Writes canonical JSONL for downstream eval.
Validate and hand-craft toy dataset ⚠ Decision Point
Human spot-checks KB quality for generated dataset, verifies ambiguous actions are genuinely compatible with both meanings, and hand-crafts a smaller toy dataset for pipeline testing. Confirms JSONL schema integrity (key, kb, inconclusive, conclusive, neutral_query, not_neutral).
Visualize knowledge structure
agent_mentalmap_html reads mental_map.md and mental_map_index.md, renders an interactive D3.js force-directed graph with theme-coded nodes, edge types, hover sidebar, and filter controls. Produces a self-contained HTML file.
JSONL Dataset
data/toy_data.jsonl | data/generated_data.jsonl
Each line: {key, kb, inconclusive, conclusive, neutral_query, not_neutral}
eval_record skill
Loads JSONL, applies optional --key and --type filters, builds flat work queue (one item per key × query_type). Dispatches agent_logic and agent_NLI in parallel for each item.
/eval_record data/toy_data.jsonl
- Parse KB atomic facts
- Identify referential scope / possible worlds
- Evaluate per world (True / False / Undecidable)
- Aggregate verdict: True / False / Inconclusive / Paradox
- Applies Closed-World Assumption (CWA)
- Premise inventory from KB
- Monotonicity scan (upward/downward)
- Polarity analysis (negation, quantifiers)
- NLI label: Entailment / Contradiction / Neutral
- MacCartney Natural Logic chain (⊏ ⊐ ≡ ^ | ⌣ #)
agent_eval sonnet
- Step 0: Load full research context (concepts/, experiments/, ideation/, papers/, projects/, models/, tools/)
- Cross-check logic ↔ NLI verdicts (expected: True↔E, False↔C, Inconclusive↔N)
- Fallacy scan: Equivocation, Amphiboly, Hasty Generalization, Affirming Consequent, False Dichotomy, Paradox, False Premise
- Natural Logic chain validation (monotonicity consistency)
- MCQ answer selection from provided options
- Dataset quality flag: GOLD / REVIEW / CONFLICT / ERROR
Results files
agents_match, fallacies_detected, mcq_answer_selected,
mcq_correct, dataset_quality_flag, eval_reasoning
per-key blocks, cross-key analysis,
fallacy frequency, dataset quality assessment
👤 Human interrogates report
Human reads eval_report.md, identifies failure modes, asks follow-up questions about Failure Mode 2 (universal KB scoping), and requests elaboration. Q&A recorded.
🤝 results/reflection.md
Recommendation: Yes, progress — with targeted fixes first. Not solved by evolving LLMs — systematic failures in mouse/con & mouse/not reveal reasoning-scope gaps. Reframed research question: "can LLMs apply pragmatic sense-restriction to universal KB assertions?" Recommends Polysemy Expert agent addition.
Expected Verdict Correspondences
| Logic (agent_logic) | NLI (agent_NLI) | Neutral Classification | Dataset Quality |
|---|---|---|---|
| True | Entailment | Not Neutral | GOLD if agents agree |
| False | Contradiction | Not Neutral | GOLD if agents agree |
| Inconclusive | Neutral | Neutral | GOLD if agents agree |
| Paradox | — | Manual review | REVIEW / ERROR |
| Any | Any | Divergent | CONFLICT |
Fallacy Frequency (32 items evaluated)
| Fallacy Type | Occurrences | % of items | Notes |
|---|---|---|---|
| Equivocation | 32 | 100% | Root cause: polysemy noun with two referents |
| Hasty Generalization | 22 | 68.8% | Inferring referent from single observation predicate |
| Affirming the Consequent | 16 | 50.0% | P→Q, Q ∴ P (e.g., flew → mammal) |
| Amphiboly | 5 | 15.6% | Structural sentence ambiguity |
| False Premise | 2 | 6.3% | KB fact contradicts world knowledge |
Iterative Collaboration Loop
Example cycle: Human writes eval_report.md → Claude runs eval_record → Human reads report, identifies Failure Mode 2 framing error → Human corrects framing (universals are intentional) → Claude elaborates reflection.md with corrected framing → Human accepts reflection → agent_eval.md updated with Step 0 context-loading requirement.