Human-AI Collaborated AI4Science

Overview

This project demonstrates how Claude Code can serve as an AI-assisted "second brain" to help researchers organize, ideate, and plan around prior work — including accepted and non-accepted papers — in the domains of Natural Language Inference (NLI), Natural Logic, and neurosymbolic methods for logical inconclusiveness.

The workflow ingests raw research materials (papers, meeting notes, code repositories, posters) and produces a structured wiki plus an agentic simulation that evaluates logical inconclusiveness on a toy knowledge-base dataset.

Natural Language Inference Natural Logic Logical Inconclusiveness Agentic AI Claude Code AI4Science

Interactive Visualizations

🔬

Research Pipeline

Annotated diagram of every stage in the human-AI collaboration — ingestion, wiki generation, agentic simulation, and evaluation.

🕸️

Knowledge Graph

Interactive graph linking concepts, papers, projects, experiments, models, and tools extracted from the research materials.

🧪

Architecture

Agentic System Architecture for Research Ideation, Toy Data Evaluation and Reflection

📊

Evaluation Report

Full per-key results from the three-agent pipeline on 32 items (8 polysemous KB keys × 4 query types). 71.9% MCQ accuracy.

💡

Reflection

Analysis of failure modes, whether evolving LLMs solve the problem, and a roadmap for the council-of-experts architecture.

🗃️

Dataset Viewer

Interactive viewer for the 32-item toy KB dataset (8 polysemous nouns × 4 query types) with per-item logic and NLI verdicts.

🧪

Hypothesis Test

Formal test of whether logical inconclusiveness is structurally analogous to NLI Neutral — protocol, correspondence metrics, and verdict thresholds.

Research Context

Two prior research threads underpin this work:

SICCK / Compositional NLI — Accepted at ACL 2023 NLRSE workshop. A synthetic dataset of 1,304 sentence pairs testing NLI model compositionality under Natural Logic modifiers (upward/downward monotonicity, negation, etc.).
Inconclusiveness-based Abstention — Proof-of-concept for detecting satisfiable-but-inconclusive KB queries via polysemy nouns, resolution refutation, and a proposed council-of-experts agentic architecture. Submitted to AAAI 2026 LMReasoning bridge workshop.

Connecting thread: NLI's "Neutral" label ≈ logical inconclusiveness when the evidence is insufficient to decide entailment direction — a principled bridge between the two works.

Methodology — Second Brain via Claude Code

The project uses Claude Code directly (CLI) to build a structured wiki by running a sequence of natural-language prompts against the raw research materials. No custom code is required for the wiki-building stage.

Step-by-step prompt sequence

1Initialize the second-brain schema: directory structure, CLAUDE.md, INDEX.md, LOG.md.
2Ingest the accepted paper markdown (SICCK/Compositional NLI).
3Ingest background meeting notes; route entries to concepts, experiments, projects.
4Ingest the NLRSE ACL 2023 poster as a separate summary.
5Ingest the non-accepted paper (Inconclusiveness-based Abstention).
6Ingest the associated code repository; route summaries to experiments/tools/models.
7Generate an ideation entry synthesizing both threads into a council-of-experts agentic architecture proposal.
8Query Claude Code for agentic framework recommendations (AutoGen, ADK, CrewAI).
9Run the three-agent simulation (Logic Expert · NLI Expert · Reasoning Coordinator) on the toy KB dataset.

Reproducibility — Prompt Template

Given a raw/ folder containing markdown versions of two papers and research notes, the following template reproduces the wiki-building and ideation steps:

You will be acting as an AI research organization assistant, helping to build
a "second brain" wiki system for academic research.

<accepted_paper>
{"raw/Synthetic Dataset for Evaluating Complex Compositional Knowledge ..."}
</accepted_paper>

<rejected_paper>
{"raw/A proof-of-concept method for Inconclusiveness-based Abstention.md"}
</rejected_paper>

<code_repository>
{[anonymous code repository URL]}
</code_repository>

<background_notes>
{"assets/Natural-logic-group-meeting.md"}
</background_notes>

<research_question>
Can a council of experts help an Arbitrator decide if a query is logically
inconclusive or not?
</research_question>

Organize these materials into: CONCEPTS · PAPERS · PROJECTS · EXPERIMENTS
· TOOLS · MODELS · IDEATION · RECOMMENDATIONS

Agentic Simulation — Three-Agent Architecture

The second phase runs an agentic evaluation on the toy KB dataset (logically-inconclusive-kbs/kbs/**combined.jsonl). Three agents interact in sequence:

Agent 1 — Logic Expert

Analyzes KB-query pairs for logical structure, formal validity, completeness, inconclusiveness, and contradictions using FOL/SAT reasoning.

Agent 2 — NLI Expert

Evaluates entailment, contradiction, and neutrality via Natural Logic; applies monotonicity, polarity, and semantic-relation reasoning.

Agent 3 — Reasoning Coordinator

Synthesizes both expert analyses; detects fallacies, ambiguities, paradoxes, and produces a final logical-inconclusiveness verdict.

Framework options evaluated: AutoGen (best for the deliberation loop), Google ADK (best for free-tier Gemini-primary setup), and CrewAI (simpler alternative).

Repository Structure

human-ai-collaborated-ai4science/
├── input/raw/              ← source papers and meeting notes (markdown)
├── claude-code-agentic-simulation/
│   ├── concepts/           ← extracted theoretical concepts
│   ├── papers/             ← paper summaries (accepted, rejected, poster)
│   ├── projects/           ← ongoing research project descriptions
│   ├── experiments/        ← experimental configurations and datasets
│   ├── models/             ← ML and logical models
│   ├── tools/              ← tool and framework summaries
│   ├── ideation/           ← council-of-experts synthesis
│   ├── raw/                ← input copies used by this simulation
│   └── output/             ← pipeline visualization, knowledge graph, eval reports
├── old-prompt-based-simulation/   ← earlier prompt-only simulation run
└── docs/                   ← GitHub Pages (this site)

Theoretical Motivation

The council-of-experts architecture is grounded in Agency in Collaborative Learning (Simons Institute for the Theory of Computing) and the emerging literature on Power and Limitations of Aggregation in Compound AI Systems.

The project was motivated by the "Second Brain" initiative popularized on social media (Karpathy, 2024) and adapted for academic research organization using existing Obsidian + Claude Code workflows.