gabriel / muse public
Open #69 Enhancement
filed by gabriel human · 19 days ago

Research Papers Domain — implementation plan (TDD-first, comprehensive)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Research Papers Domain — Implementation Plan

Background

Muse treats "domain" as a first-class primitive (muse/domain.py::MuseDomainPlugin). A plugin implements six methods — snapshot, diff, merge, drift, apply, schema — and gets the full VCS engine free: commit DAG, branching, checkout, lineage, log graph, merge-base finder. code/midi proved the model for symbolic data; the CAD (#66), 3D Scenes (#67), and Video Timelines (#68) plans are proving it for continuous geometry, spatial hierarchies, and temporal edit structures.

Research papers are a sixth shape of problem, and the most conceptually ambitious: the artifact being versioned isn't just structured content, it's an argument — claims, evidence, methods, and results connected by a citation graph that reaches outside the document into the rest of the scientific literature. Today a paper is either a LaTeX source tree (semantically rich but tool-fragile), a PDF (semantically opaque — no claims, no citation graph, just glyphs on a page), or a DOCX (semi- structured, proprietary). None of these let a VCS answer "what claim changed between these two revisions" or "does this citation still support what it's cited for." If Muse can version a paper the way it versions code — diffing claims, not just text; merging sections, not just paragraphs; validating a citation graph, not just spelling — it proves the domain-plugin model generalizes to argumentative/scholarly knowledge, not just executable, geometric, or temporal structure. The stated goal is not to version documents. It is to version the knowledge itself.

This issue is the plan only. Each numbered phase in §11 becomes its own staging issue once reviewed and accepted.

Goal

Ship a muse/plugins/paper/ domain plugin, structured like muse/plugins/midi/ and the CAD/3D-Scenes/Timeline plans (a plugin.py entrypoint plus focused modules for diff/merge/query/invariants/import-export), that:

  • Registers in muse domains --json alongside code, midi, identity, mist, social, and (once landed) cad/scene3d/timeline.
  • Passes muse code test --json for every module it touches.
  • Snapshots, diffs, and merges a paper with section/claim/citation-level semantic awareness — never as an opaque PDF/DOCX blob.
  • Supports a query DSL (muse paper query "...") answering the examples in §6.
  • Detects the drift/validation classes in §7 via muse status/muse check.
  • Has test coverage per §8, with test IDs (PAP_01, PAP_02, …) referenced from both this plan and the tests that satisfy them.

"Done" for the plan means: reviewed, decomposed into §11's breakdown, with acceptance criteria concrete enough for a different agent to implement test-first.

1. Domain Scope

Supported initial formats

LaTeX (the dominant authoring format in math/physics/CS) and Markdown-with-frontmatter (the lightweight alternative, increasingly common for preprints and technical blogs that function as papers). v0 targets single-paper documents — journal articles, conference papers, preprints — not multi-document theses or books.

Two-layer strategy, consistent with every prior plan:

  1. Canonical semantic layer (Muse-owned): a JSON-serializable paper IR (§2) that Muse snapshots, diffs, merges, and queries directly — sections, claims, citations, and figures as addressable elements, not text spans.
  2. Interchange boundary: JATS XML (Journal Article Tag Suite, the NLM/NISO standard already used by PubMed Central, eLife, and most scholarly publishing pipelines) as the primary structured-interchange target, because it is (a) an open, mature standard purpose-built to express exactly the elements in §2 (sections, claims-adjacent structures, citations, figures, tables) rather than visual layout, and (b) already the output of mature LaTeX-to-JATS and Pandoc conversion pipelines, so v0 does not need to write a LaTeX parser from scratch. LaTeX source is the primary v0 authoring format (via a LaTeX→JATS→IR import pipeline); Markdown is a secondary authoring format via a simpler Markdown→IR path (no JATS intermediate needed, since Markdown's structure already maps directly onto §2's section/figure/citation model). PDF is import-only, and only as a last-resort, lossy fallback: text and layout are extracted, but claims/ methods/results sections are inferred by heading-pattern heuristics, not guaranteed accurate — always flagged as a lossy import (§7), never silently treated as equivalent to a JATS/LaTeX/Markdown-sourced snapshot.

Relationship to LaTeX, Markdown, JATS XML, DOCX, PDF, and arXiv

Format Role
LaTeX Primary v0 authoring format; imported via LaTeX→JATS (existing tools: latexml, Pandoc) then JATS→IR, so Muse owns one XML-to-IR bridge rather than a bespoke LaTeX parser.
JATS XML v0's structured interchange format and integration point — the single format the IR's import/export adapter targets natively.
Markdown Secondary authoring format; direct Markdown→IR path (frontmatter → Metadata/Authors, headings → Section, fenced math → Equation, a bibliography block → Bibliography).
DOCX Import-only, via a DOCX→JATS conversion path (Pandoc) — same fidelity ceiling reasoning as the CAD/3D-Scenes/Timeline plans' lower-fidelity formats: useful as a fixture source, not a first-class round-trip target given DOCX's weak semantic structure for citations/equations.
PDF Import-only, lossy fallback (see above) — never a round-trip target; v0 has no PDF export (a paper's canonical form is always LaTeX/Markdown/JATS, and PDF is generated downstream by existing tools, not owned by this domain).
arXiv Not a format — a distribution channel. arXiv submissions are themselves LaTeX source bundles, so a paper already versioned in this domain can be exported to an arXiv-submission-ready LaTeX tarball as a downstream convenience, not a separate import/export adapter.

What is in scope for v0

  • Single-paper documents with standard IMRaD-family structure (Introduction, Methods, Results, Discussion — or the equivalent sections for non-empirical papers).
  • Structured metadata: title, authors, affiliations, abstract, keywords, publication venue/date, version history.
  • Figures and tables as first-class, content-addressed elements with captions.
  • Equations as addressable elements (not flattened into surrounding text).
  • Citations and a full bibliography, with a preserved reference graph (which claim cites which bibliography entry, and what that entry's own metadata is).
  • A lightweight claims layer: explicit, extractable statements the paper asserts (a hypothesis, a finding, a limitation) — extracted via structural cues (topic sentences, explicit "we claim/show/find that" phrasing, and heading-level context) rather than full NLP entailment in v0 (see §10 for the deferred, fuller version).
  • Dataset and code-repository references as first-class external references.
  • Supplementary materials as external references with resolution state.

What is explicitly out of scope for v0

  • Full NLP-based claim extraction, entailment checking, or automatic "does the evidence support this claim" reasoning — v0's claims layer is structurally extracted (explicit statements + heading context), not semantically verified. Full evidence-graph reasoning is a v1+ concern (§10).
  • Multi-paper corpora, systematic-review aggregation, and meta-analysis structures.
  • Peer review workflow state (reviewer comments, editorial decisions) as domain content — this may belong to MuseHub's proposal/review layer (which already exists generically), not to a paper-specific reweighted Review element in v0.
  • LaTeX macro/package-level fidelity: custom LaTeX macros that redefine document structure in non-standard ways are flagged as lossy import (§7), not modeled.
  • Live/executable papers (embedded runnable code, live data bindings) — deferred to §10 as a named future direction, not attempted in v0.
  • Full citation-graph crawling beyond the current paper's own bibliography (i.e., Muse does not fetch and index the entire citation network of every cited paper in v0 — see §6 and §10 for the boundary between "this paper's local citation graph" and "the global scholarly graph").

2. Canonical Model

All elements are content-addressed (sha256:<hex>) and referenced by ID, never positional index — same identity discipline as every prior domain plan — so reordering a section or renumbering a citation never produces a spurious diff.

Element Fields Notes
PaperMetadata id, title, abstract: Section.id, keywords, venue, publication_date, doi, version_label, schema_version Title/abstract are metadata and content — abstract points at a real Section so it diffs like any other section, not as a flat string.
Author id, name, orcid \| null, affiliation_refs: [Affiliation.id], contribution_roles (CRediT taxonomy: conceptualization, methodology, writing, etc.), corresponding: bool Content-addressed by (name, orcid) when ORCID is present (globally stable identity), else by name alone (weaker, flagged as such).
Affiliation id, institution, department \| null, country Deduplicated across authors sharing an institution.
Section id, heading, level (nesting depth), parent_ref: Section.id \| null, kind (introduction|methods|results|discussion|conclusion|generic), paragraphs: [Paragraph.id] (ordered), subsections: [Section.id] (ordered) kind is inferred at import time from heading text/position (heuristic, not authoritative) — used to scope claims/methods/results queries (§6).
Paragraph id, text, citation_refs: [Citation.id] (inline, ordered by appearance), claim_refs: [Claim.id] The atomic diffable text unit — Muse diffs at paragraph granularity, not line/sentence, mirroring the CAD/3D-Scenes plans' choice of a coarsest-useful-unit for content that doesn't need finer diffing in v0.
Figure id, caption, image_ref: content_id, source_data_ref: content_id \| null (e.g. the plotting script/data behind a chart, if provided), referenced_in: [Paragraph.id] Image bytes are content-addressed, same treatment as every prior plan's binary payloads.
Table id, caption, rows: [[cell_value]], referenced_in: [Paragraph.id] Structured (not an image) when source data permits — enables the table-diff granularity in §4.
Equation id, latex_source, label \| null (e.g. eq:loss-function, for cross-reference), variables: [{symbol, description}], referenced_in: [Paragraph.id] variables is populated heuristically from surrounding prose in v0 (not a full CAS parse) — enables the "find equations containing variable X" query (§6) without requiring symbolic-math infrastructure.
Citation id, bibliography_ref: BibEntry.id, context: Paragraph.id, intent (background|method|comparison|support|contrast — inferred from surrounding phrasing, best-effort) An in-text citation instance, distinct from the BibEntry it points to — the same citation target can be cited many times with different intent.
BibEntry id, doi \| null, title, authors, venue, year, cited_by: [Citation.id] Content-addressed by DOI when present (globally stable, dedups across papers), else by (title, first-author, year) — flagged as weaker identity when DOI is absent.
Footnote id, text, anchor_ref: Paragraph.id
Claim id, text, kind (hypothesis|finding|limitation|contribution), section_ref: Section.id, supporting_refs: [Figure.id \| Table.id \| Equation.id \| Citation.id], status (asserted|supported|unsupported — computed, see §7) The domain's most novel element — see §1 for extraction method and §10 for the deferred fuller evidence-graph version.
Method id, description, section_ref: Section.id, dataset_refs: [Dataset.id], code_repo_refs: [CodeRepository.id]
Experiment id, method_ref: Method.id, hypothesis_ref: Claim.id \| null, result_refs: [Result.id] Links a method to the claim it tests and the results it produced — the backbone of the domain's argument structure.
Result id, description, figure_or_table_ref: Figure.id \| Table.id \| null, supports_claim_ref: Claim.id \| null
Dataset id, name, uri_or_doi, content_id_at_link_time \| null, resolved: bool, license \| null External reference with resolution state, same pattern as the CAD/3D-Scenes/Timeline plans' ExternalReference, promoted to first-class since Method/Experiment reference it directly.
CodeRepository id, uri (a Muse repo, GitHub URL, etc.), commit_ref \| null, resolved: bool Same promotion rationale as Dataset.
SupplementaryMaterial id, kind, content_id \| null, uri_or_path \| null, resolved: bool
VersionMetadata authors_at_version: [Author.id], changelog_summary \| null, source_format (latex|markdown|jats|docx|pdf), source_checksum, schema_version

Design invariant carried over from every prior plan: identity is content- and reference-based, never positional. Reordering sections, renumbering citations, or resequencing bibliography entries is never itself a diff — only a change to an element's own content or its *_ref fields is.

3. Snapshot Design

Deterministic document normalization

snapshot() walks the live state (a LaTeX project or Markdown file under state/, same pattern as every prior plan's format walk) and produces the canonical IR as the SnapshotManifest:

  1. Resolve every element to a stable ID (below).
  2. Canonicalize citation keys, whitespace/line-wrapping in prose (paragraph text is normalized to a single logical string — line-wrap position is a LaTeX/Markdown source-formatting artifact, not semantic content, and must never produce a diff).
  3. Canonicalize ordering: Section.subsections and Paragraph lists preserve authored order (semantically meaningful — argument order matters); citation_refs within a paragraph preserve appearance order; Bibliography/BibEntry lists are sorted by ID for serialization (citation numbering is a rendering concern, not semantic content — renumbering references on a style change must never diff).
  4. Compute each Section's word count and citation count at snapshot time — a cheap invariant used to short-circuit full-section diffs (§4), mirroring every prior plan's use of a cheap pre-filter before expensive comparison.

Stable identity for semantic elements

  • Section identity is derived from (parent_identity, heading_text) at first snapshot, then — critically — pinned across subsequent snapshots by matching against the previous snapshot's sections via heading-text + position similarity, not re-derived from scratch each time. This is necessary because a section rename (e.g. "Related Work" → "Prior Work") should not cascade into every subsequent section being reidentified, the same rename-cascade problem the 3D Scenes plan solved with structural/display identity — but sections don't have a content hash independent of their own heading the way scene nodes have geometry, so the fix here is a matching algorithm at diff time (§4) rather than a hash-construction trick: identity is provisionally content-derived at import, then explicitly carried forward and only reassigned when matching confidence is too low to reasonably carry it forward (a genuine section split/merge/rewrite).
  • Paragraph identity is content-derived from a fuzzy hash of its own text (not its position) plus its parent Section's identity — a paragraph edited in place keeps its identity if the edit distance is below a threshold (tracked as an open question in §11: what specific threshold), and is treated as delete+insert (not a ReplaceOp) above that threshold, since a total rewrite is better read as "old paragraph gone, new one added" than as a giant opaque replace.
  • BibEntry/Citation identity: DOI when present (globally stable across papers, enabling cross-paper bibliography dedup); (title, first-author, year) fallback when absent, explicitly flagged as weaker identity (§7).

Content-addressed figures and supplementary assets

Figure/table image bytes, any source data/plotting scripts, and supplementary material blobs are extracted and hashed independently (sha256:<hex>), same content- addressing discipline as every prior plan — a figure reused unchanged across paper revisions collapses to one stored blob.

Citation normalization

A citation's intent and a BibEntry's metadata are normalized independent of citation style (numeric [3], author-year (Smith, 2023), etc.) — style is a rendering concern applied at export time from the canonical IR, never stored as part of the semantic snapshot. This mirrors the Video Timelines plan's separation of RationalTime value from drop-frame display formatting.

Reference graph preservation

The full Citation → BibEntry and Claim → supporting_refs graphs are preserved as explicit edges in the IR (not re-derived at query time from prose), so muse paper query/citation-graph traversal (§6) is a direct graph walk over committed data, not a re-parse of the document on every query — the same "snapshot-time graph, not query- time inference" principle the CAD plan applies to its feature-dependency DAG.

Deterministic serialization

JATS XML round-trips (via latexml/Pandoc) are not guaranteed byte-stable across tool versions — Muse never hashes JATS/LaTeX bytes directly for content-addressing; it always re-serializes through the canonical IR's fixed field order and ID scheme (§2), the same discipline as every prior plan's "never hash the interchange format's bytes directly" rule. A golden-snapshot suite (§8) locks this down.

Large asset handling

Figures/supplementary data follow the CAD/3D-Scenes plans' model (content-addressed blobs in .muse/objects/, no new storage tier) rather than the Timeline plan's external-reference-only model — paper figures/supplementary files are typically kilobytes-to-low-megabytes, not the gigabyte-scale media the Timeline plan had to externalize. Datasets, however, follow the Timeline plan's model exactly: referenced externally (Dataset.uri_or_doi), never vendored, since datasets can be arbitrarily large and are usually already hosted on a dedicated repository (Zenodo, a data journal, institutional storage).

4. Semantic Diff

diff(base, target, *, repo_root=None) returns a StructuredDelta, same contract as every prior plan.

  • Section-level diffs: InsertOp/DeleteOp/ReplaceOp keyed on the pinned section identity (§3); a heading-only change is a RenameOp, distinct from content changes within the section.
  • Paragraph diffs: InsertOp/DeleteOp/ReplaceOp keyed on paragraph identity (§3); reported with a text-similarity score so a reviewer can see "85% similar, likely an edit" vs. "0% similar, likely unrelated new content."
  • Figure/table/equation changes: ReplaceOp when image_ref/rows/latex_source changes; caption-only changes reported distinctly from content changes (mirrors every prior plan's value-change vs. structural-change distinction).
  • Citation additions/removals: InsertOp/DeleteOp keyed on Citation.id; a citation's intent changing (e.g. background → contrast) is reported separately from the citation being added/removed outright.
  • Bibliography diffs: BibEntry additions/removals; a BibEntry's own metadata changing (e.g. a preprint's DOI updates once published) is reported as "reference resolved/updated," distinct from the entry being added or removed.
  • Author changes: Author add/remove/reorder — author order is semantically meaningful in academic authorship (unlike most ordering in this domain, this is the one deliberate exception, mirroring the Video Timelines plan's Track-order exception) — reordering authors is a reported diff.
  • Claim additions/modifications/removals: keyed on Claim.id; a change to supporting_refs (evidence added/removed for a claim) is reported distinctly from a change to the claim's own text.
  • Methodology changes: Method.description/dataset_refs/code_repo_refs diffed field-level.
  • Experimental result changes: Result diffs report whether the supporting data (figure/table) changed vs. whether the interpretation (description) changed — these are different kinds of change with different review implications.
  • Dataset reference changes: a content_id_at_link_time change ("dataset updated") is reported distinctly from a resolved flag flip, same pattern as every prior plan's ExternalReference treatment.
  • Rename and move detection for sections: a section moved to a different parent (e.g. promoted from a subsection to a top-level section) is a MoveOp, not delete+insert, using the pinned-identity matching from §3.

5. Merge Strategy

merge(base, left, right, *, repo_root=None) — three-way, merge_mode: three_way, consistent with every prior plan.

  • Independent section merges (the common case): edits to disjoint sections merge automatically via the engine's default address-keyed map merge, driven by schema()'s dimensions (§9) — one author revises Methods while another revises Discussion.
  • Same-paragraph conflicts: both branches edit the same Paragraph.text differently — always a reported conflict (no auto-resolution of prose, consistent with every prior plan's rejection of silently picking a value neither author chose); if the edit-distance-based identity split (§3) determines the branches produced genuinely different paragraphs (not just different edits of one), both are kept as a non-conflicting insert of two paragraphs rather than forced into a conflict — an explicit, tested boundary case (§8).
  • Citation conflicts: both branches change the same Citation.bibliography_ref or intent differently — reported conflict.
  • Bibliography conflicts: both branches edit the same BibEntry's metadata differently — reported conflict; both branches adding the same paper (same DOI, resolved independently) auto-merges into one BibEntry by construction, since DOI- based content addressing (§2) makes them the same object, not two competing edits.
  • Figure conflicts: both branches replace the same Figure.image_ref with different content — reported conflict, same coarse "pick one" treatment as the CAD/3D-Scenes plans' geometry conflicts (no semantic image diff in v0).
  • Equation conflicts: both branches change the same Equation.latex_source differently — reported conflict; a label cross-reference field change is independent and auto-merges if only one branch touched it.
  • Dataset conflicts: both branches point the same Dataset.id at different uri_or_doi — reported conflict; a resolved flag change is not (mirrors the Timeline plan's missing-media treatment: a dataset becoming resolved/unresolved is a state change, not a value dispute).
  • Concurrent author edits: both branches add different new authors — auto- merges (disjoint additions), but the resulting combined author order requires explicit reconciliation, always surfaced as a review item (not a hard conflict, since there's no "correct" merged order, but not silent either) — a third category beyond auto-merge/conflict, mirroring the Timeline plan's missing-media-warning category.
  • Human-readable merge reports: every conflict carries both branches' values, the base value, and a plain-language cause string (e.g. "Both branches edited the same paragraph in section 'Methods': left changes the sample size description, right changes the statistical test description — 40% textual overlap, likely independent edits misidentified as the same paragraph, escalate to human review"), consistent with the workspace-wide conflict-marker convention.

6. Query Support

A query DSL mirroring muse code query and the CAD/3D-Scenes/Timeline plans' equivalents (muse paper query "<predicate>" --json), plus purpose-built subcommands.

muse paper query "claim.status == 'unsupported'" --json
muse paper query "citation.target.doi == '10.1038/s41586-023-00000-0'" --json
muse paper query "equation.variables contains 'lambda'" --json
  • Find all claims: muse paper query "element.type == 'claim'" --json / muse paper claims --json
  • Find unsupported claims: muse paper query "claim.status == 'unsupported'" --json / muse paper unsupported-claims --json — status is computed at snapshot/drift time from whether supporting_refs is non-empty (§7).
  • Find citations to a paper: muse paper query "bibentry.doi == '<doi>'" --json / muse paper cited-by "<doi>" --json
  • Find figures discussing a topic: muse paper query "figure.caption contains 'attention mechanism'" --json
  • Find equations containing variables: muse paper query "equation.variables contains 'theta'" --json / muse paper find-equations --variable theta --json
  • Find experimental methods: muse paper query "element.type == 'method'" --json / muse paper methods --json
  • Find datasets used: muse paper query "element.type == 'dataset'" --json / muse paper datasets --json
  • Find all references to a concept: muse paper query "paragraph.text contains 'gradient descent'" --json (a lexical search in v0; semantic/embedding-based concept search is a v1+ concern, §10)
  • Find papers by author, institution, keyword, or publication date: this query spans multiple papers/repos — muse hub search papers --author "..." --institution "..." --keyword "..." --after 2024-01-01 --json, a MuseHub-level cross-repo search endpoint (§9), not a single-repo muse paper query.
  • Traverse citation graphs: muse paper citation-graph --depth 2 --json — walks Citation → BibEntry edges outward from the current paper; depth beyond 1 requires the cited papers to themselves be Muse-versioned repos (graceful degradation to "external, unresolved" otherwise) — the boundary called out in §1's out-of-scope section (no global citation-graph crawling in v0).

7. Validation and Drift Detection

Surfaces through muse status (via drift()) and muse check/muse code invariants, same pattern as every prior plan's _invariants.py.

  • Broken citations: a Citation.bibliography_ref pointing at a missing BibEntry — commit-blocking (structurally invalid).
  • Missing bibliography entries: a BibEntry with no DOI and unresolved metadata (couldn't be validated against a DOI registry at import time) — flagged informational, not blocking (mirrors the Timeline plan's offline-media-is-routine policy: many valid citations, e.g. older papers, simply lack a DOI).
  • Missing figures: a Figure/Table/SupplementaryMaterial with an unresolved content_id/uri — commit-blocking for figures actually referenced by a paragraph (broken reader experience), informational for unreferenced ones.
  • Missing supplementary assets: SupplementaryMaterial.resolved == false — flagged, not blocking, same offline-reference policy as datasets.
  • Invalid references: an Equation.label cross-reference or Section parent pointer referring to a nonexistent element — commit-blocking.
  • Duplicate citations: two BibEntry elements resolving to the same DOI that failed to dedup (an import bug, since DOI-based content addressing should prevent this by construction) — flagged as a snapshot determinism failure (hard error, same treatment as every prior plan's non-deterministic-import check), not merely a warning, since it indicates the identity scheme itself failed.
  • Orphaned sections: a Section with no parent_ref that isn't a top-level section per the document's own structure (a structural anomaly from a bad import) — flagged, not blocking, surfaced for human review.
  • Broken equation references: a paragraph referencing an Equation.label that doesn't resolve to any Equation — commit-blocking, same class as broken citations.
  • Non-deterministic imports: re-importing the same source (LaTeX/Markdown/JATS) twice must produce identical canonical IR — hard-error regression gate, same treatment as every prior plan.
  • Schema migration risks: VersionMetadata.schema_version mismatches flagged before diff/merge across the boundary, never auto-migrated silently, consistent with the workspace-wide muse code migrate permission rule.

8. TDD Plan

Test IDs use the PAP_NN prefix, per the workspace convention (CAD_NN, SCN_NN, VID_NN).

Tier Coverage Example test IDs
Unit Canonical model (de)serialization for every element type; DOI-based vs. fallback BibEntry identity; paragraph fuzzy-hash identity with edit-distance threshold. PAP_01–PAP_03
Golden snapshot Fixed fixtures (a short IMRaD paper, a math-heavy paper with labeled equations, a multi-author paper with CRediT roles, a paper with datasets/code references) snapshotted once, checked byte-for-byte against committed golden JSON. PAP_10–PAP_13
Round-trip import/export LaTeX→JATS→IR→JATS→LaTeX and Markdown→IR→Markdown must reproduce input within fidelity ceiling and produce identical IR on re-import; DOCX→JATS→IR must produce a valid fidelity-reduced subset, not silently divergent. PAP_20–PAP_22
Semantic diff One test per §4 category (section, paragraph, figure/table/equation, citation, bibliography, author-order, claim, method, result, dataset, section rename/move). PAP_30–PAP_41
Merge One test per §5 case, incl. the paragraph-identity-split non-conflict, DOI-based bibliography auto-merge, and concurrent-author-order review-item category. PAP_50–PAP_58
Citation graph Citation → BibEntry graph construction correctness; depth-2 traversal with a mix of resolved/unresolved external papers; duplicate-DOI dedup. PAP_60–PAP_62
Cross-reference integrity Broken citation/equation-label/section-parent detection (§7); orphaned-section detection. PAP_70–PAP_73
Property-based Randomized paragraph-edit generation across the edit-distance threshold boundary, asserting identity-preservation vs. delete+insert classification matches the declared threshold exactly. PAP_80
Validation/drift One test per remaining §7 bullet (missing bib entry informational, missing figure blocking-if-referenced, non-deterministic import, schema mismatch). PAP_90–PAP_93
Query One test per §6 example against a fixture with known results. PAP_100–PAP_108

Fixture strategy using representative papers

Fixtures are authored/adapted from real, permissively-licensed papers (e.g. arXiv preprints under CC-BY, since real LaTeX source exercises real macro packages and real author habits that synthetic fixtures would miss) — not generated programmatically for the golden/round-trip tiers, same "realistic tool output" reasoning as every prior plan. A minimal library: a short single-author IMRaD paper; a math-heavy multi-equation paper; a multi-author paper with CRediT contribution roles and multiple affiliations; a paper citing both DOI-bearing and DOI-less references; a paper with an intentionally broken equation cross-reference (for the validation-gate test); a Markdown-authored preprint (for the secondary-format path). Property-based tests use synthetic paragraph-edit generators, not the fixture library, for systematic edit-distance-threshold coverage.

9. Backend Integration

Domain plugin implementation

muse/plugins/paper/ following the muse/plugins/midi/ layout:

muse/plugins/paper/
  __init__.py
  plugin.py           # snapshot/diff/merge/drift/apply/schema
  entity.py           # Canonical model dataclasses (§2)
  manifest.py         # SnapshotManifest (de)serialization, normalization (§3)
  paper_diff.py       # Diff algorithms (§4)
  paper_merge.py      # Merge algorithms (§5)
  _query.py           # Query DSL (§6)
  _invariants.py      # Validation/drift checks (§7)
  _import_jats.py     # JATS <-> IR adapter (the core integration point)
  _import_latex.py    # LaTeX -> JATS -> IR pipeline (via latexml/Pandoc)
  _import_markdown.py # Markdown <-> IR direct path
  _import_pdf.py      # PDF -> IR lossy fallback (import-only)

schema() declares (mirroring every prior plan's domains --json shape):

{
  "schema_version": "<muse_version>",
  "merge_mode": "three_way",
  "description": "Semantic version control for scientific and academic papers...",
  "dimensions": [
    {"name": "structure", "description": "Sections, paragraphs, and document hierarchy."},
    {"name": "authorship", "description": "Authors, affiliations, and contribution roles."},
    {"name": "media", "description": "Figures, tables, equations, and their content-addressed assets."},
    {"name": "citations", "description": "In-text citations and the bibliography graph."},
    {"name": "argument", "description": "Claims, methods, experiments, and results."},
    {"name": "external_refs", "description": "Datasets, code repositories, and supplementary materials."},
    {"name": "metadata", "description": "Publication metadata and schema version."}
  ]
}

Storage implications

Figures/supplementary blobs use .muse/objects/ (small payloads, like the CAD/3D- Scenes plans); datasets are external references only (large payloads, like the Timeline plan). muse count-objects/gc/prune need no paper-specific changes.

API endpoints needed (MuseHub)

  • GET /:owner/:repo/paper/:ref/sections — section/claim/citation listing at a ref.
  • GET /:owner/:repo/paper/:ref/diff?base=&target= — structured diff for proposal review rendering.
  • GET /:owner/:repo/paper/:ref/citation-graph?depth=N — citation graph traversal.
  • GET /hub/search/papers?author=&institution=&keyword=&after= — cross-repo search endpoint (§6), new since this is the first domain plan requiring cross-repo query, not just within-repo query like the CAD/3D-Scenes/Timeline plans.

CLI implications

New muse paper subcommand group: muse paper query, muse paper claims, muse paper unsupported-claims, muse paper cited-by, muse paper find-equations, muse paper methods, muse paper datasets, muse paper citation-graph. All accept --json.

MuseHub UI implications

  • Proposal/diff view needs a paper diff renderer: rendered prose (not raw LaTeX) with changed paragraphs/claims/citations highlighted inline, a figure/table diff panel, and a conflict panel in plain-language editorial terms (§5).
  • Repo browse view needs a rendered-paper reading view (typeset prose, not source) with a section outline sidebar, keyed on the Section hierarchy (§2).

Rich paper visualization

  • Full rendered view (LaTeX-quality typesetting from the canonical IR, via existing IR→LaTeX→PDF or IR→HTML rendering) as the default "read this paper" experience — never raw source, mirroring how the CAD/3D-Scenes plans render geometry, not raw STEP/glTF text.
  • Claims panel: a sidebar listing every Claim with its status (supported/ unsupported) and a jump-to-evidence link — the paper-specific analog of the CAD plan's constraint-conflict panel.

Citation graph visualization

  • An interactive graph view (nodes = papers, edges = citations) rooted at the current paper, depth-configurable (§6), with unresolved/external nodes visually distinct from Muse-versioned ones — directly serves the "traverse citation graphs" query.

MCP/tooling implications

Expose paper_query, paper_diff, paper_citation_graph, paper_unsupported_claims as MCP tools, mirroring every prior plan's exposure pattern — lets agentception workers check "does this revision leave any claim unsupported" or "did this edit break a citation" before a merge or review completes.

10. Future Semantic Extensions

Named here so the v0 scope boundary (§1) is legible as a deliberate step toward a larger vision, not a ceiling — none of these are designed or scheduled in this plan:

  • Machine-readable claims: structured claim representations (subject-predicate- object triples, or a formal logic fragment) beyond v0's extracted-text-plus-status model, enabling actual entailment checking against evidence.
  • Evidence graphs: a full graph connecting claims to the specific data points, statistical tests, and prior citations that justify them, with confidence scoring — v0's Claim.supporting_refs is the seed of this, not the full realization.
  • Argument graphs: modeling the paper's rhetorical structure (premises, objections, rebuttals) as a first-class graph, not just linear section order.
  • Citation influence analysis: quantifying how a cited claim's later retraction, correction, or contradiction propagates forward through everything that cites it — requires the global citation-graph crawling explicitly deferred in §1.
  • Scientific provenance: full chain-of-custody from raw instrument/survey data through every transformation to the reported result, versioned end-to-end.
  • Reproducibility tracking: automated re-execution of Method/Experiment against Dataset/CodeRepository references to verify Results still hold — connects to muse code test-style automated verification, generalized to science.
  • Dataset lineage: versioning datasets themselves as a Muse domain (a natural extension of the CAD plan's content-addressing applied to tabular/scientific data), with papers referencing specific dataset versions, not just current-state URIs.
  • Peer review history: if MuseHub's generic proposal/review layer proves insufficient for scholarly review's specific needs (structured referee reports, editorial decision letters), a dedicated Review element may be promoted from "out of scope" (§1) to a first-class part of this domain.
  • Living papers: papers that are continuously updated as new evidence arrives, with full version history as the actual scholarly record rather than a series of disconnected "versions of record" — this domain's commit history is already a strong foundation for this once merge/diff are proven at v0.
  • Executable papers: embedding runnable code/live data bindings directly in the paper (results computed at render time, not pasted in as static figures) — the furthest extension, requiring integration with code/dataset-lineage domains.

11. Staging Deliverables

Proposed issue breakdown

  1. PAP-1 — Canonical model + snapshot (§2, §3): entity dataclasses, section/ paragraph identity (incl. pinning and edit-distance threshold), JATS import bridge, golden-snapshot tests (PAP_01–PAP_13). Load-bearing — nothing else starts until this lands.
  2. PAP-2 — LaTeX/Markdown/DOCX/PDF import + JATS export (§1, §3): depends on PAP-1. PAP_20–PAP_22.
  3. PAP-3 — Semantic diff (§4): depends on PAP-1. PAP_30–PAP_41.
  4. PAP-4 — Merge strategy (§5): depends on PAP-1, PAP-3. PAP_50–PAP_58.
  5. PAP-5 — Citation graph + cross-reference integrity (§6 graph portion, §7): depends on PAP-1; parallel with PAP-3/PAP-4. PAP_60–PAP_73.
  6. PAP-6 — Query DSL + CLI (§6, §9): depends on PAP-1, PAP-3. PAP_90–PAP_108.
  7. PAP-7 — Property-based paragraph-identity suite (§8): depends on PAP-1 only. PAP_80.
  8. PAP-8 — MuseHub API endpoints + cross-repo paper search (§9): depends on PAP-1, PAP-3.
  9. PAP-9 — MuseHub UI: paper reader, diff renderer, claims panel (§9): depends on PAP-8. Not scoped in detail until PAP-3/PAP-4 output shapes are proven.
  10. PAP-10 — Citation graph visualization (§9): depends on PAP-5, PAP-8.
  11. PAP-11 — MCP tooling exposure (§9): depends on PAP-6.

Milestones

  • M1 (Foundation): PAP-1 merged, muse domains --json lists paper as active, golden-snapshot suite green.
  • M2 (Interop): PAP-2 merged; LaTeX/Markdown round-trip fixtures pass at their fidelity ceilings.
  • M3 (Semantic core): PAP-3 + PAP-4 + PAP-5 merged; a two-branch conflict fixture (both branches edit the same paragraph) produces a correct conflict report; DOI- based bibliography auto-merge verified working as designed.
  • M4 (Usable): PAP-6 + PAP-7 merged; muse paper query/claims/ unsupported-claims/citation-graph all answer correctly against fixtures.
  • M5 (Ecosystem): PAP-8, PAP-9, PAP-10, PAP-11 merged; a paper proposal on staging MuseHub renders a real prose-level diff with a claims panel and citation graph view.

Acceptance criteria (whole-plan level)

  • muse domains --json shows paper with active: true and the schema in §9.
  • Every fixture round-trips through LaTeX/JATS/Markdown (§8) with zero diff against itself; DOCX produces a valid fidelity-reduced subset, not a divergent result.
  • Renaming a section produces exactly one RenameOp with no spurious diffs elsewhere — verifies the pinned-identity matching in §3.
  • The broken-equation-reference fixture is commit-blocking (§7); the missing-DOI fixture is informational only, with no false positives on well-formed fixtures.
  • A merge of disjoint section edits auto-resolves with zero conflicts; two branches independently citing the same DOI-bearing paper merge into one BibEntry, not a conflict; a merge where both branches edit the same paragraph, but with low text overlap, resolves as two independent paragraphs rather than a forced conflict.
  • muse paper query correctly answers all examples in §6 against fixtures.
  • No test depends on network access, a specific latexml/Pandoc version's exact output, or wall-clock time (§3/§7 determinism requirement).

Risks

  • Claim-extraction accuracy risk: v0's heuristic claim extraction (§1) will miss claims phrased unconventionally and may over-extract topic sentences that aren't really claims. Mitigation: Claim.status and extraction confidence are always visible, never presented as ground truth — the claims panel (§9) should read as "papers's structurally-detected claims," not an authoritative semantic audit.
  • Paragraph-identity threshold risk: the edit-distance boundary between "same paragraph, edited" and "different paragraph" (§3) is a single tunable value with real consequences for diff/merge readability; too strict and ordinary edits look like rewrites, too loose and genuine rewrites look like tiny edits. Needs real- fixture calibration, not a guessed default (open question below).
  • LaTeX macro fidelity risk: real LaTeX papers use enormous macro-package diversity; latexml/Pandoc conversion coverage is good but not universal. Mitigation: unsupported constructs are flagged as lossy import (§1, §7), never silently dropped, and the fixture library should include at least one paper with nonstandard macros to surface real gaps early.
  • Cross-repo search risk: §6/§9's cross-repo paper search is architecturally different from every prior domain plan (which only needed within-repo query) — this is new surface area for MuseHub's API layer, not just the domain plugin, and should be scoped carefully in PAP-8 rather than assumed to be a thin wrapper.

Open questions

  • What specific edit-distance threshold should separate "same paragraph, edited" from "different paragraph" (§3)? Needs calibration against the real-paper fixture library before PAP-1 is finalized, not guessed.
  • Should claim extraction (§1) support a pluggable/upgradable extraction backend from the start (heuristic v0, NLP-based later) via an explicit interface boundary, or is that premature abstraction until a v1 extraction approach is actually designed?
  • Where exactly does peer review history (§10) belong — a first-class domain element, or entirely on MuseHub's generic proposal/review layer? Flagged for a decision once real usage shows whether generic review metadata is sufficient.
  • Should dataset lineage (§10) be its own domain plugin (muse/plugins/dataset/) that this domain merely references, rather than being absorbed into the paper domain later? Leaning toward "own domain," consistent with this plan's own external-reference-not-ownership treatment of datasets in v0 — flagged for confirmation before that work starts.
  • How should cross-repo paper search (§9) handle privacy/visibility — does it respect existing repo-level access controls transparently, or does it need its own indexing/permission layer? Needs resolution before PAP-8 implementation.

Implementation order

PAP-1 → (PAP-2, PAP-3, and PAP-7 in parallel) → PAP-4 → PAP-5 → PAP-6 → PAP-8 → (PAP-9, PAP-10, and PAP-11 in parallel). Mirrors §1's dependency structure: nothing can diff, merge, validate, or query state that hasn't been normalized into the canonical model yet.

This is a plan-only issue. Each deliverable in §11 should be filed as its own staging issue once reviewed, referencing the test IDs and criteria above.

◎ Activity
●
gabriel opened this issue 19 days ago
◎
No activity yet. Use the CLI to comment.