gabriel / muse public
Open #70 Enhancement
filed by gabriel human · 19 days ago

Biological Pathways Domain — implementation plan (TDD-first, comprehensive)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Biological Pathways Domain — Implementation Plan

Background

Muse treats "domain" as a first-class primitive (muse/domain.py::MuseDomainPlugin). A plugin implements six methods — snapshot, diff, merge, drift, apply, schema — and gets the full VCS engine free: commit DAG, branching, checkout, lineage, log graph, merge-base finder. code/midi proved the model for symbolic data; the CAD (#66), 3D Scenes (#67), Video Timelines (#68), and Research Papers (#69) plans are proving it for geometry, spatial hierarchy, temporal edits, and scholarly argument structure respectively.

Biological pathways are a seventh shape of problem: a directed, typed, evidence- backed graph of molecular entities (genes, proteins, metabolites) connected by reactions and regulatory relationships, where every edge is a scientific claim that ought to be traceable to literature and experimental support — structurally close to the Research Papers plan's claim/evidence model, but graph-native rather than prose- native, and cross-referencing external biological databases (UniProt, ChEBI, Gene Ontology) the way the Research Papers plan cross-references DOIs. Today pathway knowledge lives in static diagrams, spreadsheets, or exchange files (SBML, BioPAX, KGML) that a generic VCS diffs as opaque XML — a single reaction's evidence updating looks identical to the entire pathway being rewritten. If Muse can version a pathway the way it versions code — diffing entities and reactions, not XML tags; merging disjoint regions of a regulatory network; validating graph connectivity and evidence provenance — it proves the domain-plugin model generalizes to networked biological knowledge. The goal is not to version pathway diagrams. It is to version the biology.

This issue is the plan only. Each numbered phase in §11 becomes its own staging issue once reviewed and accepted.

Goal

Ship a muse/plugins/pathway/ domain plugin, structured like muse/plugins/midi/ and the CAD/3D-Scenes/Timeline/Paper plans (a plugin.py entrypoint plus focused modules for diff/merge/query/invariants/import-export), that:

  • Registers in muse domains --json alongside code, midi, identity, mist, social, and (once landed) cad/scene3d/timeline/paper.
  • Passes muse code test --json for every module it touches.
  • Snapshots, diffs, and merges a pathway with entity/reaction/interaction-level semantic awareness — never as an opaque SBML/BioPAX blob.
  • Supports a query DSL (muse pathway query "...") answering the examples in §6, including graph traversal (upstream/downstream, shortest path, feedback loops).
  • Detects the drift/validation classes in §7 via muse status/muse check.
  • Has test coverage per §8, with test IDs (BIO_01, BIO_02, …) referenced from both this plan and the tests that satisfy them.

"Done" for the plan means: reviewed, decomposed into §11's breakdown, with acceptance criteria concrete enough for a different agent to implement test-first.

1. Domain Scope

Supported initial pathway types

Signaling and metabolic pathways at the scale of a single well-characterized network (e.g. MAPK/ERK signaling, glycolysis) — tens to low-hundreds of entities and reactions, not whole-genome interactomes or whole-cell models.

Two-layer strategy, consistent with every prior plan:

  1. Canonical semantic layer (Muse-owned): a JSON-serializable pathway IR (§2) that Muse snapshots, diffs, merges, and queries directly — entities and reactions as addressable graph nodes/edges, not markup.
  2. Interchange boundary: SBML (Systems Biology Markup Language, the dominant machine-readable standard for reaction networks, with mature parsing via libSBML) as the primary v0 interchange format for reaction/species-level detail, and BioPAX as the secondary interchange format for pathway-level and regulatory-relationship detail that SBML's reaction-centric model expresses more awkwardly (SBML models "what reacts with what," BioPAX models "what regulates what" and pathway membership more naturally). v0 imports/exports through both libSBML and pybiopax-style bridges into the same canonical IR, so the two formats become two views of one model rather than two competing sources of truth.

Relationship to SBML, BioPAX, SBGN, CellML, KGML, Reactome, KEGG, and Cytoscape

Format/tool Role
SBML v0 primary interchange format for reactions, species, compartments — mature libSBML bindings mean Muse does not write its own SBML parser.
BioPAX v0 secondary interchange format for regulatory relationships and pathway-level grouping, where SBML's model is a poor fit.
SBGN Systems Biology Graphical Notation — a visualization standard (standardized diagram glyphs), not a semantic model. v0 treats SBGN as an export-only rendering target for §9's visualization hooks, generated from the canonical IR, never an import source (it encodes layout, not additional biology beyond what SBML/BioPAX already capture).
CellML Focused on quantitative ODE-based models (continuous kinetics) rather than qualitative reaction/regulation graphs — out of scope for v0 (§1, below); a natural v1 target once quantitative kinetic parameters are added to the canonical model (§10).
KGML KEGG's native XML format — import-only in v0, since KEGG pathways are widely used reference pathways and a valuable fixture source, but KGML's proprietary layout-coupled structure makes it a lower-fidelity round-trip target than SBML/BioPAX, mirroring how the Research Papers plan treats DOCX.
Reactome A curated pathway database, not a format — Reactome pathways are distributed as SBML/BioPAX exports, so this domain reaches Reactome content transitively through those adapters, the same "integration point, not per-source adapter" leverage the Video Timelines plan gets from OTIO.
KEGG Both a database and (via KGML) a format — same transitive-access treatment as Reactome, through the KGML import adapter.
Cytoscape A visualization/analysis tool, not an interchange format — v0 treats Cytoscape's .cys/GraphML export as out of scope for import (Cytoscape session files carry visual layout state, not additional canonical biology) but the canonical IR's graph structure is designed to export cleanly to GraphML/Cytoscape.js JSON for downstream analysis in Cytoscape itself (§9's graph exploration hooks).

What is in scope for v0

  • Pathways as a named, bounded collection of entities and reactions/interactions (not an unbounded whole-organism graph).
  • Genes, proteins, RNAs, metabolites, and molecular complexes as first-class, ontology-mapped entities.
  • Reactions (stoichiometric transformations), regulatory interactions (activation/ inhibition), catalysis, binding, and transport events as distinct, typed edges.
  • Cellular compartments and each entity's localization within them.
  • Species (organism) metadata per pathway and per entity (a human vs. mouse ortholog of the same protein are distinct entities with an explicit ortholog relationship, not silently conflated).
  • Evidence and literature references attached to reactions/interactions, structurally parallel to the Research Papers plan's Citation/BibEntry model.
  • Ontology mappings to external identifiers: UniProt (proteins), ChEBI (metabolites/ small molecules), Gene Ontology (biological process/function/component), NCBI Gene (genes) — v0 stores and validates these mappings but does not itself host or crawl the referenced ontologies (see out-of-scope, below).

What is explicitly out of scope for v0

  • Quantitative kinetic modeling (rate laws, ODE parameters, flux balance analysis) — v0 is a qualitative graph of entities and typed relationships, not a simulatable dynamical system. CellML-style quantitative modeling is a named v1+ direction (§10), not attempted now.
  • Whole-genome interactome or whole-cell-model scale graphs — performance work at that scale is deferred until v0 is proven at single-pathway scale.
  • Hosting or crawling external ontologies/databases (UniProt, ChEBI, GO, NCBI Gene, Reactome, KEGG) — this domain stores references to external identifiers and validates their format (§7), but resolving/caching the referenced database records is explicitly out of scope, mirroring the Research Papers plan's stance on not crawling the global citation graph.
  • Multi-species comparative pathway alignment (automatically mapping one species' pathway onto another's orthologs) — named as a future direction (§10), not designed now; v0 supports species metadata per pathway/entity but not automated cross-species graph alignment.
  • Protein structural data (3D structure, folding) — a protein entity in this domain is a graph node with identity and function annotations, not a structural model (which would more naturally belong to a future CAD-adjacent molecular-structure domain, out of scope here).
  • Single-cell/spatial transcriptomics data integration — named in §10, not designed.

2. Canonical Model

All elements are content-addressed (sha256:<hex>) and referenced by ID, never positional index — same identity discipline as every prior plan — so graph traversal order never produces a spurious diff, and the model is a genuine graph (entities as nodes, reactions/interactions as typed edges) rather than a serialized list.

Element Fields Notes
Pathway id, name, description, entities: [BiologicalEntity.id], reactions: [Reaction.id], interactions: [RegulatoryInteraction.id], species_ref: Species.id, metadata The bounding container (§1) — a named, scoped subgraph, not the whole organism graph.
BiologicalEntity id, kind (gene|protein|rna|metabolite|complex), name, synonyms: [string], ontology_refs: [OntologyMapping.id], compartment_ref: Compartment.id, species_ref: Species.id Base type; kind-specific fields below extend rather than replace this.
Complex id, entity_ref: BiologicalEntity.id, member_refs: [BiologicalEntity.id], stoichiometry: {entity_id: count} A molecular complex is itself a BiologicalEntity (so it can participate in reactions like any entity) and has this record describing its composition — a two-level model mirroring the 3D Scenes plan's Node/Mesh split (identity vs. structural detail).
Compartment id, name, kind (cytoplasm|nucleus|mitochondrion|extracellular|membrane|other), parent_ref: Compartment.id \| null Nested (e.g. inner mitochondrial membrane within mitochondrion), same nesting discipline as the 3D Scenes plan's node hierarchy.
Reaction id, name, reactants: [{entity_ref, stoichiometry}], products: [{entity_ref, stoichiometry}], catalyst_refs: [BiologicalEntity.id], compartment_ref: Compartment.id, reversible: bool, evidence_refs: [Evidence.id] Stoichiometric transformation — the graph's primary "reaction" edge type, deliberately separate from regulatory interactions below.
RegulatoryInteraction id, kind (activation|inhibition|catalysis|binding|transport), source_ref: BiologicalEntity.id, target_ref: BiologicalEntity.id \| Reaction.id, mechanism \| null, evidence_refs: [Evidence.id] The graph's typed regulatory edge — distinct from Reaction because a regulatory relationship (A activates B) is not itself a stoichiometric transformation; target_ref can point at a Reaction (regulating a reaction's rate, qualitatively) or directly at an entity.
Evidence id, kind (experimental|computational|curated|inferred), literature_refs: [LiteratureReference.id], confidence (high|medium|low|predicted), description \| null Structurally parallel to the Research Papers plan's evidence-for-claims model, applied to graph edges instead of prose claims.
LiteratureReference id, doi \| null, pmid \| null, title, authors, year Content-addressed by DOI/PMID when present (globally stable), else by (title, first-author, year) — same fallback-identity pattern as the Research Papers plan's BibEntry.
OntologyMapping id, ontology (uniprot|chebi|go|ncbi_gene|other), identifier, term \| null, resolved: bool A reference to an external ontology term, not a cached copy of it (§1) — resolved tracks whether the identifier's format/existence was validated at import time.
Species id, taxon_id (NCBI Taxonomy ID), common_name, scientific_name Content-addressed by taxon_id — globally stable, dedups across pathways referencing the same organism.
ExperimentalAnnotation id, entity_or_reaction_ref, assay_type, conditions \| null, evidence_ref: Evidence.id Distinct from Evidence itself: this records how a piece of evidence was obtained (assay/experimental conditions), while Evidence records what it supports and with what confidence.
VersionMetadata curator \| null, source_format (sbml|biopax|kgml), source_database (reactome|kegg|custom|null), source_checksum, schema_version

Design invariant carried over from every prior plan: identity is content- and reference-based, never positional. The graph's edges (Reaction, RegulatoryInteraction) reference entities by ID; reordering a pathway's entity/reaction lists is never itself a diff — only a change to an element's own content or its *_ref fields is.

3. Snapshot Design

Deterministic pathway normalization

snapshot() walks the live state (an SBML or BioPAX document under state/, same pattern as every prior plan's format walk) and produces the canonical IR as the SnapshotManifest:

  1. Resolve every element to a stable ID (below).
  2. Canonicalize stoichiometry (integer ratios, not floats where the source expresses whole-number stoichiometry — avoids float-precision jitter across SBML re-exports, the same representational fix the Video Timelines plan applies to frame counts).
  3. Canonicalize ordering: reactants/products lists sorted by entity ID; evidence_refs/literature_refs sorted by ID — reaction direction (reactants vs. products) is semantically meaningful and preserved, but internal list order within each side is not.
  4. Compute each Pathway's entity/reaction/interaction counts and a graph-degree summary per entity at snapshot time — a cheap invariant used to short-circuit full-graph diffs (§4), mirroring every prior plan's pre-filter discipline.

Stable biological entity identity

  • Entities with an external ontology mapping (UniProt accession, ChEBI ID, NCBI Gene ID) are content-addressed by (ontology, identifier) — globally stable across pathways and papers, enabling cross-pathway entity dedup (the same protein appearing in two different signaling pathway repos resolves to compatible, though not automatically merged across repos, identity).
  • Entities without a resolved ontology mapping (common for less-characterized genes/ metabolites, or a curator's placeholder pending annotation) fall back to (name, species_ref, kind) — explicitly flagged as weaker identity (§7), same fallback-identity discipline as the Research Papers plan's DOI-less BibEntry.
  • Synonym handling: a BiologicalEntity.synonyms list absorbs alternate names (many genes/proteins have several); a rename is detected as "same ontology-mapped identity, name changed, old name moved to synonyms" — a RenameOp, not delete+insert, the same rename-vs-cascade discipline every prior plan applies.

Canonical graph representation

The pathway is stored and diffed as an actual graph structure (adjacency derivable from Reaction/RegulatoryInteraction *_ref fields), not a serialized node/edge list that happens to look graph-shaped — this matters concretely for §6's traversal queries (upstream/downstream, shortest path, feedback loops), which must be graph walks over committed snapshot data, not a re-derivation from XML on every query, the same "snapshot-time graph, not query-time inference" principle the Research Papers plan applies to its citation graph.

External database references

OntologyMapping/LiteratureReference external identifiers are validated for format at import time (does a UniProt accession match the expected pattern, does a DOI resolve to a well-formed identifier) but never fetched/cached — resolved tracks format validity, not live database resolution, consistent with §1's scope boundary against hosting external databases.

Ontology normalization

When an entity carries mappings to multiple ontologies (e.g. a gene with both an NCBI Gene ID and a UniProt accession for its protein product), each mapping is a distinct OntologyMapping rather than a single overloaded field — avoids forcing an artificial "primary identifier" choice that would bias identity resolution (§3, above) toward one ontology over another.

Evidence preservation

Evidence/LiteratureReference graphs attached to Reaction/RegulatoryInteraction edges are preserved as explicit snapshot data, not re-derived — mirrors the Research Papers plan's reference-graph-preservation principle, applied to graph edges instead of prose claims.

Deterministic serialization

SBML/BioPAX round-trips through libSBML/pybiopax are not guaranteed byte-stable across library versions — Muse never hashes SBML/BioPAX bytes directly for content- addressing; it always re-serializes through the canonical IR's fixed field order and ID scheme (§2), the same discipline as every prior plan's "never hash the interchange format's bytes directly" rule. A golden-snapshot suite (§8) locks this down.

4. Semantic Diff

diff(base, target, *, repo_root=None) returns a StructuredDelta, same contract as every prior plan.

  • Entity additions/removals: InsertOp/DeleteOp keyed on entity identity (§3); an entity gaining/losing an OntologyMapping (e.g. a placeholder gene getting annotated) is reported as "identity strengthened," distinct from an ordinary content change, since it may retroactively resolve prior weak-identity ambiguity.
  • Reaction modifications: ReplaceOp on reactants/products/stoichiometry; a stoichiometry-only change (e.g. 2 A + B → C becomes A + B → C) is reported distinctly from a participant swap (a different entity added/removed from the reaction) — different biological meanings, different review implications.
  • Interaction changes: RegulatoryInteraction.kind changing (e.g. a relationship previously modeled as activation reclassified as catalysis after further characterization) is reported as a reclassification, distinct from the interaction's source_ref/target_ref changing (a different biological claim entirely).
  • Regulatory network changes: additions/removals of RegulatoryInteraction edges reported with the affected entities named directly in the diff summary (not just IDs), so a reviewer sees "X now inhibits Y" in the diff text, not opaque identifiers.
  • Evidence updates: an Evidence.confidence change (e.g. predicted → experimental as new data arrives) is reported distinctly from a literature_refs addition/removal — confidence-level changes and citation-list changes are different kinds of update with different review weight.
  • Literature reference changes: InsertOp/DeleteOp on literature_refs, same distinction as the Research Papers plan draws between adding a citation and changing an existing one's resolved metadata.
  • Ontology mapping changes: an OntologyMapping.identifier changing (a correction, or a database re-annotation) is reported distinctly from a mapping being added/removed outright — the former is "this identifier was wrong," the latter is "we didn't know this before" or "we're removing this claim."
  • Compartment changes: an entity's compartment_ref changing (relocalization) is a ReplaceOp reported explicitly as "relocalized from X to Y," since compartment is often biologically load-bearing (e.g. a transcription factor's nuclear vs. cytoplasmic localization determines its activity).
  • Annotation changes: ExperimentalAnnotation field-level diffs, same granularity as every prior plan's small-element treatment (materials, effects).
  • Rename and synonym detection: per §3 — a rename is a single RenameOp when ontology-mapped identity is stable; when identity itself is weak (no ontology mapping) and a name changes, this is flagged as ambiguous (could be a rename or could be two different entities coincidentally similar) rather than silently assumed to be a rename — an explicit, tested boundary case (§8).
  • Topological graph diffs: beyond individual entity/edge diffs, a diff summary reports graph-level changes when they cross a threshold — a new feedback loop introduced, a previously-connected subgraph becoming disconnected — computed via the same graph-traversal machinery as §6's queries, surfaced in the diff output as a structural callout, not buried in a long list of individual edge changes.

5. Merge Strategy

merge(base, left, right, *, repo_root=None) — three-way, merge_mode: three_way, consistent with every prior plan.

  • Independent pathway edits (the common case): edits to disjoint entities/ reactions/interactions merge automatically via the engine's default address-keyed map merge, driven by schema()'s dimensions (§9) — one curator adds evidence to an existing reaction while another adds a new regulatory interaction elsewhere in the network.
  • Same-entity conflicts: both branches change the same entity's name or ontology_refs differently — reported conflict; but one branch adding a new OntologyMapping (a different ontology than any the other branch touched) and the other editing an existing mapping are independent fields and auto-merge, same field-level-independence principle as prior plans' material/effect merges.
  • Interaction conflicts: both branches change the same RegulatoryInteraction's kind or target_ref differently — reported conflict (no auto-resolution of competing biological claims, consistent with every prior plan's rejection of silently picking a value neither author chose).
  • Evidence conflicts: both branches set the same Evidence.confidence to different values — reported conflict; both branches independently adding different literature_refs to the same evidence record auto-merges (disjoint additions, union semantics) — an explicit exception to "same-element edits always conflict," justified the same way the Video Timelines plan's marker union-merge is: supporting citations for the same claim are almost always meant to coexist, not compete.
  • Annotation conflicts: same-field edits on the same ExperimentalAnnotation conflict; different-field edits auto-merge.
  • Ontology conflicts: both branches map the same entity to different identifiers within the same ontology (e.g. two different UniProt accessions) — reported conflict, since this indicates a genuine disagreement about entity identity, not a value dispute; both branches mapping to the same identifier independently (both correctly annotate to the same accession) resolves by construction, since ontology-mapped content addressing (§3) makes them the same object.
  • Literature conflicts: both branches edit the same LiteratureReference's metadata differently — conflict; both branches independently citing the same DOI/PMID auto-merges into one reference, same DOI-based dedup principle as the Research Papers plan's bibliography merge.
  • Species-specific conflicts: both branches change a Species.taxon_id-scoped entity's data assuming different species assumptions were correct (a genuine cross-species mismatch, not just concurrent edits) — flagged as a species- consistency conflict, escalated with extra context (both branches' declared species) rather than a generic field conflict, since resolving it requires biological judgment about which species assumption is correct for this pathway.
  • Human-readable merge reports: every conflict carries both branches' values, the base value, and a plain-language cause string in biological terms (e.g. "Both branches reclassified the MEK→ERK interaction: left changed kind 'activation' to 'catalysis', right changed kind 'activation' to 'binding' — base was 'activation'"), consistent with the workspace-wide conflict-marker convention.

6. Query Support

A query DSL mirroring muse code query and every prior plan's equivalent (muse pathway query "<predicate>" --json), plus purpose-built graph-traversal subcommands — this domain's query surface leans more heavily on traversal than any prior plan, since biological reasoning is fundamentally about paths through a regulatory network.

muse pathway query "entity.kind == 'protein' and entity.ontology == 'uniprot:P04049'" --json
muse pathway query "reaction.evidence.confidence == 'predicted'" --json
muse pathway query "interaction.kind == 'inhibition'" --json
  • Find upstream regulators: muse pathway upstream "<entity_id>" --depth 3 --json — walks RegulatoryInteraction edges backward from the target.
  • Find downstream targets: muse pathway downstream "<entity_id>" --depth 3 --json — forward walk, the mirror query.
  • Traverse signaling cascades: muse pathway cascade "<entity_id>" --json — a directed walk following activation/inhibition edges specifically (a named subset of the general downstream traversal, since signaling cascades are the most common biological question this graph answers).
  • Find shortest biological paths: muse pathway shortest-path <from> <to> --json — standard graph shortest-path over the combined reaction+interaction edge set.
  • Find feedback loops: muse pathway feedback-loops --json — cycle detection over the directed graph, explicitly distinguished from §7's inappropriate circular-dependency check (a feedback loop is often a real, intended biological feature — e.g. negative feedback in a signaling cascade — not a structural error; §7 flags cycles only where the specific edge types involved make a cycle biologically nonsensical, such as a stoichiometric reaction cycle with no mass balance).
  • Find orphaned entities: muse pathway orphans --json — entities with zero incident Reaction/RegulatoryInteraction edges (mirrors the CAD/3D-Scenes plans' dead-reference-style queries, applied to graph connectivity).
  • Find highly connected hubs: muse pathway hubs --top 10 --json — entities ranked by graph degree, mirroring muse code gravity's "blast radius" framing applied to biological centrality.
  • Find reactions involving a protein: muse pathway query "reaction.participants contains '<entity_id>'" --json
  • Find pathways containing a metabolite: this spans multiple pathway repos — muse hub search pathways --metabolite "<chebi_id>" --json, a MuseHub-level cross-repo search endpoint (§9), the same architectural pattern the Research Papers plan needed for cross-repo author/keyword search.
  • Find evidence supporting an interaction: muse pathway query "interaction.evidence_refs is not empty" --json / muse pathway evidence-for "<interaction_id>" --json
  • Query by ontology term: muse pathway query "entity.ontology_refs contains 'GO:0007173'" --json (EGFR signaling pathway GO term, as an example)
  • Cross-reference external biological databases: muse pathway query "entity.ontology == 'uniprot:P04049'" --json resolves format-validated references (§3) — v0 does not fetch live UniProt/ChEBI/GO records (§1), so this query returns the stored reference, not a live-fetched record; a --resolve-live flag that fetches from the external database at query time (not commit time) is a reasonable v1 CLI convenience, not designed in this plan.

7. Validation and Drift Detection

Surfaces through muse status (via drift()) and muse check/muse code invariants, same pattern as every prior plan's _invariants.py.

  • Broken graph connectivity: a Reaction/RegulatoryInteraction referencing a missing BiologicalEntity — commit-blocking (structurally invalid).
  • Invalid references: a compartment_ref/species_ref pointing at a nonexistent element — commit-blocking.
  • Missing ontology mappings: an entity with zero OntologyMapping entries — flagged informational, not blocking (mirrors the Research Papers plan's missing- DOI policy: many legitimately novel or poorly-characterized entities lack a mapping and that's a normal curation state, not an error).
  • Duplicate biological entities: two entities resolving to the same (ontology, identifier) that failed to dedup (an import bug, since ontology-based content addressing should prevent this by construction) — flagged as a snapshot determinism failure, hard error, same treatment as every prior plan's non-deterministic-import check.
  • Circular dependency detection where inappropriate: cycle detection scoped to edge types where a cycle is structurally nonsensical (e.g. a stoichiometric reaction chain implying infinite mass creation) — commit-blocking only for those specific cases; ordinary regulatory feedback loops are explicitly not flagged here (§6 surfaces them as a query result, not a validation failure) — this distinction must be implemented carefully and tested directly (§8), since conflating "any cycle" with "invalid cycle" would make this check useless for real signaling biology.
  • Inconsistent species annotations: an entity's species_ref disagreeing with its containing Pathway.species_ref without an explicit cross-species/ortholog annotation — flagged, not blocking (cross-species pathway comparisons are a legitimate use case, §10, so this is informational unless nothing explains the mismatch).
  • Missing evidence: a Reaction/RegulatoryInteraction with zero evidence_refs — flagged informational (a curator may model a well-known, textbook-level reaction without immediately attaching literature, and that's a normal intermediate curation state) — never commit-blocking, but visible so it can be tracked toward completion.
  • Invalid reaction definitions: a Reaction with no reactants and no products (a degenerate, meaningless reaction) — commit-blocking; a reaction with unbalanced stoichiometry where the source format declares strict mass balance is flagged (informational, since not all pathway models enforce balance — many qualitative signaling reactions have no meaningful stoichiometric balance at all).
  • Non-deterministic imports: re-importing the same source (SBML/BioPAX/KGML) twice must produce identical canonical IR — hard-error regression gate, same treatment as every prior plan.
  • Schema migration risks: VersionMetadata.schema_version mismatches flagged before diff/merge across the boundary, never auto-migrated silently, consistent with the workspace-wide muse code migrate permission rule.

8. TDD Plan

Test IDs use the BIO_NN prefix, per the workspace convention (CAD_NN, SCN_NN, VID_NN, PAP_NN).

Tier Coverage Example test IDs
Unit Canonical model (de)serialization for every element type; ontology-based vs. fallback entity identity; stoichiometry rational-integer normalization. BIO_01–BIO_03
Golden snapshot Fixed fixtures (a small linear signaling cascade, a metabolic pathway with balanced stoichiometry, a pathway with a regulatory feedback loop, a multi-compartment pathway with relocalization) snapshotted once, checked byte-for-byte against committed golden JSON. BIO_10–BIO_13
Round-trip import/export SBML→IR→SBML and BioPAX→IR→BioPAX must reproduce input within fidelity ceiling and produce identical IR on re-import; KGML→IR must produce a valid fidelity-reduced subset, not silently divergent. BIO_20–BIO_22
Graph integrity Connectivity/cycle-detection correctness on both the "inappropriate cycle" fixture (must block) and the "legitimate feedback loop" fixture (must not block, must surface via §6's query instead). BIO_30, BIO_31
Semantic diff One test per §4 category (entity, reaction, interaction reclassification, evidence, literature, ontology, compartment/relocalization, annotation, rename/synonym incl. the ambiguous-weak-identity case, topological graph diff). BIO_40–BIO_49
Merge One test per §5 case, incl. the evidence-literature-union auto-merge, ontology-based bibliography-style dedup, and species-consistency-conflict escalation. BIO_50–BIO_58
Traversal correctness Upstream/downstream/cascade/shortest-path/feedback-loop/hub queries against a fixture with known expected results, including depth-limited traversal boundary behavior. BIO_60–BIO_65
Query engine One test per remaining §6 example (orphans, reactions-involving, ontology-term query, cross-database reference resolution). BIO_70–BIO_73
Property-based Randomized graph generation asserting cycle-detection correctness holds across arbitrary topologies (no false positives on any generated feedback structure that the "inappropriate cycle" rule shouldn't catch), and that traversal queries never crash on disconnected/degenerate graphs. BIO_80, BIO_81
Validation/drift One test per remaining §7 bullet (missing ontology mapping informational, duplicate-entity hard error, species-mismatch informational, missing-evidence informational, invalid reaction blocking, non-deterministic import, schema mismatch). BIO_90–BIO_96

Fixture strategy using representative pathways

Fixtures are adapted from real, well-characterized reference pathways (e.g. a Reactome or KEGG export of MAPK/ERK signaling, and a core glycolysis pathway from a public SBML repository) — real exported SBML/BioPAX exercises real database export quirks that synthetic fixtures would miss, same "realistic tool output" reasoning as every prior plan. A minimal library: a short linear signaling cascade (simplest case); a metabolic pathway with balanced stoichiometry (exercises reaction/ compartment structure); a pathway with an intentional, legitimate regulatory feedback loop (must not be blocked by §7's cycle check); a multi-compartment pathway with an entity relocalization event between snapshots; a pathway with a deliberately invalid reaction (zero reactants/products, for the validation-gate test); a KGML export of one of the above (for the lower-fidelity cross-format test). Property-based tests use synthetic graph generators, not the fixture library, for systematic cycle/traversal coverage across topologies the curated fixtures don't include.

9. Backend Integration

Domain plugin implementation

muse/plugins/pathway/ following the muse/plugins/midi/ layout:

muse/plugins/pathway/
  __init__.py
  plugin.py            # snapshot/diff/merge/drift/apply/schema
  entity.py            # Canonical model dataclasses (§2)
  manifest.py          # SnapshotManifest (de)serialization, normalization (§3)
  pathway_diff.py       # Diff algorithms (§4)
  pathway_merge.py       # Merge algorithms (§5)
  graph.py              # Graph traversal primitives shared by diff/merge/query (§4, §6)
  _query.py             # Query DSL + traversal subcommands (§6)
  _invariants.py        # Validation/drift checks (§7)
  _import_sbml.py        # SBML <-> IR adapter (via libSBML)
  _import_biopax.py      # BioPAX <-> IR adapter (via pybiopax-style bridge)
  _import_kgml.py         # KGML -> IR adapter (import-only)
  _export_sbgn.py         # IR -> SBGN rendering (export-only, visualization)

schema() declares (mirroring every prior plan's domains --json shape):

{
  "schema_version": "<muse_version>",
  "merge_mode": "three_way",
  "description": "Semantic version control for biological pathways...",
  "dimensions": [
    {"name": "entities", "description": "Genes, proteins, RNAs, metabolites, and complexes."},
    {"name": "structure", "description": "Compartments and cellular localization."},
    {"name": "reactions", "description": "Stoichiometric transformations."},
    {"name": "regulation", "description": "Activation, inhibition, catalysis, binding, transport edges."},
    {"name": "evidence", "description": "Evidence, literature references, and experimental annotations."},
    {"name": "ontology", "description": "External ontology and database identifier mappings."},
    {"name": "metadata", "description": "Species, curator, source format, schema version."}
  ]
}

Storage implications

This domain's snapshots are small, structured graph data — no large binary payloads at all (unlike CAD/3D-Scenes' geometry blobs or the Timeline plan's media references). .muse/objects/ storage needs are minimal; muse count-objects/gc/ prune need no pathway-specific changes.

API endpoints needed (MuseHub)

  • GET /:owner/:repo/pathway/:ref/graph — full entity/reaction/interaction graph at a ref, for visualization (§9, below).
  • GET /:owner/:repo/pathway/:ref/diff?base=&target= — structured diff for proposal review rendering, including the topological-graph-diff callout (§4).
  • GET /:owner/:repo/pathway/:ref/traverse?from=&kind=&depth= — server-side graph traversal (upstream/downstream/cascade/shortest-path), backing §6's CLI commands for UI use.
  • GET /hub/search/pathways?metabolite=&protein=&ontology_term= — cross-repo search endpoint (§6), same architectural pattern the Research Papers plan required for cross-repo author/keyword search.

CLI implications

New muse pathway subcommand group: muse pathway query, muse pathway upstream, muse pathway downstream, muse pathway cascade, muse pathway shortest-path, muse pathway feedback-loops, muse pathway orphans, muse pathway hubs, muse pathway evidence-for. All accept --json.

MuseHub UI implications

  • Proposal/diff view needs a pathway diff renderer: an interactive graph view with changed entities/reactions/interactions highlighted, and a conflict panel in plain biological language (§5) — the graph-native analog of the CAD plan's feature-tree diff and the Research Papers plan's prose diff.
  • Repo browse view needs a graph-exploration view (pan/zoom/filter by entity kind or compartment) as the default "read this pathway" experience, not a raw SBML dump.

Interactive pathway visualization

  • SBGN-standard rendering (§1) generated from the canonical IR as the default visual representation — standardized glyphs (activation arrows, inhibition bars, etc.) that biologists already read fluently, rather than a bespoke node/edge style.
  • Compartment-aware layout: entities rendered nested within their Compartment boundaries, mirroring how the 3D Scenes plan's viewer respects node hierarchy.

Graph exploration

  • Click-to-traverse: selecting an entity in the UI surfaces its upstream regulators and downstream targets inline (directly backed by the /traverse endpoint above), making §6's CLI queries available as an interactive exploration affordance, not just a scriptable API.
  • Export to Cytoscape.js JSON / GraphML for users who want to continue analysis in Cytoscape itself (§1) — a one-way export, not a round-trip.

MCP/tooling implications

Expose pathway_query, pathway_upstream, pathway_downstream, pathway_shortest_path, pathway_diff as MCP tools, mirroring every prior plan's exposure pattern — lets agentception workers check "does this revision disconnect any part of the network" or "what regulates this target" before a merge or review completes.

10. Future Semantic Extensions

Named here so the v0 scope boundary (§1) is legible as a deliberate step, not a ceiling — none of these are designed or scheduled in this plan:

  • Gene regulatory networks: a dedicated transcriptional-regulation graph layer (transcription factors, binding sites, expression levels) — v0's RegulatoryInteraction is the seed, not the full realization.
  • Metabolic flux models: quantitative flux-balance analysis over the stoichiometric graph, requiring the kinetic parameters explicitly deferred in §1.
  • Protein interaction networks: dedicated physical-interaction graphs (PPI) at a scale and density different from curated signaling pathways — likely warrants its own domain scoping once this one is proven, not folded in prematurely.
  • Cell signaling simulations: executing the qualitative graph as an actual dynamical model (requires CellML-style quantitative extension, §1).
  • Temporal pathway evolution: modeling how a pathway's topology itself changes over developmental or evolutionary time, distinct from this domain's commit history (which tracks curation history, not biological time).
  • Comparative pathways across species: automated ortholog-based pathway alignment across Species — v0 supports species metadata but not automated cross-species graph alignment (§1).
  • Experimental provenance: full chain-of-custody from raw assay data through every analysis step to a reported interaction — parallels the Research Papers plan's scientific-provenance direction, applied to pathway evidence specifically.
  • AI-assisted pathway construction: literature-mining-assisted entity/interaction extraction (structurally similar to the Research Papers plan's claim-extraction direction) to accelerate curation — v0's evidence model is designed to make machine-suggested edges auditable (via Evidence.kind == 'inferred') from day one, even though the extraction pipeline itself is not built now.
  • Multi-omics integration: linking pathway entities to transcriptomic/proteomic/ metabolomic experimental datasets at scale, beyond the single-Evidence-record model in v0.
  • Digital twin support: a fully simulatable, patient- or condition-specific instance of a pathway network — the furthest extension, requiring the quantitative modeling layer (§1, this section) as a prerequisite.

11. Staging Deliverables

Proposed issue breakdown

  1. BIO-1 — Canonical model + snapshot + graph representation (§2, §3): entity dataclasses, ontology-based identity, canonical graph structure, SBML import bridge, golden-snapshot tests (BIO_01–BIO_13). Load-bearing — nothing else starts until this lands.
  2. BIO-2 — BioPAX/KGML import + SBML/BioPAX export (§1, §3): depends on BIO-1. BIO_20–BIO_22.
  3. BIO-3 — Graph integrity (cycle detection, connectivity) (§7 graph portion): depends on BIO-1. BIO_30, BIO_31.
  4. BIO-4 — Semantic diff (§4): depends on BIO-1, BIO-3. BIO_40–BIO_49.
  5. BIO-5 — Merge strategy (§5): depends on BIO-1, BIO-4. BIO_50–BIO_58.
  6. BIO-6 — Graph traversal + query DSL + CLI (§6, §9): depends on BIO-1, BIO-3. BIO_60–BIO_73.
  7. BIO-7 — Property-based graph-invariant suite (§8): depends on BIO-1, BIO-3. BIO_80, BIO_81.
  8. BIO-8 — Remaining validation & drift checks (§7): depends on BIO-1; parallel with BIO-4/BIO-5/BIO-6. BIO_90–BIO_96.
  9. BIO-9 — MuseHub API endpoints + cross-repo pathway search (§9): depends on BIO-1, BIO-4, BIO-6.
  10. BIO-10 — MuseHub UI: graph explorer, diff renderer, SBGN rendering (§9): depends on BIO-9. Not scoped in detail until BIO-4/BIO-5 output shapes are proven.
  11. BIO-11 — MCP tooling exposure (§9): depends on BIO-6.

Milestones

  • M1 (Foundation): BIO-1 merged, muse domains --json lists pathway as active, golden-snapshot suite green, graph structure verified against the feedback- loop fixture without false-blocking.
  • M2 (Interop): BIO-2 merged; SBML/BioPAX round-trip fixtures pass at their fidelity ceilings.
  • M3 (Semantic core): BIO-3 + BIO-4 + BIO-5 merged; a two-branch conflict fixture (both branches reclassify the same interaction differently) produces a correct conflict report; evidence-literature union-merge verified working as designed.
  • M4 (Usable): BIO-6 + BIO-7 + BIO-8 merged; muse pathway upstream/ downstream/cascade/shortest-path/feedback-loops/hubs all answer correctly against fixtures.
  • M5 (Ecosystem): BIO-9, BIO-10, BIO-11 merged; a pathway proposal on staging MuseHub renders a real graph-level diff with an interactive traversal-capable graph explorer.

Acceptance criteria (whole-plan level)

  • muse domains --json shows pathway with active: true and the schema in §9.
  • Every fixture round-trips through SBML/BioPAX (§8) with zero diff against itself; KGML produces a valid fidelity-reduced subset, not a divergent result.
  • The legitimate-feedback-loop fixture is not flagged by §7's cycle check; a deliberately invalid stoichiometric cycle fixture is — verifies the appropriate-vs-inappropriate cycle distinction (§7) is implemented correctly, not merely present.
  • Renaming an ontology-mapped entity produces exactly one RenameOp with no spurious diffs elsewhere; the same rename attempted on a weak-identity (no ontology mapping) entity is flagged as ambiguous, not silently treated as a confident rename.
  • A merge of disjoint entity/reaction/interaction edits auto-resolves with zero conflicts; two branches independently citing the same DOI/PMID merge into one LiteratureReference; two branches adding different literature to the same evidence record auto-merge (union semantics).
  • muse pathway upstream/downstream/shortest-path/feedback-loops correctly answer against fixtures with known expected results, including depth-limit boundary behavior.
  • No test depends on network access, a specific libSBML/pybiopax version's exact output, or wall-clock time (§3/§7 determinism requirement).

Risks

  • Cycle-detection scoping risk: §7's "inappropriate cycle" rule needs a precise, testable definition of which edge-type combinations make a cycle biologically invalid vs. a legitimate feedback loop — getting this wrong (too permissive misses real errors; too strict blocks normal signaling biology) is the highest-risk decision in this plan. Mitigation: BIO-3's fixtures include both a must-block and a must-not-block cycle case, required to pass before BIO-4 starts.
  • Weak-identity ambiguity risk: entities without ontology mappings make rename detection genuinely ambiguous (§4, §8) — a real epistemic limitation, not a bug; the UI/CLI must present ambiguous renames as requiring human judgment, never silently resolved either way.
  • SBML/BioPAX dual-source-of-truth risk: importing formats that model overlapping biology differently (§1) risks two divergent IRs for the same pathway if the bridge logic isn't reconciled. Mitigation: BIO-2's round-trip tests include a same-pathway-via-both-formats equivalence check, mirroring the CAD/3D-Scenes plans' cross-format equivalence tests.
  • Cross-repo search risk: same architectural risk class the Research Papers plan flagged — pathway search-by-metabolite/protein across repos is new MuseHub API surface, not just a domain-plugin concern, and should be scoped carefully in BIO-9 rather than assumed to be a thin wrapper.

Open questions

  • What precise rule distinguishes an "inappropriate" cycle from a legitimate feedback loop (§7)? This plan states the principle (mass-balance-violating stoichiometric cycles vs. qualitative regulatory feedback) but the exact implementable predicate needs to be nailed down during BIO-3, ideally validated against several real feedback-loop-containing reference pathways before finalizing.
  • Should weak-identity entities (no ontology mapping) support a curator-assigned stable ID as an escape hatch, distinct from both ontology-based and name-based identity — giving curators a way to pin identity for not-yet-annotated entities without waiting for database annotation? Flagged for a decision before BIO-1 is finalized.
  • Where should the SBML/BioPAX reconciliation logic (§1, Risks) live — a shared bridge module both adapters call into, or should one format be treated as primary and the other as a lossy secondary view? Needs a decision before BIO-2 starts.
  • Should cross-species ortholog relationships (§1, §10) be modeled now as a minimal explicit edge type (even though full comparative-pathway alignment is deferred), so §7's species-consistency check has a real mechanism to check against rather than only ever being informational? Worth deciding before BIO-1, since adding it later would be a schema migration.
  • How should cross-repo pathway search (§9) handle privacy/visibility, mirroring the same open question the Research Papers plan raised for its cross-repo search? Likely warrants one shared answer across both domains rather than two independent designs.

Implementation order

BIO-1 → BIO-3 → (BIO-2, BIO-4, and BIO-7 in parallel) → BIO-5 → BIO-6 → BIO-8 → BIO-9 → (BIO-10 and BIO-11 in parallel). Graph integrity (BIO-3) is pulled forward ahead of diff/merge since both depend on the same traversal primitives it establishes.

Plan-only issue. Each deliverable in §11 should be filed as its own staging issue once reviewed, referencing the test IDs and criteria above.

◎ Activity
●
gabriel opened this issue 19 days ago
◎
No activity yet. Use the CLI to comment.