gabriel / muse public
Open #72 Enhancement
filed by gabriel human · 19 days ago

Knowledge Graphs Domain — implementation plan (TDD-first, comprehensive)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Knowledge Graphs Domain — Implementation Plan

Background

Muse treats "domain" as a first-class primitive (muse/domain.py::MuseDomainPlugin). A plugin implements six methods — snapshot, diff, merge, drift, apply, schema — and gets the full VCS engine free: commit DAG, branching, checkout, lineage, log graph, merge-base finder. The CAD (#66), 3D Scenes (#67), Video Timelines (#68), Research Papers (#69), Biological Pathways (#70), and Datasets (#71) plans are proving the model across geometry, spatial hierarchy, temporal edits, scholarly argument, biological graphs, and tabular data respectively.

Knowledge graphs are the ninth shape of problem, and the most general: a set of typed entities connected by typed, provenance-bearing relationships, governed by an ontology that constrains what relationships are even meaningful, where identity is usually anchored by globally unique IRIs rather than content hashing invented for the occasion. This domain is structurally the closest to Biological Pathways (#70) — both are graphs of entities and typed edges with ontology mappings and evidence — but where Pathways is a specific, biologically-scoped graph shape, this domain is the general-purpose semantic graph substrate: arbitrary entities, arbitrary relationship types, an evolving ontology, and (eventually) logical entailment over both. Today knowledge graphs live in RDF dumps, JSON-LD documents, or graph-database exports that a generic VCS diffs as opaque serialized text — reordering triples in an RDF/XML file, or reserializing blank node IDs, produces a byte-diff with zero semantic content. If Muse can version a knowledge graph the way it versions code — diffing assertions, not serialized bytes; merging disjoint regions of a graph; validating ontology consistency and provenance — it proves the domain-plugin model generalizes to open-ended, ontology-governed semantic knowledge, not just the domain-specific graphs prior plans cover. The goal is not to version RDF files. It is to version meaning, relationships, ontologies, provenance, and evolving knowledge itself.

This issue is the plan only. Each numbered phase in §11 becomes its own staging issue once reviewed and accepted.

Goal

Ship a muse/plugins/kg/ domain plugin, structured like muse/plugins/midi/ and the CAD/3D-Scenes/Timeline/Paper/Pathway/Dataset plans, that:

  • Registers in muse domains --json alongside code, midi, identity, mist, social, and (once landed) cad/scene3d/timeline/paper/pathway/dataset.
  • Passes muse code test --json for every module it touches.
  • Versions the RDF graph/dataset itself, never a particular serialization — Turtle, N-Triples, RDF/XML, and JSON-LD are import/export surfaces only, and must never influence snapshot identity, semantic diffs, merges, or history (§2's "Canonical RDF Representation" makes this a normative requirement, not a preference).
  • Snapshots, diffs, and merges a knowledge graph with entity/relationship/ontology- level semantic awareness — never as an opaque RDF/JSON-LD blob.
  • Supports a SPARQL-inspired query DSL (muse kg query "...") answering the examples in §6, including graph traversal.
  • Detects the drift/validation classes in §7 via muse status/muse check.
  • Has test coverage per §8, with test IDs (KG_01, KG_02, …) referenced from both this plan and the tests that satisfy them.

"Done" for the plan means: reviewed, decomposed into §11's breakdown, with acceptance criteria concrete enough for a different agent to implement test-first.

1. Domain Scope

Supported initial graph shapes

RDF-model graphs (subject–predicate–object triples, in the RDF sense, regardless of serialization) bounded to a single named graph per pathway-scale scope — tens of thousands of triples, not web-scale linked-data crawls. Property-graph exports (Neo4j-style labeled nodes/edges with properties) are supported via a lossless mapping into the same triple-based canonical model (§2), not as a second parallel model.

Two-layer strategy, consistent with every prior plan:

  1. Canonical semantic layer (Muse-owned): a JSON-serializable graph IR (§2) — entities, relationships, and ontology terms as addressable elements, not serialized triples in a text stream. Muse snapshots, diffs, merges, and queries this directly.
  2. Interchange boundary: RDF (Turtle/N-Triples as the canonical textual forms, via rdflib) as the primary v0 interchange format, since it is the open, W3C-standardized model this entire domain is built on, and JSON-LD as a secondary interchange format for web/API-facing round-trips (JSON-LD is RDF with a JSON serialization, so both formats target the same RDF data model — one rdflib-backed bridge serves both, the same "one integration point reaches multiple formats" leverage the Video Timelines plan gets from OTIO).

Relationship to RDF, OWL, SHACL, JSON-LD, Neo4j/property graphs, SPARQL, Wikidata/DBpedia-style graphs

Format/tool Role
RDF (Turtle/N-Triples/RDF-XML) v0 primary interchange format and data model — the canonical IR (§2) is a structured view of what RDF already expresses (triples + typed literals + IRIs), via rdflib.
OWL Ontology language layered on RDF — v0 imports class/property axioms (subclass, domain/range, functional/inverse-functional) as first-class elements (§2) but does not run an OWL-DL reasoner (§1).
SHACL v0's QualityShape (§2) is SHACL-inspired (target class + property constraints); a v1 SHACL adapter is low-risk once proven, not designed now.
JSON-LD Secondary interchange format — same RDF data model as Turtle, reached through the same rdflib bridge.
Neo4j / property graphs Import-only, via a lossless labeled-property-graph→RDF mapping — the weakest transitive format, since edge-attached properties need reification (§2) to express faithfully in RDF.
SPARQL v0's own DSL (§6) is SPARQL-inspired (triple-pattern matching) without committing to full SPARQL algebra.
Wikidata/DBpedia-style graphs Not a format — a scale/provenance model. v0's Assertion/Evidence split (§2) is inspired by Wikidata's statement model, scaled to a single-repo, bounded graph.

What is in scope for v0

  • Entities as typed, IRI-anchored nodes (or blank-node-anchored, with explicit weaker-identity handling — see §3) with rdf:type class membership.
  • Relationships as typed, directed edges (predicate-anchored, matching RDF's triple model) between entities, or between an entity and a typed literal value.
  • An evolving ontology: classes, properties, and basic OWL-style axioms (subclass-of, domain/range, functional/inverse-functional) as first-class, versioned elements — the ontology is part of the graph's committed state, not an external, unversioned assumption.
  • Assertions with provenance: an assertion is a specific triple instance, distinct from the relationship type it uses, carrying its own evidence/source — the same statement-vs.-schema distinction Wikidata draws, letting two different sources assert the same relationship type between the same entities with independently tracked provenance.
  • Named graphs / contexts as a grouping mechanism (an RDF quad's fourth element) for scoping assertions to a source or a versioned sub-collection within one repo.
  • Basic structural validation against SHACL-inspired shapes (§2's QualityShape) — required properties, cardinality, type constraints — without full SHACL conformance in v0.
  • A lightweight, opt-in RDFS entailment closure (subclass/subproperty transitive closure, domain/range inference) computed at query time, not stored as committed data — see §1 out-of-scope for the reasoning boundary.

What is explicitly out of scope for v0

  • Full OWL-DL/OWL 2 reasoning (satisfiability checking, full class-expression entailment) — v0 validates basic axiom structure (§7) but does not run a description-logic reasoner. Named as a future extension (§10), not attempted now.
  • Full SPARQL 1.1 (aggregates, property paths beyond simple traversal, federated SERVICE queries, subqueries) — v0's query DSL (§6) is a deliberately smaller, SPARQL-inspired subset.
  • Web-scale/multi-million-triple graphs and any performance work targeting that scale — v0 targets single-pathway-scale graphs (§1, above), consistent with every prior plan's scale boundary.
  • Federated/distributed knowledge graphs spanning multiple repos as one logical graph at commit time — cross-repo query is supported (§6, mirroring the Research Papers/Pathways/Datasets plans' cross-repo search), but this domain does not merge or version multiple repos' graphs as a single federated commit history.
  • Automatic ontology alignment (mapping two independently-authored ontologies onto each other) — named as a future extension (§10).
  • Literal-value full-text search / embedding-based semantic similarity search over entity descriptions — v0's query DSL is exact-match/pattern-based (§6), not embedding-based; named as a future extension (§10).
  • SHACL language conformance (full shape validation with all SHACL constraint components) — v0's QualityShape is inspired by SHACL's model but implements a practical subset (§1, above), not the full spec.

2. Canonical Model

Canonical RDF Representation — the graph, not the serialization

The canonical Muse Knowledge Graph state MUST be derived from the RDF graph/ dataset itself, never from any particular serialization. Turtle, N-Triples, RDF/XML, and JSON-LD are import/export surfaces only — how bytes enter or leave a Muse repo, nothing more. None may influence snapshot identity, semantic diffs, merges, or commit history — the same posture every prior plan already takes toward its own interchange formats (never hash the interchange bytes directly, §3), stated explicitly here because this domain's formats are the worst offenders for serialization non-determinism of any prior plan (RDF serializers routinely reorder triples and relabel blank nodes on every write, §3).

Concretely: two RDF documents asserting exactly the same facts, in different triple order, in different serializations (Turtle vs. RDF/XML), with different blank-node labels for identical anonymous structures, MUST produce byte-identical canonical snapshots and zero semantic diff against each other. Any implementation that lets re-serialization or reordering leak into a snapshot, diff, or merge is non-conforming, not merely suboptimal — verified directly by KG_04/KG_20–KG_22 (§8), not an aspiration.

The versioned semantic unit is a normalized RDF assertion: subject, predicate, object, datatype/language tag where applicable, graph/context where applicable, and provenance/evidence/statement metadata where available. snapshot() (§3) normalizes every equivalent RDF representation into this deterministic canonical form before diff/merge/query ever run — they never operate on serialized text or a serializer's incidental choices, only on this normalized form. Assertion (below) is this normalized unit; every other element (Entity, OntologyClass, NamedGraph, Evidence) exists to give its fields addressable, versioned structure — not to describe serialization concerns.

Triples vs. assertions/quads

A plain RDF triple (subject, predicate, object) is too small once Muse needs provenance, named graphs, sources, evidence, or statement metadata — none of which a bare triple can carry. Whenever graph context or provenance is present (the common case here), the versioned semantic unit is a normalized assertion — effectively a quad: subject, predicate, object, graph/context. Assertion models exactly this: subject_ref/relationship_ref/object_ref-or- literal_value are the triple, named_graph_ref is the fourth, quad-completing element, and evidence_refs carries the provenance a bare quad still can't express (extending further where Wikidata-style statement qualification is needed, via Evidence below). This is what lets Muse report:

This assertion changed — :France :hasCapital changed from :Paris to :Lyon in named graph :2024-import.

rather than:

Line 47 of data.ttl changed.

Every diff/merge/query in §4–§6 operates on the former. The latter never appears anywhere in this domain's committable output — exactly the failure mode a generic, format-blind VCS produces today, and exactly what this plan exists to eliminate.

All elements are content-addressed (sha256:<hex>) and referenced by ID, never positional index — same identity discipline as every prior plan — with one domain-specific strengthening: most entities also carry a globally unique IRI, which is a stronger identity anchor than any prior plan's content hash, since it's externally coined and stable by convention across the entire linked-data web, not just within this repo.

Element Fields Notes
Entity id, iri \| null, blank_node_label \| null (mutually exclusive with iri — see §3), types: [OntologyClass.id], label \| null, literal_properties: [{predicate_ref, value, datatype}] An IRI-anchored entity's id is derived from iri directly (globally stable); a blank node's id is scoped to this snapshot only (§3).
OntologyClass id, iri, label, subclass_of: [OntologyClass.id], description \| null Classes are always IRI-anchored (v0 does not support anonymous/blank-node classes) — a deliberate simplification, since ontology terms are meant to be shared/stable by nature.
OntologyProperty id, iri, label, domain_ref: OntologyClass.id \| null, range_ref: OntologyClass.id \| null, functional: bool, inverse_functional: bool, inverse_of_ref: OntologyProperty.id \| null The subset of OWL property axioms v0 stores and validates structurally (§7) without full reasoning (§1).
Relationship id, predicate_ref: OntologyProperty.id, is_type_definition: bool The relationship type as used in this graph — distinct from any specific instance of it (Assertion, below), mirroring the Biological Pathways plan's Reaction-vs.-instance-data split and the Research Papers plan's Citation-vs.-BibEntry split.
Assertion id, subject_ref: Entity.id, relationship_ref: Relationship.id, object_ref: Entity.id \| null, literal_value \| null (mutually exclusive with object_ref), named_graph_ref: NamedGraph.id \| null, evidence_refs: [Evidence.id] The actual triple instance — subject/predicate/object plus provenance, the domain's primary versioned unit, structurally parallel to Wikidata's "statement."
Evidence id, kind (asserted|inferred|imported|curated), source_ref: LiteratureReference.id \| DataSource.id \| null, confidence (high|medium|low|disputed) inferred marks an assertion produced by the opt-in RDFS closure (§1) — inferred assertions are always distinguishable from asserted ones, never silently conflated, the same "never present a guess as ground truth" discipline the Research Papers plan applies to extracted claims.
NamedGraph id, iri \| null, description, source_ref: DataSource.id \| null Groups assertions by provenance/source, the RDF quad model's fourth element promoted to a first-class grouping element.
QualityShape id, name, target_class_ref: OntologyClass.id, constraints: [{property_ref, min_count, max_count, datatype \| class_ref}] SHACL-inspired (§1), not SHACL-conformant — a practical validation subset.
QualityCheckResult id, shape_ref: QualityShape.id, entity_ref: Entity.id, passed: bool, violations: [string], evaluated_at_content_id Same rule-vs.-result split as the Datasets plan's QualityRule/QualityCheckResult, applied to graph shape conformance instead of column constraints.
DataSource id, uri_or_path, content_id_at_link_time \| null, format (turtle|ntriples|rdfxml|jsonld|property_graph), resolved: bool External reference to an imported source, same pattern as every prior plan's DataSource/ExternalReference.
LiteratureReference id, doi \| null, title, authors, year Reused structurally from the Research Papers plan's BibEntry model — a knowledge graph assertion's evidence is frequently a citation, and this domain should not invent a second, incompatible citation model.
VersionMetadata curator \| null, source_format, source_checksum, schema_version

Design invariant carried over from every prior plan: identity is content- and reference-based, never positional. Triple serialization order in any RDF document is never semantically meaningful and never itself produces a diff.

3. Snapshot Design

Deterministic graph normalization

snapshot() reads the live state (an RDF/JSON-LD document or property-graph export under state/, via rdflib, same "walk the format, produce the canonical IR" pattern as every prior plan):

  1. Resolve every element to a stable ID (below).
  2. Canonicalize literal datatypes (an untyped RDF literal is normalized to xsd:string explicitly, never left ambiguous — avoids the classic RDF footgun where two triples that "look the same" differ only in an implicit vs. explicit datatype).
  3. Canonicalize ordering: Assertions are sorted by (subject_ref, relationship_ref, object_ref) for serialization — triple order in the source document is a serialization artifact, never semantic content, mirroring every prior plan's "reordering is not a diff" discipline, applied here to the domain's single most notorious source of spurious diffs (RDF serializers routinely reorder triples and renumber blank nodes on every write).
  4. Compute per-OntologyClass instance counts and per-entity degree at snapshot time — a cheap invariant used to short-circuit full-graph diffs (§4), same pre-filter discipline as every prior plan.

Stable entity identity

  • IRI-anchored entities: id is derived directly from the IRI — globally stable, the strongest identity anchor of any domain plan so far, since it doesn't even need Muse's own content-hashing scheme to survive re-serialization; two different source documents asserting facts about the same IRI are unambiguously about the same entity.
  • Blank nodes: RDF blank nodes have no stable identity across documents or even across re-serializations of the same document by different tools (this domain's single hardest identity problem, harder than the Datasets plan's column-rename ambiguity, since blank nodes have no name to hint at continuity). v0's resolution: a blank node's id is derived from a content hash of its local subgraph (its own literal_properties plus the set of (predicate, object-identity) pairs it participates in) — stable within one snapshot, and across snapshots only if that local subgraph is unchanged. A blank node whose local subgraph changes at all is, by construction, treated as a different entity (delete+insert), never "same entity, edited" — an explicit, honest limitation (not a bug), documented prominently and tested directly (§8), not discovered by surprise. Skolemization (converting blank nodes to fresh, deterministic IRIs at import time) is the recommended mitigation, flagged as an open question (§11) for whether v0 should apply it by default.
  • Ontology classes/properties: always IRI-anchored (§2) — no blank-node-class ambiguity to resolve, by deliberate scope restriction.

Canonical graph representation

The graph is stored and diffed as an actual traversable structure (adjacency derivable from Assertion.subject_ref/object_ref), not a serialized triple list that happens to look graph-shaped — same "snapshot-time graph, not query-time inference" principle as the Biological Pathways and Research Papers plans, enabling §6's traversal queries to be direct graph walks over committed data.

External database references

DataSource.content_id_at_link_time is a fingerprint of the referenced source's bytes at snapshot time, when locally accessible — same "fingerprint for drift detection, not a stored object" discipline as every prior plan's external-reference model. IRIs referring to external linked-data resources (e.g. a dbpedia: or wikidata: IRI used as an entity or class) are never fetched/cached in v0 (§1) — the IRI itself is the reference, and its resolution (does it dereference to a real resource right now) is a query-time, not commit-time, concern (§6).

Ontology normalization

Ontology axioms (subclass_of, domain_ref/range_ref, functional) are stored exactly as declared in the source, normalized only for serialization order (§3, above) — v0 does not compute or store the transitive closure of the class hierarchy as committed data (that would conflate asserted structure with derived structure); the opt-in RDFS closure (§1) is always a query-time computation over the stored, un-closed axioms, never baked into a snapshot.

Evidence preservation

Evidence/LiteratureReference graphs attached to Assertions are preserved as explicit snapshot data, not re-derived — mirrors the Biological Pathways and Research Papers plans' evidence-preservation principle, applied to arbitrary assertions instead of domain-specific claims/reactions.

Deterministic serialization

RDF serializers are famously non-deterministic (triple order, blank node labeling) across library versions and even across runs of the same library — Muse never hashes RDF/JSON-LD bytes directly for content-addressing; it always re-serializes through the canonical IR's fixed field order and ID scheme (§2), the strictest version yet of every prior plan's "never hash the interchange format's bytes directly" rule, since this domain's source format is the worst offender for serialization non-determinism of any format covered by a prior plan. This is the concrete implementation of §2's "Canonical RDF Representation" requirement: a golden-snapshot suite (§8, KG_04/KG_10–KG_13) locks it down by asserting that the same graph, imported from two different serializations or from the same serialization with triples reordered, produces byte-identical canonical JSON — never merely "close."

4. Semantic Diff

diff(base, target, *, repo_root=None) returns a StructuredDelta, same contract as every prior plan.

  • Entity additions/removals: InsertOp/DeleteOp keyed on entity identity (§3); an IRI-anchored entity gaining/losing an rdf:type is reported as a type-membership change, distinct from other property changes.
  • Relationship/assertion diffs: an Assertion added/removed is reported with the subject/predicate/object named directly (not just IDs) in the diff summary, same reviewer-legibility principle as the Biological Pathways plan's regulatory- network diff summaries — "X now hasParent Y," not opaque hashes.
  • Ontology changes: OntologyClass/OntologyProperty additions/removals/axiom changes reported distinctly from ordinary Assertion changes — a change to the schema (what relationships are meaningful) and a change to the data (what's currently asserted) are different review concerns, mirroring the Datasets plan's schema-vs.-data-content distinction.
  • Assertion provenance changes: an Assertion.evidence_refs change is reported distinctly from the assertion's own subject/predicate/object changing — evidence accrual vs. a changed claim are different kinds of update, same distinction the Research Papers/Pathways plans draw for evidence-vs.-claim changes.
  • Named graph / context diffs: an Assertion.named_graph_ref changing (the same fact reattributed to a different source context) reported distinctly from the assertion's truth-content changing.
  • Quality shape/result diffs: QualityShape.constraints changes (the shape itself changed) reported distinctly from QualityCheckResult changes (an entity's conformance status changed against an unchanged shape) — same rule-vs.- result distinction as the Datasets plan.
  • Inferred-vs-asserted diffs: a diff computed with the opt-in RDFS closure active (§1, §6) must clearly separate asserted-triple diffs (real committed changes) from inferred-triple diffs (consequences of the closure changing because the ontology or base assertions changed) — inferred changes are never committable (nothing to commit — they're derived), but useful diagnostic output when reviewing an ontology change's downstream effect, surfaced as an annotation, not folded into the StructuredDelta's committable ops.
  • Blank node diffs: given §3's blank-node-identity limitation, a blank node whose local subgraph changed is always reported as delete+insert, never as a ReplaceOp — flagged explicitly in the diff output as "blank node identity is not preserved across this change" so a reviewer isn't misled into thinking a large blank-node-anchored subgraph rewrite is a small edit.
  • Rename detection: for IRI-anchored entities, a "rename" only exists as a label field change (the IRI itself never changes without becoming a different entity by definition, §3) — reported as an ordinary field diff, not a special RenameOp, since IRI-based identity makes this domain's rename story simpler than every prior plan's.

5. Merge Strategy

merge(base, left, right, *, repo_root=None) — three-way, merge_mode: three_way, consistent with every prior plan.

  • Independent graph edits (the common case): edits to disjoint entities/ assertions/ontology terms merge automatically via the engine's default address- keyed map merge, driven by schema()'s dimensions (§9).
  • Same-subject-predicate conflicts: both branches assert different object values for the same (subject, functional-predicate) pair — reported conflict only when the predicate is declared functional (§2 — at most one value per subject, so two different values are a genuine contradiction); for a non- functional predicate, different objects for the same subject/predicate is normal (multi-valued) and auto-merges as two independent Assertion additions — this functional-vs-non-functional distinction is the domain's central, ontology-aware merge rule, implemented against the declared OntologyProperty.functional flag, never assumed.
  • Ontology conflicts: both branches change the same OntologyClass.subclass_of or OntologyProperty.domain_ref/range_ref differently — reported conflict (a genuine schema disagreement, consistent with every prior plan's rejection of silently picking a value neither author chose); both branches adding different new subclasses to the same parent auto-merges (disjoint hierarchy extensions).
  • Assertion conflicts: both branches change the same Assertion's named_graph_ref differently — conflict; both branches independently adding different evidence_refs to the same assertion auto-merges (union semantics, same "supporting evidence is almost always meant to coexist" principle the Video Timelines plan's marker merge and the Datasets plan's quality-rule-addition merge both apply).
  • Quality shape conflicts: both branches edit the same QualityShape.constraints differently — conflict; disjoint constraint additions to the same shape auto- merge.
  • Evidence conflicts: both branches set the same Evidence.confidence to different values (e.g. one branch marks a fact disputed, the other leaves it high) — reported conflict, since this is a genuine epistemic disagreement requiring human adjudication, not a value dispute with an obvious resolution.
  • Blank node conflicts: given §3/§4's blank-node-identity limitation, a merge touching a blank-node-anchored subgraph on either branch is conservatively treated as if the subgraph were replaced (never field-merged) — deliberately coarser than IRI-anchored merge behavior, documented and tested as an explicit, honest limitation (§8), not silently smoothed over.
  • Source-context conflicts: both branches assert contradictory facts within the same NamedGraph (two conflicting imports from the same claimed source) — flagged as a source-consistency conflict with extra context, escalated similarly to the Biological Pathways plan's species-consistency conflicts, since resolving it requires judgment, not a mechanical merge rule.
  • Human-readable merge reports: every conflict carries both branches' values, the base value, and a plain-language cause string (e.g. "Both branches asserted a different 'hasCapital' value for entity ':France': left says ':Paris', right says ':Lyon' — 'hasCapital' is a functional property, so this is a genuine contradiction requiring resolution"), consistent with the workspace- wide conflict-marker convention.

6. Query Support

A SPARQL-inspired query DSL (muse kg query "<pattern>" --json), plus purpose-built graph-traversal subcommands — this domain's query surface is deliberately modeled after SPARQL's triple-pattern syntax, familiar to anyone who's used RDF, without committing to full SPARQL 1.1 (§1).

muse kg query "?s rdf:type :Person" --json
muse kg query "?s :hasCapital ?o . ?s rdf:type :Country" --json
muse kg query "assertion.evidence.confidence == 'disputed'" --json
  • Find all assertions about an entity: muse kg query "<entity_iri> ?p ?o" --json / muse kg describe "<entity_iri>" --json
  • Traverse relationships (outgoing/incoming): muse kg traverse "<entity_iri>" --direction outgoing --depth 3 --json — walks Assertion edges, the general- purpose analog of the Biological Pathways plan's upstream/downstream.
  • Find shortest path between entities: muse kg shortest-path <from> <to> --json — standard graph shortest-path over Assertion edges.
  • Find entities by ontology class: muse kg query "?s rdf:type :Protein" --json / muse kg instances-of ":Protein" --json
  • Find disputed/low-confidence assertions: muse kg query "assertion.evidence.confidence in ('low', 'disputed')" --json
  • Find orphaned entities: muse kg orphans --json — entities with zero incident assertions, same connectivity-check pattern as the Biological Pathways plan's orphan query.
  • Find highly connected entities: muse kg hubs --top 10 --json — entities ranked by assertion-graph degree, same centrality framing as muse code gravity and the Biological Pathways plan's hubs query.
  • Query with opt-in RDFS entailment: muse kg query "?s rdf:type :Mammal" --with-inference --json — without --with-inference, only asserted rdf:type triples match; with it, the query engine computes the RDFS subclass closure first (§1) and includes entailed type memberships, always labeling which results are asserted vs. inferred in the output (never silently blending them).
  • Find entities failing a quality shape: muse kg query "shape_result.passed == false" --json / muse kg shape-violations "<shape_id>" --json
  • Query by external IRI / cross-reference: muse kg query "?s owl:sameAs <http://dbpedia.org/resource/X>" --json — returns the stored sameAs assertion; v0 never dereferences the external IRI live (§1, §3).
  • Cross-repo graph search: this spans multiple KG repos — muse hub search knowledge-graphs --class ":Protein" --predicate ":interactsWith" --json, the same cross-repo search architecture the Research Papers, Biological Pathways, and Datasets plans all required.

7. Validation and Drift Detection

Surfaces through muse status (via drift()) and muse check/muse code invariants, same pattern as every prior plan's _invariants.py.

  • Dangling IRI references: an Assertion.subject_ref/object_ref pointing at an Entity.id not present in the current snapshot — commit-blocking (structurally invalid), same class as every prior plan's broken-reference check.
  • Missing ontology imports: an Entity.types or Assertion.relationship_ref referring to an OntologyClass/OntologyProperty not declared in this snapshot's ontology — flagged, not necessarily blocking, since a valid workflow may reference a well-known external ontology term (e.g. foaf:Person) without fully importing that ontology's axioms — same "external reference, informational unless genuinely broken" policy as every prior plan.
  • Inconsistent OWL axioms: a structural (not full-reasoning, §1) consistency check — e.g. a class declared subclass_of itself transitively (invalid regardless of full DL semantics), or a property declared functional with more than one distinct object value for the same subject in the same snapshot (an immediate, structurally-detectable contradiction, not requiring a reasoner) — commit-blocking for these cheaply-detectable cases; deeper OWL-DL unsatisfiability is out of scope (§1).
  • Literal type mismatches: a literal value that doesn't parse as its declared datatype (e.g. "abc"^^xsd:integer) — commit-blocking.
  • Non-deterministic imports: re-importing the same source (RDF/JSON-LD) twice must produce identical canonical IR — hard-error regression gate, same treatment as every prior plan, and especially load-bearing here given §3's note that this domain's source formats are the worst offenders for serialization non-determinism of any domain plan so far.
  • Blank node identity churn: a snapshot where a large fraction of blank-node- anchored entities were reidentified as delete+insert (§3, §4) between otherwise- similar snapshots — flagged as an informational quality signal (not blocking) suggesting the source data would benefit from skolemization (§3) — this is a "your data model has a known rough edge" nudge, not a validity error.
  • Missing evidence: an Assertion with zero evidence_refs — flagged informational (a curator may assert a well-known fact without immediately attaching provenance), never commit-blocking, same policy as every prior plan's missing-evidence treatment.
  • Duplicate assertions: two Assertions with identical (subject, relationship, object, named_graph) that failed to dedup (an import bug, since this should be prevented by content-addressing) — flagged as a snapshot determinism failure, hard error, same treatment as every prior plan's non-deterministic-import check.
  • Schema migration risks: VersionMetadata.schema_version mismatches flagged before diff/merge across the boundary, never auto-migrated silently, consistent with the workspace-wide muse code migrate permission rule.

8. TDD Plan

Test IDs use the KG_NN prefix, per the workspace convention (CAD_NN, SCN_NN, VID_NN, PAP_NN, BIO_NN, DAT_NN).

Tier Coverage Example test IDs
Unit Canonical (de)serialization per element type; IRI vs. blank-node identity; datatype canonicalization; serialization-order invariance (§2). KG_01–KG_04
Golden snapshot Fixtures (IRI-only graph, functional-property ontology, blank nodes, multi-named-graph) snapshotted once, checked byte-for-byte. KG_10–KG_13
Round-trip import/export Turtle/JSON-LD round-trips reproduce input and identical IR on re-import; property-graph import produces a valid fidelity-reduced result, not divergent. KG_20–KG_22
Semantic diff One test per §4 category (entity, assertion, ontology, provenance, named-graph, quality, inferred-vs-asserted, blank-node flagging). KG_30–KG_37
Merge One test per §5 case, incl. functional-vs-non-functional merge rule, evidence union-merge, hierarchy disjoint-extension, blank-node conservative-replace. KG_50–KG_58
Traversal Outgoing/incoming/shortest-path/hubs/orphans against known-result fixtures, incl. depth-limit boundaries. KG_60–KG_64
Reasoning RDFS closure correctness against a known-answer fixture; inferred vs. asserted always labeled distinctly. KG_70, KG_71
Property-based Randomized axiom generation for cycle/contradiction checks; randomized blank-node mutation for delete+insert classification. KG_80, KG_81
Validation/drift One test per §7 bullet. KG_90–KG_97

Fixture strategy using representative graphs

Fixtures use real, small, permissively-licensed RDF datasets (a FOAF excerpt, a small bio-ontology subset, a fixture with genuine blank nodes from a real Turtle export) — real serializer quirks are exactly what §3's normalization must handle. A minimal library: an IRI-only graph with a simple class hierarchy; a functional- property-plus-contradiction fixture; a genuine-blank-node fixture; a multi-named- graph fixture with overlapping facts; a class-hierarchy-cycle fixture; a property- graph export. Property-based tests use synthetic generators for systematic axiom-validation and blank-node-mutation coverage.

9. Backend Integration

Domain plugin implementation

muse/plugins/kg/ follows the muse/plugins/midi/ layout:

muse/plugins/kg/
  __init__.py
  plugin.py           # snapshot/diff/merge/drift/apply/schema
  entity.py           # Canonical model dataclasses (§2)
  manifest.py         # SnapshotManifest (de)serialization, normalization (§3)
  kg_diff.py           # Diff algorithms (§4)
  kg_merge.py           # Merge algorithms (§5)
  graph.py             # Graph traversal + opt-in RDFS closure (§6, §1)
  _query.py            # SPARQL-inspired query DSL (§6)
  _invariants.py       # Validation/drift checks (§7)
  _import_rdf.py        # Turtle/N-Triples/RDF-XML/JSON-LD <-> IR adapter (via rdflib)
  _import_pg.py          # Property-graph -> IR adapter (import-only)

schema() declares (mirroring every prior plan's shape):

{
  "schema_version": "<muse_version>",
  "merge_mode": "three_way",
  "description": "Semantic version control for knowledge graphs...",
  "dimensions": [
    {"name": "entities", "description": "IRI- and blank-node-anchored entities and their types."},
    {"name": "ontology", "description": "Classes, properties, and structural axioms."},
    {"name": "assertions", "description": "Triple instances with provenance and named-graph context."},
    {"name": "evidence", "description": "Evidence and literature references supporting assertions."},
    {"name": "quality", "description": "Shape constraints and their conformance results."},
    {"name": "metadata", "description": "Curator, source format, schema version."}
  ]
}

Storage implications

Snapshots are small, structured graph data — no large binary payloads. No knowledge-graph-specific changes to .muse/objects//gc/prune.

API endpoints needed (MuseHub)

  • GET /:owner/:repo/kg/:ref/graph — full entity/assertion/ontology graph at a ref.
  • GET /:owner/:repo/kg/:ref/diff?base=&target= — structured diff for review rendering, incl. the inferred-vs-asserted annotation (§4).
  • GET /:owner/:repo/kg/:ref/traverse?from=&direction=&depth= — server-side traversal backing §6's CLI.
  • GET /hub/search/knowledge-graphs?class=&predicate= — cross-repo search (§6), same pattern the Papers/Pathways/Datasets plans required.

CLI implications

New muse kg subcommand group: query, describe, traverse, shortest-path, instances-of, orphans, hubs, shape-violations. All accept --json.

MuseHub UI implications

  • Proposal/diff view needs a knowledge graph diff renderer: an interactive graph view with changed entities/assertions highlighted, an ontology-change panel distinct from the data-change panel (§4), and a conflict panel in plain language (§5) — the graph-native analog of the Biological Pathways plan's diff renderer.
  • Repo browse view needs a graph-exploration view (pan/zoom/filter by class or named graph), plus an ontology browser (class hierarchy tree) distinct from the instance-data graph.

Interactive graph visualization

  • Standard node/edge graph rendering, entities colored/grouped by rdf:type, with an optional overlay toggling inferred edges (RDFS closure, §6) visually distinct from asserted ones.
  • Ontology-hierarchy view: a dedicated tree/graph view of just the OntologyClass subclass_of structure, separate from the instance-data graph, since "did the schema change" is a different review task from "did the data change."

Graph exploration

  • Click-to-traverse: selecting an entity surfaces its outgoing/incoming assertions inline (backed by /traverse above), the same interactive affordance the Biological Pathways plan's UI provides.
  • Export to Turtle or property-graph JSON for external analysis — one-way, not a round-trip guarantee beyond §8's tested fidelity ceilings.

MCP/tooling implications

Expose kg_query, kg_traverse, kg_diff, kg_shape_violations as MCP tools, mirroring every prior plan — lets agentception workers check "does this change introduce a functional-property contradiction" before a merge completes.

10. Future Semantic Extensions

Named here so the v0 scope boundary (§1) is legible as a deliberate step, not a ceiling:

  • Full OWL-DL reasoning: satisfiability checking and full class-expression entailment beyond v0's structural-axiom validation and lightweight RDFS closure (§1) — a real description-logic reasoner, a substantial follow-on effort.
  • Automatic ontology alignment: mapping two independently-authored ontologies onto each other — deferred, §1.
  • Full SPARQL 1.1 support: aggregates, property paths, federated SERVICE queries beyond v0's SPARQL-inspired subset (§6).
  • Embedding-based semantic search: similarity search over entity descriptions beyond v0's exact-match query DSL — connects to whatever vector-search infrastructure MuseHub eventually adopts for other domains.
  • Skolemization-by-default import mode: converting blank nodes to deterministic, minted IRIs at import time to sidestep §3's blank-node-identity limitation for graphs that don't need genuine anonymity — the most likely near-term follow-up, given how sharp that limitation is (§3–§5, §7).
  • Federated cross-repo graph queries as one logical graph: beyond §1's per-repo scope — a natural extension of §6's cross-repo search once proven.
  • SHACL full conformance: extending QualityShape (§2) from its practical subset to full constraint-component coverage.
  • Provenance-aware trust scoring: derived trust/confidence scores aggregating Evidence.confidence and source reputation across a graph, beyond v0's per-assertion confidence alone.
  • Knowledge-graph-to-paper/pathway/dataset cross-linking: this domain's general-purpose model is a natural substrate other plans' evidence/citation elements could reference into, rather than each maintaining a disconnected bibliography model — worth revisiting once several graph-shaped plans (#69, #70, #71, this one) are implemented and the overlap is concretely visible.
  • Temporal/versioned assertions: modeling assertions true only during a specific interval ("X was CEO of Y from 2015–2020") as first-class, rather than v0's implicit "true as of this commit" model.

11. Staging Deliverables

Proposed issue breakdown

  1. KG-1 — Canonical model + snapshot + entity identity (§2, §3): entity dataclasses, IRI/blank-node identity handling, RDF import bridge (rdflib), golden-snapshot tests (KG_01–KG_13). Load-bearing — nothing else starts until this lands.
  2. KG-2 — JSON-LD/property-graph import + Turtle/JSON-LD export (§1, §3): depends on KG-1. KG_20–KG_22.
  3. KG-3 — Semantic diff (§4): depends on KG-1. KG_30–KG_37.
  4. KG-4 — Merge strategy (§5): depends on KG-1, KG-3. KG_50–KG_58.
  5. KG-5 — Graph traversal + opt-in RDFS closure (§6, §1): depends on KG-1; parallel with KG-3/KG-4. KG_60–KG_71.
  6. KG-6 — Query DSL + CLI (§6, §9): depends on KG-1, KG-5. Folded into the same test range as KG-5's traversal work.
  7. KG-7 — Property-based axiom/blank-node suite (§8): depends on KG-1, KG-3. KG_80, KG_81.
  8. KG-8 — Remaining validation & drift checks (§7): depends on KG-1; parallel with KG-3/KG-4/KG-5. KG_90–KG_97.
  9. KG-9 — MuseHub API endpoints + cross-repo graph search (§9): depends on KG-1, KG-3, KG-5.
  10. KG-10 — MuseHub UI: graph explorer, ontology browser, diff renderer (§9): depends on KG-9. Not scoped in detail until KG-3/KG-4 output shapes are proven.
  11. KG-11 — MCP tooling exposure (§9): depends on KG-5, KG-6.

Milestones

  • M1 (Foundation): KG-1 merged, muse domains --json lists kg as active, golden-snapshot suite green, blank-node identity behavior verified against the blank-node fixture without silent misclassification.
  • M2 (Interop): KG-2 merged; Turtle/JSON-LD round-trip fixtures pass at their fidelity ceilings.
  • M3 (Semantic core): KG-3 + KG-4 + KG-5 merged; a two-branch conflict fixture (both branches assert different values for the same functional property) produces a correct conflict report; the non-functional-property auto-merge case verified working as designed; RDFS closure correctness verified against a known-answer fixture.
  • M4 (Usable): KG-6 + KG-7 + KG-8 merged; muse kg traverse/shortest-path/ hubs/orphans/shape-violations all answer correctly against fixtures.
  • M5 (Ecosystem): KG-9, KG-10, KG-11 merged; a KG proposal on staging MuseHub renders a real graph-level diff with an interactive explorer and ontology browser.

Acceptance criteria (whole-plan level)

  • muse domains --json shows kg with active: true and the schema in §9.
  • The same graph imported from two different serializations (e.g. Turtle vs. RDF/XML), or from the same serialization with triples reordered and blank nodes relabeled, produces byte-identical canonical snapshots and zero semantic diff — the direct, testable expression of §2's "Canonical RDF Representation" requirement, not merely a documented intention.
  • Every fixture round-trips through Turtle/JSON-LD (§8) with zero diff against itself; the property-graph import produces a valid fidelity-reduced result, not a divergent one.
  • The functional-property-contradiction fixture is commit-blocking (§7); the class-hierarchy-cycle fixture is commit-blocking; well-formed fixtures produce no false positives on either check.
  • A merge of two branches asserting different values for the same non-functional property auto-resolves as two independent assertions; the same scenario against a declared functional property reports a correctly-worded conflict.
  • The blank-node fixture, when mutated, is consistently reported as delete+insert (never a false ReplaceOp), with the "identity not preserved" flag present.
  • muse kg traverse/shortest-path/hubs correctly answer against fixtures with known expected results, including depth-limit boundary behavior; RDFS-closure query results are always labeled asserted vs. inferred, never blended.
  • No test depends on network access (including no live dereferencing of external IRIs), a specific rdflib version's exact serialization output, or wall-clock time (§3/§7 determinism requirement).

Risks

  • Blank node identity risk: this domain's hardest, most visible limitation (§3) — real-world RDF uses blank nodes constantly, so this is routine, not rare. Mitigation: documented prominently, tested (§8), flagged in every relevant diff/ merge/drift output rather than silently producing misleading small-looking diffs — skolemization-by-default (§10) is the likely near-term fix, revisit post-M1.
  • Functional-property-merge-rule risk: §5's central merge decision depends entirely on OntologyProperty.functional being correctly declared — an undeclared property gets the conservative auto-merge default, which could silently merge what should have been a contradiction. Mitigation: §7 flags undeclared functional/non-functional properties as an informational gap.
  • Reasoning-scope risk: the boundary between always-on validation (§7) and opt-in RDFS closure (§6) must stay crisp. Mitigation: §7 stays scoped to cheaply-detectable, non-reasoning cases only.
  • Cross-repo search risk: same risk class every prior plan with cross-repo search flagged (#69, #70, #71) — should share one implementation across all four.

Open questions

  • Should v0 skolemize blank nodes by default at import time (§3, §10)? The highest-leverage decision left open — resolve before KG-1, given how much of §3–§5, §7 exists to work around not skolemizing.
  • What is the precise, minimal set of "cheaply structural" OWL checks that belong in §7's always-on validation vs. the opt-in reasoning path (§6)? Needs enumeration during KG-1/KG-8 scoping.
  • Should cross-repo search (§9) share one implementation with the Papers/Pathways/ Datasets plans' equivalent endpoints?
  • Should LiteratureReference (§2) be extracted into a shared cross-domain element rather than redefined per-plan? Flagged once duplication is visible.
  • How aggressively should the opt-in RDFS closure (§6) be cached, given it's recomputed at query time (§3)? Needs real performance data during KG-5.

Implementation order

KG-1 → (KG-2, KG-3, KG-7 in parallel) → KG-4 → KG-5 → KG-6 → KG-8 → KG-9 → (KG-10, KG-11 in parallel).

Plan-only. Each §11 deliverable becomes its own staging issue once reviewed.

◎ Activity
●
gabriel opened this issue 19 days ago
◎
No activity yet. Use the CLI to comment.