gabriel / muse public
Open #71 Enhancement
filed by gabriel human · 19 days ago

Datasets Domain — implementation plan (TDD-first, comprehensive)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Datasets Domain — Implementation Plan

Background

Muse treats "domain" as a first-class primitive (muse/domain.py::MuseDomainPlugin). A plugin implements six methods — snapshot, diff, merge, drift, apply, schema — and gets the full VCS engine free: commit DAG, branching, checkout, lineage, log graph, merge-base finder. The CAD (#66), 3D Scenes (#67), Video Timelines (#68), Research Papers (#69), and Biological Pathways (#70) plans are proving the model across geometry, spatial hierarchy, temporal edits, scholarly argument, and biological graphs respectively.

Datasets are an eighth shape of problem: tabular/columnar records governed by a schema, at a scale that ranges from kilobytes to terabytes, where the meaningful unit of change is rarely "the bytes changed" and almost always "the schema changed," "the distribution shifted," "N rows were added/removed," or "a transformation upstream altered this column's meaning." Every dataset already referenced by the Biological Pathways plan's Evidence/experimental data and the Research Papers plan's Dataset external reference is exactly this domain's subject — this plan is the one those two deferred to. Today a CSV/Parquet/SQLite file is diffed by a generic VCS as an opaque byte blob (or, worse, as text — a single added column produces a full-file diff). If Muse can version a dataset the way it versions code — diffing schema and row-level changes, not bytes; merging disjoint partitions; tracking lineage through transformations; validating quality rules — it proves the domain- plugin model generalizes to data engineering, not just creative/scientific artifacts. The goal is not to version files. It is to version data meaning, schema, quality, lineage, and provenance.

This issue is the plan only. Each numbered phase in §11 becomes its own staging issue once reviewed and accepted.

Goal

Ship a muse/plugins/dataset/ domain plugin, structured like muse/plugins/midi/ and the CAD/3D-Scenes/Timeline/Paper/Pathway plans, that:

  • Registers in muse domains --json alongside code, midi, identity, mist, social, and (once landed) cad/scene3d/timeline/paper/pathway.
  • Passes muse code test --json for every module it touches.
  • Snapshots, diffs, and merges a dataset with schema-level, column-level, and distribution-level semantic awareness — never as an opaque CSV/Parquet/SQLite blob.
  • Supports a query DSL (muse dataset query "...") answering the examples in §6.
  • Detects the drift/validation classes in §7 via muse status/muse check.
  • Has test coverage per §8, with test IDs (DAT_01, DAT_02, …) referenced from both this plan and the tests that satisfy them.

"Done" for the plan means: reviewed, decomposed into §11's breakdown, with acceptance criteria concrete enough for a different agent to implement test-first.

1. Domain Scope

Supported initial data shapes

Tabular data with a defined or inferable schema: flat tables (CSV/Parquet/dataframe- shaped) and single-table SQLite/DuckDB exports in v0. Semi-structured nested data (JSON/JSONL with variable shape per record) is supported for schema inference and drift detection but not full nested-path diffing in v0 (see out-of-scope). Multi- table relational exports (a full SQLite database with foreign keys across tables) are in scope for schema/relationship modeling (§2) but full multi-table join-aware diffing is deferred.

Two-layer strategy, consistent with every prior plan, with one domain-specific twist: datasets can be arbitrarily large, so the canonical layer is explicitly not a full copy of the data.

  1. Canonical semantic layer (Muse-owned): a JSON-serializable dataset IR (§2) covering schema, column statistics/profile, quality rules, lineage, and a bounded sample of records — never the full row set. This is the layer Muse snapshots, diffs, merges, and queries directly.
  2. Interchange/storage boundary: the dataset's full row data is always an external reference (mirrors the Video Timelines plan's media-essence boundary), content-fingerprinted at link time. v0 reads CSV/JSON/JSONL/Parquet directly (via existing columnar libraries — Arrow/Parquet readers, no bespoke parser) and SQLite/DuckDB via their native query interfaces, to compute schema and statistics without ever loading the full dataset into Muse's own storage.

Relationship to CSV, JSON, JSONL, Parquet, SQLite, DuckDB, dataframes, warehouse exports

Format Role
CSV v0 primary flat-table import target — schema inferred (typed, since CSV itself is untyped text), profiled, sampled. Lowest fidelity for types (everything round-trips as strings unless explicit type inference is applied and recorded) — flagged, not silently assumed.
Parquet v0 primary typed columnar format — schema is embedded in the file itself (no inference needed), making it the highest-fidelity v0 import target and the recommended canonical storage format for any sample/profile data Muse itself persists.
JSON/JSONL v0 import target for semi-structured/nested records; schema inference operates on a sampled record set and produces a shape (§2) that captures optionality and nesting, flagged as inferred (not authoritative) since JSON has no schema of its own.
SQLite v0 import target via native SQL introspection (PRAGMA table_info, foreign keys) — schema is authoritative (the database enforces it), unlike CSV/JSON.
DuckDB Same treatment as SQLite — native introspection, authoritative schema — and additionally usable as the query engine Muse's own profiling/sampling step runs through (DuckDB can query Parquet/CSV directly without a load step), not just an import source.
Dataframes (pandas/Polars) Not a file format — a runtime object. v0 supports snapshotting a dataframe via its in-memory schema/dtype info directly (no serialization round-trip needed) when the plugin is invoked from a Python context that holds one; this is a convenience entrypoint on top of the same canonical IR, not a separate model.
Warehouse exports (e.g. a BigQuery/Snowflake table dump) Typically arrive as Parquet or CSV already — reached transitively through those adapters, the same "integration point, not per-source adapter" leverage the Video Timelines plan gets from OTIO. Native warehouse connection (querying a live warehouse table without an export step) is a named v1+ direction (§10), not built in v0.

What is in scope for v0

  • Schema definition: column names, types, nullability, and (where the source provides it) constraints (primary key, uniqueness, foreign key).
  • Column-level statistics: type, null rate, cardinality/distinct-count, and for numeric columns min/max/mean/stddev/quantiles — a profile, not the full data.
  • A bounded, deterministic sample of records (for human review and fixture-style golden testing), not the full row set.
  • Row-count and basic shape drift (row count changed, column added/removed/retyped) between snapshots.
  • Lineage: an explicit, recorded chain of transformations (source dataset(s) → transformation → this dataset), even when v0 does not execute those transformations itself — lineage is asserted/recorded metadata, not automatically derived by static analysis of a transformation script in v0.
  • Quality rules and their pass/fail results as first-class, versioned artifacts (e.g. "column age must be non-negative," "column email must be unique") — the rules themselves are versioned content, and their evaluation results are drift-detectable (§7), mirroring the Research Papers plan's evidence/claim-status relationship applied to data quality instead of scientific claims.
  • Basic relational structure: primary/foreign key declarations across tables within one dataset (for the SQLite/multi-table case), enough to validate referential integrity (§7) without full multi-table diff/merge.

What is explicitly out of scope for v0

  • Storing or diffing the full row set byte-for-byte — this domain versions schema, profile, sample, lineage, and quality, never the complete data itself, mirroring the Video Timelines plan's decision to never store media essence. Full row-level diff (which specific rows changed) is a named v1+ direction (§10), not attempted now.
  • Full nested-JSON path-level diffing (deeply nested, variably-shaped documents diffed field-by-field at arbitrary depth) — v0 infers a flattened shape summary for schema/drift purposes but does not attempt a general recursive structural diff of arbitrary JSON documents.
  • Multi-table join-aware diff/merge (detecting that a change to table A's foreign key broke table B's referential integrity across a diff) — v0 validates referential integrity at snapshot time (§7) but does not diff relational structure changes across tables as a joined operation.
  • Live warehouse connections (querying a running BigQuery/Snowflake/Postgres instance directly rather than an export) — named in §10, not built now.
  • Data transformation execution (actually running a dbt model, a Spark job, or a Python script to materialize a derived dataset) — lineage records that a transformation happened and what it claims to do, but this domain does not become a pipeline orchestrator. That is explicitly out of scope, permanently, not just for v0.
  • Automatic PII/sensitive-data detection and redaction — a real need, but a distinct scope with its own correctness bar (false negatives are a compliance risk); named as a future extension (§10), not attempted here.

2. Canonical Model

All elements are content-addressed (sha256:<hex>) and referenced by ID, never positional index — same identity discipline as every prior plan.

Element Fields Notes
Dataset id, name, schema_ref: Schema.id, profile_ref: Profile.id, sample_ref: Sample.id, lineage_refs: [LineageEdge.id], quality_rules: [QualityRule.id], source_ref: DataSource.id, metadata The top-level container — one versioned table (or one SQLite/DuckDB database's worth of related tables via relationships, below).
Schema id, columns: [Column.id] (ordered — column order is preserved as authored metadata but not identity-bearing, since reordering columns is a cosmetic change, not a semantic one), primary_key: [Column.id] \| null, relationships: [ForeignKeyRelationship.id]
Column id, name, data_type (integer|float|string|boolean|date|datetime|nested|unknown), nullable: bool, inferred: bool, description \| null inferred flags whether the type came from an authoritative source (Parquet/SQLite schema) or was guessed (CSV/JSON) — load-bearing for §7's validation.
ForeignKeyRelationship id, from_column_ref: Column.id, to_dataset_ref: Dataset.id \| null, to_column_ref: Column.id, resolved: bool to_dataset_ref is null when the relationship is within the same multi-table source (§1); external cross-dataset FKs are a weaker, resolved-tracked reference, same external-reference discipline as every prior plan.
Profile id, row_count, column_stats: [ColumnStatistic.id], computed_at_content_id (a fingerprint of the source data this profile was computed from, for staleness detection, §7) The dataset's statistical summary — this, not the raw rows, is what most diffs (§4) actually compare.
ColumnStatistic id, column_ref: Column.id, null_count, distinct_count \| null (expensive on very large data — null means not computed, not zero), min \| null, max \| null, mean \| null, stddev \| null, quantiles: {p: value} \| null, top_values: [{value, count}] \| null Numeric fields are null, not zero, when not applicable/not computed — avoids conflating "no variance" with "not measured."
Sample id, record_refs: [Record.id], sampling_method (head|random|stratified), sample_size, seed \| null A bounded, deterministic sample (§1) — seed makes random sampling reproducible across re-snapshots of unchanged data, required for the determinism guarantee in §3.
Record id, values: {column_id: value} Content-addressed by its own value set — two identical sample rows across different datasets or snapshots collapse to one stored object, same dedup principle as every prior plan's content-addressing.
DataSource id, uri_or_path, content_id_at_link_time \| null, format (csv|json|jsonl|parquet|sqlite|duckdb|dataframe), resolved: bool External reference to the full data, never the data itself (§1) — same pattern as the Video Timelines plan's SourceMedia.
LineageEdge id, upstream_ref: Dataset.id \| DataSource.id, transformation: {kind, description, tool_hint}, downstream_ref: Dataset.id An explicit, asserted provenance edge — recorded metadata, not derived by executing or statically analyzing the transformation (§1).
QualityRule id, name, expression (a declarative predicate, e.g. column('age') >= 0), column_refs: [Column.id], severity (error|warning) The rule itself is versioned content — changing a quality threshold is a diffable, reviewable change (§4), not a side effect.
QualityCheckResult id, rule_ref: QualityRule.id, passed: bool, failure_count \| null, evaluated_at_content_id Analogous to the Biological Pathways plan's Evidence/Reaction split: the rule is the claim, the result is evidence of whether it currently holds — evaluated_at_content_id enables staleness detection (§7) the same way Profile.computed_at_content_id does.
VersionMetadata curator \| null, source_format, source_checksum, schema_version

Design invariant carried over from every prior plan: identity is content- and reference-based, never positional. Column order, sample-row order, and quality-rule list order are preserved as authored metadata but never identity-bearing — reordering never itself produces a diff.

3. Snapshot Design

Deterministic dataset normalization

snapshot() reads the live state (a DataSource-referenced file or connection under state/, or a live dataframe handle) and produces the canonical IR as the SnapshotManifest:

  1. Resolve every element to a stable ID (below).
  2. Infer or read Schema (authoritative for Parquet/SQLite/DuckDB, inferred with a sampling_method-documented pass for CSV/JSON).
  3. Compute Profile/ColumnStatistics via a single pass (or, for very large sources, via DuckDB querying the source directly rather than loading it into memory — the same "query engine, not load-everything" principle that makes DuckDB useful here beyond being an import format, §1).
  4. Draw Sample deterministically: head sampling is trivially deterministic; random/stratified sampling uses a recorded seed so re-snapshotting unchanged source data reproduces byte-identical sample records — the dataset-domain equivalent of every prior plan's "never let normalization introduce nondeterminism" rule.
  5. Evaluate QualityRules against the source (via the same query-engine pass as step 3) to produce QualityCheckResults.

Stable identity for schema and profile elements

  • Column identity is content-derived from (name, position-independent — see below) — actually just name plus Schema parent identity, since two columns with the same name in the same schema are the same column by construction (no duplicate-name columns are valid in any of v0's source formats). A column rename is therefore, unlike every other domain plan's rename handling, genuinely ambiguous at the schema level alone: renaming col_a to col_b and adding a new col_b are indistinguishable from schema alone. v0 resolves this the same way the Biological Pathways plan resolves weak-identity ambiguity: flagged as ambiguous when a column disappears and a new one appears in the same diff, with a configurable name-similarity heuristic as a hint, never a silent assumption (§4, §8).
  • Row-count/profile changes are not entity-identity concerns — Profile and ColumnStatistic are keyed to their parent Dataset/Column, always singular per snapshot, so there's no rename/move problem for statistics themselves, only a value-change problem (§4).
  • Sample Record identity is content-derived from its own value set (§2) — a record's identity is independent of which snapshot sampled it, so an unchanged row that happens to be resampled across two snapshots is recognized as the same object, not a spurious insert+delete.

Canonical graph/relational representation

ForeignKeyRelationship/LineageEdge are preserved as explicit snapshot data (§2), not re-derived at query time — the same "snapshot-time graph, not query-time inference" principle the Biological Pathways and Research Papers plans apply to their respective graphs, enabling §6's lineage-traversal queries to be direct graph walks over committed data.

External database references

DataSource.content_id_at_link_time is a fingerprint of the referenced file/export's bytes at snapshot time, when locally accessible — same "fingerprint for drift detection, not a stored object" discipline as the Video Timelines plan's SourceMedia. When the source is a live connection (a DuckDB query against a remote Parquet file, for instance) rather than a static file, content_id_at_link_time may be null with resolved still true if the connection itself succeeded — a live- queryable source is "resolved" even without a static byte fingerprint, an explicit nuance this domain needs that no prior plan's external-reference model required.

Deterministic serialization

Never hash the source CSV/JSON/Parquet/SQLite bytes directly for the canonical IR's content-addressing — always re-serialize schema/profile/sample through the IR's fixed field order and ID scheme (§2), same discipline as every prior plan. A golden- snapshot suite (§8) locks this down: re-snapshotting the same unchanged source twice must produce byte-identical canonical JSON, including the sample (via the recorded seed).

Large object handling

Full row data is never stored by Muse (§1) — always an external DataSource reference. The only data this plugin ever writes to .muse/objects/ is the small canonical artifacts: Schema, Profile/ColumnStatistics, and the bounded Sample records — kilobytes to low megabytes regardless of the underlying dataset's actual size, the same storage-boundary discipline as the Video Timelines plan's media- essence exclusion, load-bearing and not to be relaxed incrementally (§10 tracks it as an open question, not a default to erode).

4. Semantic Diff

diff(base, target, *, repo_root=None) returns a StructuredDelta, same contract as every prior plan.

  • Schema diffs: Column add/remove — reported plainly; a same-name column with a changed data_type is a ReplaceOp distinct from add/remove; a disappear+appear pair in the same diff is flagged ambiguous rename per §3, with a similarity- score hint, never silently resolved as a confident rename.
  • Profile/statistics diffs: ColumnStatistic field-level diffs (e.g. mean shifted, null_count increased) reported with the delta and, where meaningful, a relative-change magnitude (e.g. "+340% distinct values") — this is often the single most useful diff in the whole domain, since it's how a reviewer sees "the data distribution shifted" without inspecting rows.
  • Row-count/shape diffs: Profile.row_count delta reported explicitly (e.g. "12,000 rows added"), distinct from column-level changes — a shape change and a distribution change are different review concerns.
  • Sample diffs: Record add/remove/replace within the Sample, reported for human-readability purposes (what does the new data actually look like) — not used as a stand-in for "the data changed," since a sample diff showing no change does not imply the full dataset is unchanged (only the profile/row-count diffs make that claim, and only within their own measurement precision).
  • Quality rule diffs: QualityRule.expression/severity changes are diffed as content changes (the rule itself changed) distinctly from QualityCheckResult changes (the data now passes/fails a rule that didn't change) — mirrors the Biological Pathways plan's evidence-confidence-vs-citation-list distinction, applied to quality rules vs. their evaluation results.
  • Lineage diffs: LineageEdge add/remove/change reported with both endpoints named directly (not just IDs), same reviewer-legibility principle as the Biological Pathways plan's regulatory-network diff summaries.
  • Relationship diffs: ForeignKeyRelationship add/remove/retarget, distinct from a resolved flag flip (the referenced dataset became available/unavailable) — same distinction every prior plan draws for external references.
  • Source diffs: a DataSource.content_id_at_link_time change ("underlying file replaced") reported distinctly from a resolved flag flip, same pattern as every prior plan's ExternalReference treatment.
  • Staleness-aware diffing: when Profile.computed_at_content_id or a QualityCheckResult.evaluated_at_content_id doesn't match the current DataSource's live fingerprint, the diff output flags the comparison itself as potentially stale (comparing against an outdated profile) rather than silently presenting a diff that may not reflect the data's actual current state — this is a diff-time concern distinct from, but related to, §7's staleness drift check.

5. Merge Strategy

merge(base, left, right, *, repo_root=None) — three-way, merge_mode: three_way, consistent with every prior plan.

  • Independent schema/quality-rule edits (the common case): one branch adds a column while the other adds an unrelated quality rule — merge automatically via the engine's default address-keyed map merge, driven by schema()'s dimensions (§9).
  • Same-column conflicts: both branches change the same Column.data_type differently — reported conflict (no auto-resolution of a type dispute, consistent with every prior plan's rejection of silently picking a value neither author chose); both branches adding a description where none existed, if identical, merges by construction, if different, conflicts.
  • Column-rename ambiguity in merge: if the ambiguous-rename detection (§3, §4) fires on one branch's diff against base, and the other branch independently edited the same disappeared/appeared pair, the merge treats this as a same- entity conflict using the higher-confidence similarity match, not as two unrelated add/remove pairs — an explicit, tested case (§8) since naively merging the ambiguous ops independently would double-count the rename.
  • Profile/statistics conflicts: statistics are derived, not authored data — two branches never "conflict" on a ColumnStatistic value the way they conflict on authored content, because the merge result's profile is always recomputed fresh against the merged schema/source (§3's snapshot process re-run post-merge), not merged field-by-field. This is a deliberate departure from the field-level-merge pattern used for authored elements elsewhere in this plan and every prior plan — called out explicitly so it isn't mistaken for an oversight.
  • Quality rule conflicts: both branches edit the same QualityRule.expression differently — conflict; both branches independently adding different new rules auto-merges (disjoint additions).
  • Relationship conflicts: both branches change the same ForeignKeyRelationship target differently — conflict, same as every prior plan's reference-retarget conflict treatment.
  • Lineage conflicts: both branches record a different upstream source for the same transformation step — reported conflict (a genuine provenance disagreement, not a value dispute, so the report should surface both claimed lineages in full for human adjudication); both branches independently adding different lineage edges (e.g. documenting two different downstream consumers) auto-merges.
  • Source conflicts: both branches point the same DataSource.id at different uri_or_path — reported conflict; a resolved flag change alone is not (mirrors the Video Timelines plan's missing-media-is-a-state-change-not-a-value-dispute treatment).
  • Human-readable merge reports: every conflict carries both branches' values, the base value, and a plain-language cause string (e.g. "Both branches changed column 'user_id' type: left declared 'integer', right declared 'string' — base was 'integer'"), consistent with the workspace-wide conflict-marker convention.

6. Query Support

A query DSL mirroring every prior plan's equivalent (muse dataset query "<predicate>" --json), plus purpose-built subcommands.

muse dataset query "column.data_type == 'string' and column.nullable == true" --json
muse dataset query "quality_rule.severity == 'error' and result.passed == false" --json
muse dataset query "profile.row_count > 1000000" --json
  • Find columns by type/nullability: muse dataset query "column.data_type == 'datetime'" --json
  • Find failing quality rules: muse dataset failing-rules --json — surfaces every QualityCheckResult with passed == false, severity-sorted.
  • Find schema drift between versions: muse dataset schema-diff <ref1> <ref2> --json — a named convenience over the generic diff, scoped to just Schema.
  • Find distribution shifts: `muse dataset query "column.stats.mean.delta(<ref>)

    2 * column.stats.stddev" --json` — a statistically-aware query comparing a column's current stats against a prior ref's, useful for catching silent data drift before it's a production incident.

  • Trace lineage upstream: muse dataset lineage "<dataset_id>" --direction upstream --depth 5 --json — walks LineageEdges backward, the dataset-domain analog of the Biological Pathways plan's upstream traversal.
  • Trace lineage downstream: muse dataset lineage "<dataset_id>" --direction downstream --json — forward walk, the mirror query, useful for impact analysis ("what breaks if I change this dataset's schema").
  • Find datasets by source format/size: muse dataset query "source.format == 'parquet' and profile.row_count > 10000000" --json
  • Find referential integrity violations: muse dataset query "relationship.resolved == false" --json / muse dataset broken-relationships --json
  • Find datasets referencing a given dataset (downstream consumers): this spans multiple dataset repos — muse hub search datasets --upstream-of "<dataset_id>" --json, a MuseHub-level cross-repo search endpoint (§9), the same architectural pattern the Research Papers and Biological Pathways plans required for cross-repo search.
  • Sample inspection: muse dataset sample --json — returns the current Sample.record_refs, the closest thing to "show me the data" this domain offers without resolving the full external source.

7. Validation and Drift Detection

Surfaces through muse status (via drift()) and muse check/muse code invariants, same pattern as every prior plan's _invariants.py.

  • Schema drift: a live DataSource's current schema (re-read at drift-check time) differing from the committed Schema — surfaced prominently, since this is the single most common and most consequential drift class for a data pipeline (an upstream schema change silently breaking downstream consumers is a classic, costly data-engineering failure mode this domain exists specifically to catch).
  • Profile staleness: Profile.computed_at_content_id not matching the live source's current fingerprint — flagged informational by default (the committed profile is simply out of date, a normal state between commits, not an error) but escalated to a warning if the live source's fingerprint suggests a schema change specifically (not just new rows), tying back to the schema-drift check above.
  • Quality rule failures: any QualityCheckResult with passed == false and severity == 'error' — commit-blocking by default (this is the domain's primary enforceable gate, structurally analogous to a CI test failure); severity == 'warning' results are flagged, not blocking.
  • Broken relationships: a ForeignKeyRelationship with resolved == false or pointing at rows that don't actually exist in the target (checkable when both sides are locally resolvable) — commit-blocking when checkable, informational when the target is an unresolved external reference (same "flagged vs. blocking" split every prior plan applies to external references it can't fully verify).
  • Missing/unresolved data source: DataSource.resolved == false — informational, not blocking, same offline-reference-is-routine policy as the Video Timelines and Research Papers plans.
  • Duplicate primary keys: a Schema.primary_key that the live source's own data violates (checkable via the same query-engine pass used for profiling, §3) — commit-blocking, since this represents genuinely invalid data against the declared schema, not a normal intermediate state.
  • Type-inference confidence: a Column.inferred == true type that the query- engine pass finds to be inconsistent with a sample of the actual data (e.g. inferred integer but the source contains non-numeric strings in that column) — flagged, not necessarily blocking, since v0's inference is heuristic and this is exactly the kind of gap the inferred flag exists to make visible (§2).
  • Ambiguous rename left unresolved: a committed snapshot where an ambiguous column rename (§3, §4) was never explicitly confirmed or rejected by a human — flagged as an open item, not blocking (this is a review-completeness signal, not a data-validity error).
  • Non-deterministic imports: re-snapshotting the same unchanged source twice must produce identical canonical IR, including the sample (via the recorded seed, §3) — hard-error regression gate, same treatment as every prior plan.
  • Schema migration risks: VersionMetadata.schema_version mismatches flagged before diff/merge across the boundary, never auto-migrated silently, consistent with the workspace-wide muse code migrate permission rule.

8. TDD Plan

Test IDs use the DAT_NN prefix, per the workspace convention (CAD_NN, SCN_NN, VID_NN, PAP_NN, BIO_NN).

Tier Coverage Example test IDs
Unit Canonical model (de)serialization for every element type; column identity and ambiguous-rename detection; deterministic sampling (seeded random/stratified). DAT_01–DAT_03
Golden snapshot Fixed fixtures (a small typed Parquet table, an untyped CSV requiring inference, a SQLite database with a foreign key, a JSON/JSONL file with nested/variable shape) snapshotted once, checked byte-for-byte against committed golden JSON, including the sample. DAT_10–DAT_13
Round-trip import CSV/JSON/Parquet/SQLite re-import of the same unchanged source must produce identical canonical IR (schema, profile, sample) on every run — this domain has no export round-trip in v0 (§1 has no PDF-style export target), so this tier tests import-determinism specifically. DAT_20–DAT_23
Semantic diff One test per §4 category (schema add/remove/retype, ambiguous rename, profile/statistics, row-count/shape, sample, quality-rule-vs-result, lineage, relationship, source, staleness-flagged diff). DAT_30–DAT_39
Merge One test per §5 case, incl. the profile-always-recomputed-not-merged rule, the ambiguous-rename-collision-with-independent-edit case, and lineage-disagreement conflict. DAT_50–DAT_58
Lineage graph tests Upstream/downstream traversal correctness against a fixture with a multi-hop transformation chain, including depth-limit boundary behavior. DAT_60, DAT_61
Query engine One test per remaining §6 example (failing-rules, schema-diff convenience, distribution-shift query, broken-relationships). DAT_70–DAT_73
Property-based Randomized schema-mutation generation asserting ambiguous-rename detection fires exactly when it should (a disappear+appear pair with high name similarity) and never on unrelated add/remove pairs; randomized numeric-column generation asserting ColumnStatistic computation is correct against known distributions. DAT_80, DAT_81
Validation/drift One test per §7 bullet (schema-drift prominent flag, profile-staleness escalation logic, quality-error blocking vs. warning non-blocking, duplicate-PK blocking, type-inference-confidence flag, non-deterministic import, schema mismatch). DAT_90–DAT_96

Fixture strategy using representative datasets

Fixtures use real, small, permissively-licensed public datasets (e.g. a well- known small CSV like Iris or Titanic for the inference/profile path, a small Parquet export with an explicit schema, a small SQLite database with a genuine foreign key relationship) — real source quirks (mixed-case headers, inconsistent null representations in CSV, nested-but-mostly-flat JSON) are exactly what §3's normalization and §7's validation need to handle, same "realistic tool output" reasoning as every prior plan. A minimal library: a clean typed Parquet table; a messy untyped CSV requiring type inference with at least one ambiguous column; a SQLite database with two related tables and a foreign key; a JSONL file with variable-shape records; a dataset with a deliberately introduced quality-rule violation (for the blocking-vs-warning severity test); a two-hop lineage chain (source → transform → derived dataset) for traversal tests. Property-based tests use synthetic schema/distribution generators, not the fixture library, for systematic rename-ambiguity and statistics-correctness coverage.

9. Backend Integration

Domain plugin implementation

muse/plugins/dataset/ following the muse/plugins/midi/ layout:

muse/plugins/dataset/
  __init__.py
  plugin.py             # snapshot/diff/merge/drift/apply/schema
  entity.py             # Canonical model dataclasses (§2)
  manifest.py           # SnapshotManifest (de)serialization, normalization (§3)
  dataset_diff.py        # Diff algorithms (§4)
  dataset_merge.py        # Merge algorithms (§5)
  profiling.py            # Schema inference + statistics computation (via DuckDB)
  _query.py              # Query DSL + lineage traversal (§6)
  _invariants.py         # Validation/drift checks (§7)
  _import_tabular.py      # CSV/JSON/JSONL/Parquet -> IR adapter (via Arrow)
  _import_sql.py          # SQLite/DuckDB -> IR adapter (native introspection)

schema() declares (mirroring every prior plan's domains --json shape):

{
  "schema_version": "<muse_version>",
  "merge_mode": "three_way",
  "description": "Semantic version control for structured and semi-structured datasets...",
  "dimensions": [
    {"name": "schema", "description": "Columns, types, and relationships."},
    {"name": "profile", "description": "Row counts and column-level statistics."},
    {"name": "sample", "description": "Bounded, deterministic sample records."},
    {"name": "quality", "description": "Quality rules and their evaluation results."},
    {"name": "lineage", "description": "Upstream/downstream transformation provenance."},
    {"name": "source", "description": "External data source references."},
    {"name": "metadata", "description": "Curator, source format, schema version."}
  ]
}

Storage implications

Full row data is never stored (§1, §3) — only schema/profile/sample, kilobytes to low megabytes regardless of underlying dataset size. .muse/objects/ footprint is small and bounded, similar in spirit to the Video Timelines and Biological Pathways plans' lightweight storage footprint; muse count-objects/gc/prune need no dataset- specific changes.

API endpoints needed (MuseHub)

  • GET /:owner/:repo/dataset/:ref/schema — schema + profile summary at a ref.
  • GET /:owner/:repo/dataset/:ref/diff?base=&target= — structured diff for proposal review rendering, including the staleness-flagged-comparison callout (§4).
  • GET /:owner/:repo/dataset/:ref/lineage?direction=&depth= — server-side lineage traversal, backing §6's CLI commands for UI use.
  • GET /hub/search/datasets?upstream_of=&format=&min_rows= — cross-repo search endpoint (§6), same architectural pattern the Research Papers/Biological Pathways plans required.

CLI implications

New muse dataset subcommand group: muse dataset query, muse dataset failing-rules, muse dataset schema-diff, muse dataset lineage, muse dataset broken-relationships, muse dataset sample. All accept --json.

MuseHub UI implications

  • Proposal/diff view needs a dataset diff renderer: a schema-change table (added/ removed/retyped columns highlighted), a statistics-delta panel (distribution shifts visualized, not just numeric deltas), and a conflict panel in plain language (§5).
  • Repo browse view needs a schema/profile summary view (column list with types and key stats) plus the bounded sample rendered as an actual table — the closest this domain gets to "read the data" without resolving the full external source.

Rich data visualization

  • Column-level distribution visualization (histograms for numeric columns, top- values bar charts for categorical columns) computed from ColumnStatistic, not from re-scanning the full dataset — the profile is the visualization's data source, reinforcing why the profile is a first-class committed artifact (§2), not a throwaway computation.
  • Lineage graph visualization: nodes = datasets/sources, edges = LineageEdge transformations — directly analogous to the Biological Pathways plan's citation/ regulatory graph view and the Research Papers plan's citation graph view.

MCP/tooling implications

Expose dataset_query, dataset_diff, dataset_lineage, dataset_failing_rules as MCP tools, mirroring every prior plan's exposure pattern — lets agentception workers check "did this change introduce a schema-breaking column removal" or "does this dataset still pass its quality gates" before a merge or downstream pipeline run.

10. Future Semantic Extensions

Named here so the v0 scope boundary (§1) is legible as a deliberate step, not a ceiling:

  • Full row-level diff/merge: actually diffing which specific rows changed (not just schema/profile/sample) — requires either bounding dataset size assumptions much more tightly than v0, or a fundamentally different storage strategy (e.g. content-addressed row chunks, closer to the CAD/3D-Scenes plans' blob model than the external-reference model v0 uses) — a substantial follow-on design, not a simple extension.
  • Live warehouse connections: querying a running BigQuery/Snowflake/Postgres instance directly, with credential/connection management this domain doesn't need to own for static file/export sources.
  • Automated transformation lineage extraction: statically analyzing a dbt project or SQL script to auto-populate LineageEdges rather than requiring them to be asserted (§1) — structurally similar to the Research Papers plan's deferred claim-extraction and the Biological Pathways plan's deferred AI-assisted construction.
  • PII/sensitive-data detection: automated scanning for personally identifiable or regulated data categories, with its own correctness bar given compliance stakes (§1) — deliberately not attempted casually as a v0 add-on.
  • Data contracts: formalizing QualityRules into enforceable, versioned contracts between data producers and consumers, with breaking-change detection integrated into CI — a natural extension of §7's schema-drift and quality-failure checks into an explicit producer/consumer agreement model.
  • Dataset-to-code lineage: linking a dataset's LineageEdge transformations to the actual code-domain symbols that implement them, connecting this domain to Muse's existing code-intelligence layer.
  • Statistical drift alerting over time: trend analysis across a dataset's full commit history (not just base-vs-target diff) to catch slow distributional drift a single two-commit diff wouldn't flag.
  • Multi-table join-aware diff: extending §1's referential-integrity validation into full relational diff/merge across tables within one dataset.
  • Synthetic/anonymized sample generation: generating a realistic but privacy- safe Sample for datasets containing sensitive data, rather than v0's direct sampling from the real source.
  • Dataset-to-paper/pathway cross-linking: this domain is exactly what the Research Papers plan's Dataset element and the Biological Pathways plan's experimental data ultimately point at — a v1+ direction is making that reference bidirectional and queryable ("which papers/pathways use this exact dataset version"), not just a one-way pointer as those plans currently define it.

11. Staging Deliverables

Proposed issue breakdown

  1. DAT-1 — Canonical model + snapshot + profiling (§2, §3): entity dataclasses, schema inference, DuckDB-backed profiling/sampling, golden-snapshot tests (DAT_01–DAT_13). Load-bearing — nothing else starts until this lands.
  2. DAT-2 — SQLite/DuckDB native introspection adapter (§1, §3): depends on DAT-1. DAT_20–DAT_23.
  3. DAT-3 — Semantic diff (§4): depends on DAT-1. DAT_30–DAT_39.
  4. DAT-4 — Merge strategy (§5): depends on DAT-1, DAT-3. DAT_50–DAT_58.
  5. DAT-5 — Lineage graph + traversal (§6 lineage portion): depends on DAT-1; parallel with DAT-3/DAT-4. DAT_60, DAT_61.
  6. DAT-6 — Query DSL + CLI (§6, §9): depends on DAT-1, DAT-3. DAT_70–DAT_73.
  7. DAT-7 — Property-based rename/statistics suite (§8): depends on DAT-1 only. DAT_80, DAT_81.
  8. DAT-8 — Remaining validation & drift checks (§7): depends on DAT-1; parallel with DAT-3/DAT-4/DAT-6. DAT_90–DAT_96.
  9. DAT-9 — MuseHub API endpoints + cross-repo dataset search (§9): depends on DAT-1, DAT-3, DAT-5.
  10. DAT-10 — MuseHub UI: schema/profile browser, diff renderer, lineage graph (§9): depends on DAT-9. Not scoped in detail until DAT-3/DAT-4 output shapes are proven.
  11. DAT-11 — MCP tooling exposure (§9): depends on DAT-6.

Milestones

  • M1 (Foundation): DAT-1 merged, muse domains --json lists dataset as active, golden-snapshot suite green including seeded-sample determinism.
  • M2 (Interop): DAT-2 merged; SQLite/DuckDB introspection fixtures produce authoritative (non-inferred) schema correctly.
  • M3 (Semantic core): DAT-3 + DAT-4 + DAT-5 merged; a two-branch conflict fixture (both branches retype the same column differently) produces a correct conflict report; the profile-always-recomputed merge rule and ambiguous-rename- collision handling verified working as designed.
  • M4 (Usable): DAT-6 + DAT-7 + DAT-8 merged; muse dataset failing-rules/ schema-diff/lineage/broken-relationships all answer correctly against fixtures.
  • M5 (Ecosystem): DAT-9, DAT-10, DAT-11 merged; a dataset proposal on staging MuseHub renders a real schema/statistics diff with a lineage graph view.

Acceptance criteria (whole-plan level)

  • muse domains --json shows dataset with active: true and the schema in §9.
  • Re-snapshotting an unchanged fixture (CSV/Parquet/SQLite/JSONL) twice produces byte-identical canonical JSON, including the sample.
  • The ambiguous-column-rename fixture is flagged as ambiguous, not silently resolved as a confident rename, with a similarity-score hint present in the diff output.
  • The quality-rule-violation fixture (severity error) is commit-blocking; a warning-severity violation is flagged but not blocking; well-formed fixtures produce no false positives.
  • A merge of disjoint schema/quality-rule edits auto-resolves with zero conflicts; a merge where both branches retype the same column differently reports a correctly-worded conflict; the merged result's Profile is recomputed fresh against the merged source, never field-merged from the two branches' stale profiles.
  • muse dataset lineage --direction upstream/downstream correctly traverses a multi-hop fixture, including depth-limit boundary behavior.
  • No test depends on network access, a specific Arrow/DuckDB library version's exact output, or wall-clock time (§3/§7 determinism requirement), except where a seed is explicitly recorded to make otherwise-nondeterministic sampling reproducible.

Risks

  • Column-rename ambiguity risk: unlike every prior plan's rename detection (which has some content-independent signal — geometry, ontology mapping, DOI — to anchor identity), a dataset column's identity is purely nominal, making rename detection structurally ambiguous, not just occasionally hard. Mitigation: this plan treats ambiguity as a first-class, always-visible state (§3, §4, §7) rather than attempting to guess confidently — the risk is that downstream UI/tooling treats the similarity-score hint as more authoritative than it is; must be presented as a hint requiring human confirmation, never auto-applied.
  • Profiling cost risk: computing full-column statistics (distinct counts, quantiles) on very large sources can be expensive even via DuckDB's optimized query engine. Mitigation: ColumnStatistic fields are individually nullable (§2) specifically so v0 can skip expensive statistics on very large sources without blocking snapshot entirely — this must be a real, tested code path (§8), not an unused escape hatch.
  • Type-inference risk: CSV/JSON type inference is inherently heuristic and can be wrong in ways that only surface later (e.g. a column that's "mostly numeric" with a few outlier strings). Mitigation: Column.inferred and the type-inference- confidence drift check (§7) make this visible rather than silently trusted — but real-world messy data will stress-test this more than the curated fixture library can, so this should be watched closely post-M1.
  • Cross-repo search risk: same architectural risk class every prior plan with cross-repo search flagged — new MuseHub API surface, not just a domain-plugin concern, should be scoped carefully in DAT-9 rather than assumed to be a thin wrapper, and ideally shares a design with the Research Papers/Biological Pathways plans' equivalent search endpoints rather than three independent implementations.

Open questions

  • What specific name-similarity threshold should the ambiguous-rename hint use (§3, §4)? Needs calibration against real schema-evolution examples before DAT-1 is finalized, not guessed — and must remain a hint, never a confidence threshold that silently flips into an auto-resolved rename.
  • Should Profile/ColumnStatistic computation have a configurable size/cost budget (e.g. skip distinct-count above N rows) as a first-class schema field rather than an implicit implementation detail? Leaning toward yes, given the profiling-cost risk above — needs a decision before DAT-1.
  • Should this domain's cross-repo search (§9) share one implementation with the Research Papers and Biological Pathways plans' equivalent endpoints, given all three raised the identical architectural need? Worth resolving once, not three times, ideally before any of the three DAT-9/PAP-8/BIO-9-equivalent issues start.
  • Where is the line between "lineage assertion" (v0, §1) and "lineage extraction" (§10, deferred) likely to actually sit once real usage data exists — should DAT-5 include even a minimal, best-effort extraction hook (e.g. parsing a dbt manifest.json if one is present) as a convenience, or should that wait entirely for a dedicated v1 effort? Flagged for a decision during DAT-5 scoping.
  • Should dataframe snapshotting (§1) be a first v0 CLI-invocable path, or only a library-level entrypoint for now (i.e., no muse dataset snapshot-dataframe CLI command until there's a concrete consumer)? Leaning toward library-only for v0, flagged for confirmation before DAT-1 scoping locks.

Implementation order

DAT-1 → (DAT-2, DAT-3, and DAT-7 in parallel) → DAT-4 → DAT-5 → DAT-6 → DAT-8 → DAT-9 → (DAT-10 and DAT-11 in parallel). Mirrors §1's dependency structure: nothing can diff, merge, validate, or query state that hasn't been normalized into the canonical model yet.

Plan-only issue. Each deliverable in §11 should be filed as its own staging issue once reviewed, referencing the test IDs and criteria above.

◎ Activity
●
gabriel opened this issue 19 days ago
◎
No activity yet. Use the CLI to comment.