gabriel / muse public
Open #67 Enhancement
filed by gabriel human · 19 days ago

3D Scenes Domain — implementation plan (TDD-first, comprehensive)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

3D Scenes Domain — Implementation Plan

Background

Muse treats "domain" as a first-class primitive (muse/domain.py::MuseDomainPlugin). A domain plugin implements six methods — snapshot, diff, merge, drift, apply, schema — and gets the full VCS engine for free: the commit DAG, branching, checkout, lineage walking, log graph, and merge-base finder. muse.plugins.midi proved the model for structured symbolic + binary-payload data. muse.plugins.code proved it for AST symbols. muse.plugins.social/identity proved it for graph-shaped state. The CAD domain plan (staging issue #66) is proving it for continuous geometry with causal feature histories and solvable constraint systems.

3D scenes are a distinct fourth shape of problem: a hierarchical scene graph (nodes nested arbitrarily deep, each carrying a local transform composed with its ancestors), referencing a library of shared assets (meshes, materials, textures) that are frequently instanced many times, plus time-varying data (animation timelines, skeletal rigs) layered on top of the static graph. Today, every engine and DCC tool (Unity, Unreal, Blender, Maya) either owns this as an opaque proprietary project file or exports to an interchange format (glTF, USD, OBJ, FBX) that Muse — or any VCS — treats as an opaque binary blob: a scene with one node moved diffs identically to a scene with every node rebuilt from scratch. If Muse can version, diff, merge, validate, and query a 3D scene graph the way it already does for code, MIDI, and (per the CAD plan) parametric geometry, it proves the domain-plugin model generalizes across spatial/hierarchical data as well as symbolic and continuous-geometric data — a third and meaningfully different axis from CAD's single-part feature history.

This issue is the plan only. No implementation code lands against this issue. Each numbered phase in §10 is expected to become its own staging issue once this plan is reviewed and accepted.

Goal

Ship a muse/plugins/scene3d/ domain plugin, structured like muse/plugins/midi/ and the planned muse/plugins/cad/ (a plugin.py entrypoint plus focused private modules for diff/merge/query/invariants/import-export), that:

  • Registers in muse domains --json alongside code, midi, identity, mist, social, and (once landed) cad.
  • Passes muse code test --json for every module it touches.
  • Snapshots, diffs, and merges a 3D scene with node-level, transform-level, material- level, and animation-level semantic awareness — never as an opaque .glb/.usd blob.
  • Supports a scene query DSL (muse scene query "...", mirroring muse code query and the CAD plan's muse cad query) capable of answering the concrete query examples in §6.
  • Detects the drift/validation classes in §7 via muse status / muse check.
  • Has test coverage across every tier in §8, with test IDs (SCN_01, SCN_02, …) referenced from both this plan and the test files that satisfy them.

"Done" for the plan itself means: reviewed, decomposed into the issue breakdown in §10, with acceptance criteria concrete enough for a different agent to implement against test-first without re-deriving decisions made here.

1. Domain Scope

Supported initial 3D scene targets

  • Static and lightly-animated scenes built from a node hierarchy, mesh/material/ texture asset library, cameras, and lights — the common denominator across glTF, USD, and every major DCC tool's internal scene representation.
  • v0 targets scenes in the tens to low-hundreds of nodes / tens of unique meshes range (a game level chunk, a product visualization scene, an architectural walkthrough) — not open-world-scale streaming scenes.

Same two-layer strategy as the CAD plan (§1 of issue #66), adapted for scenes:

  1. Canonical semantic layer (Muse-owned): a JSON-serializable scene IR — the "Canonical Model" in §2 — that Muse snapshots, diffs, merges, and queries directly.
  2. Interchange/import-export boundary: glTF 2.0 (.gltf/.glb) as the primary MVP interchange format, because it is (a) an open Khronos standard, (b) the closest thing the 3D industry has to "the STEP of real-time scenes" — nearly every engine and DCC tool can export/import it losslessly for the v0 feature set (nodes, transforms, meshes, PBR materials, textures, cameras, basic skeletal animation), and (c) already JSON + binary-buffer shaped, which maps unusually cleanly onto Muse's snapshot-as-JSON-plus-content-addressed-blobs model. USD (Universal Scene Description) is the recommended v1 follow-up target, not v0: USD's composition arcs (layers, references, variants) are semantically richer than anything glTF expresses and deserve their own design pass once the v0 IR is proven against glTF's simpler model — attempting both in v0 risks the IR being accidentally shaped around USD's composition semantics before the basics are solid.

Relationship to glTF, USD, OBJ, Blender, and game-engine scenes

Format/tool Role in this plan
glTF/GLB v0 primary interchange format — import/export adapter, round-trip target.
USD Explicitly deferred to v1 (see above). The canonical IR's node/transform/asset-reference model is designed to be USD-composition-compatible later (e.g. ExternalReference in §2 is deliberately generic enough to model a USD reference/payload arc), but v0 does not implement USD import/export.
OBJ Import-only, mesh-and-materials-only adapter (no hierarchy, no animation) — useful as the simplest possible fixture source for geometry-diff tests, not a first-class round-trip target.
Blender Recommended reference authoring tool for v0 fixtures (open source, scriptable via Python, has a mature glTF exporter) — same role FreeCAD plays for the CAD plan. Blender's native .blend is not a v0 import/export target (proprietary binary chunk format, no stable public spec) — only its glTF export path is used.
Game engines (Unity/Unreal/Godot) Out of scope as direct import/export targets in v0. Engine scenes typically already have their own project-level VCS story (or lack thereof) and engine-specific concepts (prefab overrides, lightmap bakes, navmesh data) that are a distinct, larger scope. glTF remains the interop boundary: an engine that can export/import glTF can round-trip through this domain even without a native adapter.

What is in scope for v0

  • Node hierarchy with local transforms (translation/rotation/scale), arbitrary nesting depth, instancing (multiple nodes referencing the same mesh/material).
  • Meshes: triangle-mesh geometry (positions, normals, UVs, vertex colors), stored content-addressed.
  • Materials: PBR metallic-roughness model (glTF's core material model) — base color, metallic, roughness, emissive, normal/occlusion maps.
  • Textures: 2D image textures referenced by materials, content-addressed.
  • Cameras: perspective and orthographic, with standard parameters (FOV, near/far, aspect).
  • Lights: point, directional, spot (glTF's KHR_lights_punctual extension model).
  • Animations: keyframe-based transform animation (translation/rotation/scale curves) and simple skeletal (joint-hierarchy) animation.
  • Skeletons/rigs: joint hierarchy + skin weights, glTF's skinning model.
  • Environment settings: background/ambient light representation, scene-level unit and up-axis declaration.
  • Asset references: both embedded (single-file GLB) and external (separate texture/ buffer files) reference resolution.
  • Metadata and custom properties: arbitrary key/value extras per node/scene, mirroring glTF's extras mechanism.

What is explicitly out of scope for v0

  • USD import/export and USD composition semantics (deferred to v1, see above).
  • Physically-based simulation state (cloth, rigid body, soft body) beyond declaring the presence of physics metadata as opaque, content-hashed data (§2) — no semantic diff/merge of simulation parameters in v0.
  • Particle systems, procedural/volumetric materials (shader graphs), and node-based material editors' underlying graphs — v0 materials are limited to the flattened PBR parameter set glTF already standardizes.
  • Level-of-detail (LOD) chains and streaming/paging scene structures.
  • Full skeletal animation retargeting or inverse-kinematics rig metadata — v0 stores and diffs the joint hierarchy and keyframe curves, not IK constraint solving.
  • Lightmap/baked lighting data and navmesh data (engine-specific, not scene-graph semantic content in the sense this domain targets).
  • Multi-thousand-node open-world scenes and any performance work targeting that scale.
  • Real-time collaborative (CRDT) scene editing — v0 merge is offline/three-way only (merge_mode: three_way), consistent with code/midi/the CAD plan, not social's CRDT mode.

2. Canonical Model

All elements are content-addressed (sha256:<hex>) and referenced by ID, never positional index — the same identity principle used by code's AST symbols, midi's notes, and the CAD plan's Part/Feature/Body elements — so that scene-graph reordering never produces a spurious diff.

Element Fields Notes
Scene id, name, root_nodes: [Node.id], up_axis (y_up|z_up), unit_scale (meters-per-unit), environment: Environment.id, metadata Top-level container; a document can hold multiple Scenes (glTF supports this) but v0 tooling operates on one active scene at a time.
Node id, name, parent_ref: Node.id \| null, transform: Transform.id, children: [Node.id] (ordered for rendering purposes only — not identity-bearing), mesh_ref: Mesh.id \| null, camera_ref: Camera.id \| null, light_ref: Light.id \| null, skin_ref: Skeleton.id \| null, tags: [string], extras: {key: value} A node's identity is content-derived from (name, parent content-identity, defining refs) when no stable external ID exists — see §3.
Transform id, translation: [x,y,z], rotation: [x,y,z,w] (quaternion, not Euler — avoids gimbal-order ambiguity across tools), scale: [x,y,z] Content-addressed by value; two nodes with identical transforms share one Transform object, mirroring the CAD plan's Material dedup pattern.
Mesh id, primitives: [{geometry_ref: content_id, material_ref: Material.id, mode: triangles}], bounding_box, vertex_count, triangle_count Geometry bytes (positions/normals/UVs/colors buffers) live in the object store; geometry_ref is a pointer, mirroring the CAD plan's Body.geometry_ref. vertex_count/triangle_count are cheap pre-filter fields for diff (§4), same role as CAD's topology_summary.
Material id, name, base_color, metallic, roughness, emissive, base_color_texture: Texture.id \| null, normal_texture: Texture.id \| null, occlusion_texture: Texture.id \| null, metallic_roughness_texture: Texture.id \| null, alpha_mode, double_sided: bool Content-addressed by (all fields) so identical materials across many nodes collapse to one object, same dedup principle as CAD Materials.
Texture id, image_ref: content_id, sampler (wrap/filter settings), source_format (png|jpeg|ktx2) Image bytes are a separate content-addressed blob from the Texture metadata wrapper, so two textures with different sampler settings but the same image share one image blob.
Camera id, kind (perspective|orthographic), fov_or_extent, near, far, aspect_ratio
Light id, kind (point|directional|spot), color, intensity, range, spot_angles (inner/outer cone, spot only) Mirrors glTF's KHR_lights_punctual.
Animation id, name, channels: [{target_node: Node.id, target_property: translation\|rotation\|scale\|weights, sampler_ref: AnimationSampler.id}], duration
AnimationSampler id, interpolation (linear|step|cubic_spline), keyframes_ref: content_id (time+value buffer, content-addressed like geometry) Keyframe buffers are binary payloads, same storage treatment as mesh geometry (§3).
Skeleton id, joints: [Node.id] (ordered, joint hierarchy is expressed via each joint Node's own parent_ref), inverse_bind_matrices_ref: content_id, skin_weights_ref: content_id (per-vertex joint indices + weights, keyed to a Mesh) Joints are ordinary Nodes flagged by skeleton membership — avoids a parallel hierarchy concept.
PhysicsMetadata id, kind (opaque_v0), content_ref: content_id, engine_hint (unspecified|tool name) v0 stores this as an opaque, content-hashed blob per §1 (diffed only as changed/unchanged, never semantically) — a deliberate scope boundary, not an oversight.
Environment id, ambient_color, background: {kind: color\|skybox_texture, ref}, metadata
ExternalReference id, kind (linked_texture, linked_buffer, imported_gltf, usd_reference_arc [reserved for v1]), uri_or_path, content_id_at_link_time, resolved: bool Same role as the CAD plan's ExternalReference — required for drift detection (§7).
Metadata units (unit_scale lives on Scene; this covers authoring metadata), author, created_at, source_format (gltf|glb|obj), source_checksum, schema_version

Design invariant carried over from code/midi/the CAD plan: identity is content- and reference-based, never positional. Reordering a node's position in its parent's children list is not itself a diff; only a change in parent_ref, transform, or any *_ref field is.

3. Snapshot Design

Deterministic scene normalization

snapshot() walks the live state (a .gltf/.glb bundle under state/, same pattern as MIDI's .mid walk and the CAD plan's .FCStd/STEP walk) and produces the canonical IR in §2 as the SnapshotManifest, applying normalization in a fixed order:

  1. Resolve every element to a stable ID (§3, Stable node identity, below).
  2. Canonicalize numeric values: round to a fixed epsilon-safe precision (default 1e-6 for normalized transform components after unit-scale normalization — looser than the CAD plan's 1e-9 because real-time scene data typically already round-trips through 32-bit floats, and matching CAD's tighter epsilon would manufacture false diffs from ordinary float32 precision loss).
  3. Canonicalize ordering: node children lists, animation channels, and asset library entries are sorted by ID for serialization — never by file insertion order — so a no-op re-export never diffs.
  4. Compute bounding_box, vertex_count, triangle_count for every Mesh at snapshot time (cheap invariants that short-circuit expensive geometry diffs, §4).

Stable node identity

Scene nodes usually have no globally stable ID in glTF/OBJ (glTF nodes are array- indexed; a re-export can silently renumber every node). Node identity is therefore derived as a content hash of (name, parent_identity, mesh_ref content-id, camera_ref/light_ref content-id) — i.e. identity is defined recursively from the root down, so a subtree that is structurally unchanged keeps the same IDs even if its array index shifts. Two caveats made explicit here rather than discovered during implementation:

  • Two visually-identical sibling nodes with the same name (e.g. ten instances of "Tree" under the same parent) are only disambiguated by their differing mesh_ref/ transform/child content — if two siblings are fully identical in every identity-bearing field, they are indistinguishable by content hash alone, and the plugin must fall back to a stable positional tiebreaker (declared explicitly, not silently) for that specific collision case. This is a known limitation to test directly (§8) rather than paper over.
  • Renaming a node changes its own content identity (since name feeds the hash) and every descendant's content identity (since each descendant's hash includes parent_identity). This would defeat rename detection if handled naively. The fix: node identity is split into two hashes — a structural identity (excludes name, used for diff/merge address-keying and rename detection) and a display identity (includes name, used only for human-readable output) — so renaming a node is detected as "structural identity unchanged, name changed" (a RenameOp, §4) rather than cascading as a delete+insert through the entire subtree.

Content-addressed geometry

Each Mesh primitive's vertex/index buffers are extracted as their own binary blob and hashed independently (sha256:<hex> over the canonical binary layout — fixed attribute order: position, normal, UV0, color, joints, weights — so two tools exporting the same mesh with different internal buffer interleaving still converge to the same hash after canonicalization). Identical meshes instanced many times (the common case in scenes — a rock mesh used 200 times) collapse to one stored blob, the same content-addressing win as the CAD plan's duplicate-geometry handling and MIDI's duplicate note-buffer handling.

Content-addressed textures and binary assets

Texture image bytes are hashed and stored independently from the Texture metadata wrapper (§2) — same blob-versus-metadata separation as geometry. Animation keyframe buffers are likewise content-addressed binary blobs, not inlined into the JSON IR.

External reference handling

glTF's separate (.gltf + .bin + loose image files) form and single-file (.glb) form are normalized to the same canonical IR — the choice of container format is a serialization detail, not semantic content. Every external file reference becomes an ExternalReference entry (§2) with resolved tracked, so muse status/drift (§7) can flag a texture file that went missing between commits regardless of which container form was originally used.

Large object storage strategy

Same decision as the CAD plan (§3 of issue #66): binary blobs (geometry, textures, keyframe buffers) go into .muse/objects/ — no new storage subsystem in v0. Textures in particular can be large (multi-megabyte 4K images); an LFS-style side channel is explicitly deferred as an open question (§10), contingent on real repo-size data from v0 usage, not designed speculatively now. Existing muse gc/muse prune need no scene-specific changes.

Unit and coordinate-system normalization

  • Unit: Scene.unit_scale (meters-per-unit) is read from the source format's declared unit (glTF is always meters; OBJ has no unit convention and defaults to "unspecified," which is flagged, not silently assumed — see §7) and normalized to meters internally. All diffs/merges operate in normalized units; on apply()/ checkout, values are converted back to the target format's convention.
  • Coordinate system: Scene.up_axis is read from the source format (glTF is always Y-up; Blender's native convention is Z-up, but its glTF exporter already converts to Y-up, so v0's glTF-only import boundary avoids needing an internal axis-conversion step — this is one of the reasons glTF, not a DCC-native format, is the v0 interchange choice). The up_axis field is still stored explicitly (not hardcoded to Y-up) so a future USD (v1) adapter — where Z-up is common — has a place to record it without a schema change.

4. Semantic Diff

diff(base, target, *, repo_root=None) returns a StructuredDelta of typed ops, same contract as MIDI/code/the CAD plan.

  • Node add/remove/rename/move diffs: InsertOp/DeleteOp keyed on a node's structural identity (§3). A RenameOp fires when structural identity is unchanged but display identity (name) differs. A MoveOp fires when a node's structural identity subtree is found under a different parent_ref than in base — mirrors the CAD plan's feature/part move detection.
  • Transform diffs: ReplaceOp on Node.transform, reported as which component changed (translation/rotation/scale) with before/after values — not a raw 4x4 matrix diff, which is unreadable. Below-epsilon changes (§3) are not reported at all.
  • Geometry diffs: only computed when a Mesh primitive's geometry_ref content ID changes. Cheap pre-filter first: if vertex_count/triangle_count/ bounding_box are all unchanged, geometry is reported unchanged without loading the buffer. If they differ, report at the coarsest useful level (vertex/triangle count delta, bounding-box delta) — v0 does not attempt per-vertex correspondence diffing between two arbitrary meshes (a research-grade problem, same reasoning as the CAD plan's face-correspondence scope cut); it reports "geometry changed, here is what changed about it structurally."
  • Material and texture diffs: Material diffed at the field level (base color, metallic, roughness, etc. individually) since materials are small and this level of granularity is cheap and useful (unlike geometry). Texture diffs report whether the image changed (content ID) separately from whether sampler settings changed — a texture swap and a wrap-mode tweak are different kinds of change and should read differently in a diff.
  • Lighting and camera diffs: field-level ReplaceOps on Light/Camera properties, same treatment as materials — these are small, fully-diffable elements with no geometry-scale concerns.
  • Animation timeline diffs: diffed at the Animation.channels level first (added/removed/retargeted channels are structural changes) and then, for a channel whose sampler_ref content changed, reported as "keyframe data changed" with a coarse summary (keyframe count delta, time-range delta) rather than a per-keyframe diff — mirrors the CAD plan's decision not to diff geometry buffers keyframe-by- keyframe/vertex-by-vertex.
  • Rig/skeleton diffs: Skeleton.joints list diffed by joint Node structural identity (added/removed joints are structural; a changed inverse_bind_matrices_ref or skin_weights_ref is reported as "binding data changed" at the coarse blob level, consistent with geometry/animation buffer treatment above).
  • Asset reference diffs: ExternalReference diffed by id; a reference whose resolved flag flips, or whose content_id_at_link_time changes (the linked asset was updated externally), is reported distinctly from a reference being added/ removed outright.
  • Tolerance-aware floating-point comparisons: every numeric comparison (transform components, bounding boxes, material scalar values, camera FOV, keyframe times/ values) uses the fixed epsilon from snapshot normalization (§3). This is what prevents float32 precision jitter from a re-export in a different tool producing spurious no-op diffs — directly testable via round-trip fixtures (§8).

5. Merge Strategy

merge(base, left, right, *, repo_root=None) — three-way, merge_mode: three_way, consistent with code/midi/the CAD plan.

  • Non-conflicting scene graph edits (the common case): edits to disjoint nodes/subtrees — one branch adds a new light, the other renames an unrelated mesh — merge automatically via the engine's default address-keyed map merge, driven by the dimensions declared in schema() (§9).
  • Same-node transform conflicts: both branches change the same Node.transform to different values — always a reported conflict in v0 (no auto-averaging of positions, same reasoning as the CAD plan's rejection of auto-averaging dimension values: silently picking a value neither author chose is worse than asking).
  • Geometry conflicts: both branches change the same Mesh's geometry_ref to different content — reported conflict; since v0 has no per-vertex diff (§4), there is no attempt at a semantic geometry merge, only a binary "pick one / manual resolve" conflict, explicitly weaker than the CAD plan's feature-level merge (which can merge disjoint feature edits even when they affect the same body indirectly) — this asymmetry is intentional and called out here: scenes have no equivalent of CAD's causal feature history to reason about, so mesh-level conflicts are coarser by necessity.
  • Material conflicts: field-level — if both branches change different fields of the same Material (left changes base_color, right changes roughness), this merges automatically (field-level granularity pays off here, unlike geometry); only same-field edits with different values are a reported conflict.
  • Animation conflicts: both branches modify the same Animation.channels entry's sampler_ref — reported conflict. Both branches adding different new channels to the same Animation merges automatically (disjoint channel additions).
  • Asset reference conflicts: both branches point the same ExternalReference at different uri_or_path/content — reported conflict with both target values shown.
  • Coordinate-system conflicts: both branches change Scene.up_axis or unit_scale — always a reported conflict (these are scene-global, not per-element, so there is no meaningful "auto-merge" of two different global conventions); this mirrors why the CAD plan treats Metadata.units mismatches as a hard validation gate (§7) rather than something merge silently reconciles.
  • Human-readable conflict reports: every conflict in the MergeResult carries which element(s), both branches' values, the ancestor base value, and a plain- language cause string (e.g. "Both branches moved node 'Camera_Main': left=(0,5,10), right=(2,5,8), base=(0,0,0)"), consistent with the workspace-wide three-way conflict-marker convention and the CAD plan's identical requirement.

6. Query Support

A scene query DSL mirroring muse code query and the CAD plan's muse cad query (muse scene query "<predicate>" --json), plus purpose-built subcommands for the most common questions.

Example 3D scene query DSL

muse scene query "node.type == 'mesh' and node.tag == 'foliage'" --json
muse scene query "mesh.triangle_count > 50000" --json
muse scene query "material.base_color_texture == null" --json

Find nodes by type, name, tag, material, or bounding box

muse scene query "node.type == 'light'" --json
muse scene query "node.name =~ 'Door_*'" --json
muse scene query "node.tag == 'interactive'" --json
muse scene query "node.mesh.material == 'Glass_01'" --json
muse scene find-in-bounds --min 0,0,0 --max 10,5,10 --json

Find high-poly meshes

muse scene query "mesh.triangle_count > 100000" --json
muse scene high-poly --threshold 100000 --json     # convenience alias, mirrors `muse code hotspots` framing

Find missing textures

muse scene query "texture.resolved == false" --json
muse scene missing-textures --json

Find unreferenced assets

muse scene query "asset.id not in referenced_by(scene)" --json
muse scene unreferenced-assets --json    # nearly free given content-addressing, mirrors CAD's duplicate-geometry query

Find changed transforms

muse scene query "node.transform.modified_since == HEAD~5" --json
muse scene changed-transforms --since v1.2.0 --json

Find lights/cameras

muse scene query "node.type in ('light', 'camera')" --json
muse scene list-lights --json
muse scene list-cameras --json

Find animated nodes

muse scene query "node.id in animated_targets()" --json
muse scene animated-nodes --json

Find objects within spatial regions

muse scene find-in-bounds --min -5,-5,-5 --max 5,5,5 --json
muse scene query "node.world_bounding_box intersects (-5,-5,-5)-(5,5,5)" --json

7. Validation and Drift Detection

Surfaces through muse status (via drift()) and muse check/muse code invariants (via a scene-specific invariant checker, same pattern as the CAD plan's _invariants.py).

  • Broken asset references: any ExternalReference with resolved: false, or any *_ref field pointing at an element ID not present in the current snapshot (e.g. a node's mesh_ref pointing at a deleted Mesh) — flagged as broken reference, blocks commit by default, same policy as the CAD plan.
  • Invalid transforms: non-finite values (NaN/Inf), zero-length scale on any axis (degenerate/invisible geometry), and non-unit quaternions (a malformed rotation) — detected at snapshot time.
  • Missing textures: a Material referencing a Texture whose image_ref blob is absent from the object store — distinct from a broken external reference (§7, above): this is a broken internal reference within an already-imported snapshot.
  • Unsupported material features: source data using a shader/material feature outside the v0 PBR metallic-roughness model (e.g. a custom node-based shader graph) is flagged as lossy import at snapshot time — the plugin must never silently drop data; unsupported features are preserved as opaque extras and the flag makes the loss visible rather than discovered later as an unexplained visual diff.
  • Non-manifold or malformed geometry: degenerate triangles (zero area), duplicate/ overlapping vertices beyond a tolerance, and inverted-normal detection — run at snapshot time via a mesh-validity pass; flagged as invariant violations, not silently accepted.
  • Unit mismatches: a scene whose source format declares no unit convention (e.g. OBJ) is flagged as unit unspecified rather than silently defaulting to meters — requires an explicit unit_scale declaration (or explicit acceptance of the "unspecified" state) before merge/diff comparisons against a unit-bearing scene are considered meaningful — directly mirrors the CAD plan's unit-inconsistency check and its stated rationale (silent unit bugs are exactly what semantic versioning should catch).
  • Coordinate-system inconsistencies: merging or comparing two snapshots with different up_axis without an explicit, recorded conversion — flagged before diff/ merge proceeds, not silently reconciled (ties to §5's coordinate-system conflict policy).
  • Non-deterministic importer output: re-importing the same source glTF/OBJ file twice must produce identical canonical IR and identical blob hashes (should be guaranteed by §3's normalization, but checked as a regression gate) — a hard error, not a warning, for the same reason as the CAD plan (it would silently break content- addressing).
  • Schema migration risks: Metadata.schema_version mismatches between the plugin's current schema() output and a snapshot's recorded version — flagged before diff/merge across the boundary, never auto-migrated silently, consistent with the workspace-wide rule that muse code migrate requires explicit permission.

8. TDD Plan

Test IDs use the SCN_NN prefix, referenced from both this plan and the eventual test files, per the workspace convention (IC_01, and the CAD plan's CAD_NN).

Tier Coverage Example test IDs
Unit Canonical model (de)serialization for every element type in §2; epsilon-based numeric comparison helper; structural-vs-display identity hashing (including the rename-cascade fix from §3). SCN_01 element round-trip, SCN_02 epsilon comparator boundary values, SCN_03 rename does not cascade to descendants
Golden snapshot Fixed fixtures (single mesh + material, multi-node hierarchy with instancing, a lit scene with camera, a simple skeletal-animated scene) snapshotted once, checked byte-for-byte against committed golden JSON + blob hashes on every run. SCN_10–SCN_14, one per fixture
Round-trip import/export glTF (separate-file form) → IR → glTF and GLB (single-file form) → IR → GLB must reproduce input within tolerance and identical IR; cross-container (.gltf → IR → .glb → IR) must produce identical IR (proves the IR, not the container, is canonical) — direct analog of the CAD plan's cross-format equivalence test. SCN_20 gltf round-trip, SCN_21 glb round-trip, SCN_22 cross-container equivalence
Scene graph diff One test per diff category in §4 (node add/remove/rename/move, transform, geometry, material, texture, light/camera, animation, rig, asset reference, tolerance-suppression-of-noise). SCN_30–SCN_39
Merge One test per non-conflicting case and each conflict category in §5, including the material field-level auto-merge case and the coordinate-system global conflict case. SCN_40–SCN_48
Conflict Conflict report shape/content correctness; muse resolve correctly clears a scene conflict from muse conflicts --json. SCN_50–SCN_53
Floating-point tolerance Property-based/randomized transform and keyframe value generation asserting values within epsilon never diff, and values just outside epsilon always diff (boundary-exact tests, not just spot checks). SCN_60, SCN_61
Validation/drift One test per bullet in §7 (broken reference, invalid transform, missing texture, unsupported material feature, malformed geometry, unit mismatch, coordinate-system inconsistency, non-deterministic import, schema mismatch). SCN_70–SCN_78
Query One test per query example in §6 against a fixture with known expected results. SCN_80–SCN_88

Fixture strategy with small canonical scenes

  • Fixtures are authored once, by hand, in Blender and exported via its glTF exporter (same reasoning as the CAD plan's FreeCAD choice: realistic tool output, including whatever non-determinism real exporters introduce, is exactly what §3's normalization must absorb) — never generated programmatically for the golden/round- trip tiers.
  • A small, deliberately minimal fixture library: a single textured cube (simplest possible mesh+material+texture case); a hierarchy with 3+ levels of nesting and repeated mesh instancing (identity/dedup stress case); a scene with a camera and two lights of different kinds; a simple two-bone skeletal animation (walk-cycle-style, minimal joint count); a scene with a deliberately broken external texture reference; a scene with degenerate (zero-area) triangles; an OBJ file with no declared unit (for the unit-unspecified validation case).
  • Floating-point tolerance and property-based tests use synthetic generators, not the hand-authored fixtures, since they need boundary-exact epsilon coverage the fixture library doesn't provide.

9. Backend Integration

Domain plugin interface implementation

muse/plugins/scene3d/ following the muse/plugins/midi/ layout (and the CAD plan's muse/plugins/cad/ layout):

muse/plugins/scene3d/
  __init__.py
  plugin.py            # MuseDomainPlugin implementation: snapshot/diff/merge/drift/apply/schema
  entity.py            # Canonical model dataclasses (§2)
  manifest.py          # SnapshotManifest (de)serialization, normalization (§3)
  scene_diff.py        # Diff algorithms (§4)
  scene_merge.py       # Merge algorithms (§5)
  _query.py            # Query DSL implementation (§6)
  _invariants.py       # Validation/drift checks (§7)
  _import_gltf.py       # glTF/GLB <-> IR adapter
  _import_obj.py        # OBJ (import-only) <-> IR adapter

schema() declares (mirroring code/midi/identity/the CAD plan's domains --json shape):

{
  "schema_version": "<muse_version>",
  "merge_mode": "three_way",
  "description": "Semantic version control for 3D scene graphs...",
  "dimensions": [
    {"name": "nodes", "description": "Scene graph hierarchy, transforms, and instancing."},
    {"name": "geometry", "description": "Content-addressed mesh vertex/index buffers."},
    {"name": "materials", "description": "PBR materials and their texture references."},
    {"name": "textures", "description": "Content-addressed image assets."},
    {"name": "lighting_and_cameras", "description": "Lights and cameras and their parameters."},
    {"name": "animations", "description": "Keyframe animation channels and skeletal rigs."},
    {"name": "environment", "description": "Scene-level ambient/background/unit/up-axis settings."},
    {"name": "metadata", "description": "Authorship, source format, schema version."}
  ]
}

Storage implications

No new storage subsystem — geometry, texture, and keyframe blobs live in .muse/objects/ like any other domain's binary payloads, same decision as the CAD plan. muse count-objects/gc/prune need no scene-specific changes. Repository size growth from texture-heavy scenes is tracked as an open question (§10), not designed against speculatively.

API endpoints needed (MuseHub)

  • GET /:owner/:repo/scene/:ref/nodes — scene graph listing at a ref.
  • GET /:owner/:repo/scene/:ref/diff?base=&target= — structured scene diff for proposal/review rendering.
  • GET /:owner/:repo/scene/:ref/asset/:content_id — stream a geometry/texture/ keyframe blob (for viewer hydration) — content-addressed, cacheable indefinitely.
  • Reuses existing generic muse hub proposal/issue endpoints unchanged, same as the CAD plan — no new proposal/review primitives needed, only new content rendering.

CLI implications

New muse scene subcommand group, structured like muse code/the planned muse cad: muse scene query, muse scene find-in-bounds, muse scene high-poly, muse scene missing-textures, muse scene unreferenced-assets, muse scene changed-transforms, muse scene list-lights, muse scene list-cameras, muse scene animated-nodes. All accept --json. No changes needed to domain-agnostic commands (status, diff, merge, log, etc.).

MuseHub UI implications

  • Proposal/diff view needs a scene diff renderer: a node-tree view with changed nodes highlighted, a live 3D viewport (glTF renders natively in-browser via three.js/<model-viewer> — a meaningfully simpler lift than the CAD plan's OCCT-derived viewer problem, since glTF is already a web-native format) showing before/after or a diff overlay, and a conflict panel for merge review. This should be its own issue (§10), but is lower-risk than the CAD plan's equivalent UI issue given glTF's native web renderability.
  • Repo browse view needs a scene-graph tree view (mirrors a file tree, keyed on the Node hierarchy from §2 rather than the filesystem) with an embedded live preview.

Visualization hooks

  • muse scene render-thumbnail <ref> --json — server-side headless glTF render to a static thumbnail image, used for repo browse cards and proposal summaries where a live viewport isn't warranted.
  • Diff-aware highlighting: the viewer/thumbnail hook accepts an optional diff overlay (changed nodes rendered in a highlight color) — the same rendering pipeline serves both the plain view and the diff view, driven by the StructuredDelta from §4.

MCP/tooling implications

Expose scene_query, scene_diff, scene_find_in_bounds as MCP tools (mirroring the CAD plan's cad_query/cad_diff/cad_impact exposure and whatever pattern code domain tools already follow), so agentception workers can reason about scene state programmatically — e.g. "are there any unreferenced assets before this branch merges" or "did this change move the main camera" as an agent-callable check.

10. Staging Deliverables

Proposed issue breakdown

  1. SCN-1 — Canonical model + snapshot (§2, §3): entity dataclasses, normalization, stable node identity (including the structural/display split), content-addressing, glTF/GLB import adapter, golden-snapshot tests (SCN_01–SCN_22). Load-bearing — nothing else starts until this lands.
  2. SCN-2 — Semantic diff (§4): depends on SCN-1. SCN_30–SCN_39.
  3. SCN-3 — Merge strategy (§5): depends on SCN-1, SCN-2. SCN_40–SCN_53.
  4. SCN-4 — Validation & drift (§7): depends on SCN-1; can proceed in parallel with SCN-2/SCN-3 once SCN-1 lands. SCN_70–SCN_78.
  5. SCN-5 — Query DSL + CLI (§6, CLI portion of §9): depends on SCN-1, SCN-2. SCN_80–SCN_88.
  6. SCN-6 — Floating-point tolerance / property-based suite (§8): depends on SCN-1, SCN-2, SCN-3. SCN_60–SCN_61.
  7. SCN-7 — OBJ import adapter (§1): depends on SCN-1. Smaller, independent — OBJ has no hierarchy/animation, so it only needs the mesh/material subset of the IR.
  8. SCN-8 — MuseHub API endpoints (§9): depends on SCN-1, SCN-2.
  9. SCN-9 — MuseHub UI: scene diff viewer + visualization hooks (§9): depends on SCN-8. Lower-risk than the CAD plan's equivalent (glTF is web-native) but still should not be scoped in detail until SCN-2/SCN-3 output shapes are proven against real fixtures.
  10. SCN-10 — MCP tooling exposure (§9): depends on SCN-5.
  11. SCN-11 — USD adapter (v1, tracked but not started): explicitly deferred; filed now only as a placeholder so the v0→v1 boundary is visible in the tracker, not scoped until SCN-1 through SCN-6 are complete and USD's composition-arc semantics can be designed against a proven IR.

Milestones

  • M1 (Foundation): SCN-1 merged, muse domains --json lists scene3d as active, golden-snapshot suite green.
  • M2 (Semantic core): SCN-2 + SCN-3 merged; a hand-authored two-branch conflict fixture (e.g. both branches move the main camera) produces a correct, human-readable conflict report end-to-end via muse merge --dry-run.
  • M3 (Usable): SCN-4 + SCN-5 + SCN-7 merged; muse scene query/missing-textures/ unreferenced-assets/high-poly all answer correctly against the fixture library.
  • M4 (Ecosystem): SCN-8, SCN-9, SCN-10 merged; a scene proposal on staging MuseHub renders a real node-level diff with a live glTF viewport.

Acceptance criteria (whole-plan level)

  • muse domains --json shows scene3d with active: true and the schema in §9.
  • Every fixture in §8's fixture library round-trips (§8, round-trip tier) with zero diff against itself, across both .gltf and .glb container forms.
  • Renaming a node in the multi-level-nesting fixture produces a single RenameOp with no spurious diffs on any descendant (directly verifies the structural/ display identity split in §3).
  • The broken-external-texture-reference and no-declared-unit (OBJ) fixtures are correctly flagged by muse check/muse status (§7) with no false positives on the well-formed fixtures.
  • A merge of two branches that independently edit disjoint nodes/materials/animation channels auto-resolves with zero reported conflicts; a merge where both branches edit different fields of the same material also auto-resolves (field-level granularity working as designed).
  • A merge of two branches that move the same node to different transforms is reported as a conflict with a correct plain-language cause string (§5).
  • muse scene query correctly answers all query examples in §6 against the fixture library.
  • No test in the suite depends on network access, a specific Blender version's exact binary output, or wall-clock time (determinism requirement, tying back to §3/§7).

Risks

  • Node identity collision risk: the "identical sibling nodes" edge case in §3 (structural identity ties, requiring a positional tiebreaker) is a known weak point — mitigation is to test it directly and document the tiebreaker behavior clearly rather than assume it won't occur; large procedurally-instanced scenes (e.g. a forest with hundreds of identical tree instances) will hit this case routinely, so it cannot be treated as a rare corner case.
  • Geometry-diff granularity risk: same risk class as the CAD plan — v0's coarse bounding-box/count-based geometry diff may prove too coarse for real review workflows once SCN-9's viewer makes the gap visible. Mitigation: ship coarse first, gather usage feedback via SCN-9 before investing in per-vertex diffing.
  • glTF extension coverage risk: real-world glTF files frequently use vendor/ community extensions (KHR_materials_*, KHR_texture_transform, etc.) beyond the core PBR model in §1's scope. Mitigation: the "unsupported material features" validation check (§7) makes any silent data loss visible immediately rather than discovered later as an unexplained diff — extension coverage can then be expanded incrementally based on which extensions actually appear in real fixtures.
  • Repository size risk: texture-heavy scenes can be large; same open question as the CAD plan regarding whether .muse/objects/ scales acceptably, tracked below rather than designed against speculatively.
  • USD scope-creep risk: there will be pressure to fold USD support into v0 given its industry importance (Pixar/NVIDIA/Apple backing). Mitigation: this plan explicitly defers it (§1, §10 SCN-11) with the stated reason (composition-arc semantics need their own design pass) — any attempt to widen v0 scope to include USD should be treated as a plan amendment requiring the same review this plan gets, not a mid-implementation addition.

Open questions

  • Does texture/geometry storage eventually need an LFS-style side channel separate from .muse/objects/, and at what repo-size threshold does that become necessary? (Same open question as the CAD plan — worth answering once for both domains together rather than twice independently.)
  • Should the sibling-node identity-collision tiebreaker (§3) be positional-index-based (simple, but reintroduces some position-sensitivity) or based on a secondary content signal (e.g. instance transform, which is usually distinct even for identical meshes) — needs a decision before SCN-1 implementation, not left to whoever implements it to improvise.
  • What is the v1 USD composition model's relationship to this domain's ExternalReference element — is a USD reference/payload arc modeled as an ExternalReference with richer semantics, or does it need a distinct element type? Deferred to SCN-11 design, flagged here so it isn't forgotten.
  • Should muse scene render-thumbnail run synchronously in the MuseHub API request path or as an async background job queued on commit (mirroring how fetch.mpack.prebuild already works as a background worker job per existing MuseHub architecture)? Recommend async, consistent with existing patterns, but should be confirmed during SCN-9 scoping rather than assumed here.

Implementation order

SCN-1 → SCN-2 → (SCN-3, SCN-4, and SCN-7 in parallel) → SCN-5 → SCN-6 → SCN-8 → (SCN-9 and SCN-10 in parallel) → SCN-11 (deferred, not scheduled). Mirrors §1's dependency structure: nothing can diff, merge, validate, or query state that hasn't been normalized into the canonical model yet, and nothing in MuseHub/MCP can expose behavior the plugin doesn't yet have.

This is a plan-only issue. No implementation lands against it. Each numbered deliverable in §10 should be filed as its own staging issue once this plan is reviewed, referencing the test IDs and acceptance criteria defined above so implementation can proceed test-first without re-deriving these decisions.

◎ Activity
●
gabriel opened this issue 19 days ago
◎
No activity yet. Use the CLI to comment.