gabriel / muse public
Open #68 Enhancement
filed by gabriel human · 19 days ago

Video Editing Timelines Domain — implementation plan (TDD-first, comprehensive)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Video Editing Timelines Domain — Implementation Plan

Background

Muse treats "domain" as a first-class primitive (muse/domain.py::MuseDomainPlugin). A domain plugin implements six methods — snapshot, diff, merge, drift, apply, schema — and gets the full VCS engine for free: the commit DAG, branching, checkout, lineage walking, log graph, and merge-base finder. muse.plugins.midi proved the model for structured symbolic + binary-payload data. muse.plugins.code proved it for AST symbols. muse.plugins.social/identity proved it for graph-shaped state. The CAD domain plan (staging issue #66) is proving it for continuous geometry with causal feature histories. The 3D Scenes domain plan (staging issue #67) is proving it for hierarchical scene graphs with instanced assets and time-varying animation layered on top of static structure.

Video editing timelines are a fifth distinct shape of problem: everything is fundamentally about time. A timeline is a set of tracks, each holding an ordered sequence of clips positioned at specific time ranges, referencing external source media, layered with transitions/effects/keyframes, and often nesting sub-timelines (compound clips) that recurse the whole model. Today this lives entirely inside opaque, editor-specific project files (Premiere .prproj, Resolve's project database, Final Cut's library bundle) or lossy interchange formats (EDL, XML, AAF) that flatten most semantic structure away. If Muse can version, diff, merge, validate, and query a video timeline with frame-accurate, clip-level semantic awareness — as it already does for code and MIDI, and as the CAD (#66) and 3D Scenes (#67) plans are proving for geometry and scene graphs — it proves the domain-plugin model generalizes to temporal/layered creative data too.

This issue is the plan only. Each numbered phase in §10 is expected to become its own staging issue once reviewed and accepted.

Goal

Ship a muse/plugins/timeline/ domain plugin, structured like muse/plugins/midi/ and the planned muse/plugins/cad/ and muse/plugins/scene3d/ (a plugin.py entrypoint plus focused private modules for diff/merge/query/invariants/import-export), that:

  • Registers in muse domains --json alongside code, midi, identity, mist, social, and (once landed) cad/scene3d.
  • Passes muse code test --json for every module it touches.
  • Snapshots, diffs, and merges a video timeline with track/clip/trim/effect-level semantic awareness — never as an opaque .prproj/.fcpxml/AAF blob.
  • Supports a timeline query DSL (muse timeline query "...", mirroring muse code query and the CAD/3D-Scenes plans' muse cad query/muse scene query) capable of answering the concrete query examples in §6.
  • Detects the drift/validation classes in §7 via muse status / muse check.
  • Has test coverage across every tier in §8, with test IDs (VID_01, VID_02, …) referenced from both this plan and the test files that satisfy them.

"Done" for the plan means: reviewed, decomposed into §10's issue breakdown, with acceptance criteria concrete enough for a different agent to implement test-first.

1. Domain Scope

Supported initial timeline targets

  • Single-project, single-or-multi-sequence nonlinear timelines: the common model shared by Premiere, Resolve, Final Cut, and CapCut — video/audio tracks holding ordered clips with in/out points, transitions between adjacent clips, per-clip and per-track effects with keyframed parameters, and markers/captions anchored to time.
  • v0 targets timelines in the tens to low-hundreds of clips per sequence range (a short film, a YouTube episode, a commercial) — not feature-length multi-hour timelines with thousands of clips, and not multi-user live-broadcast switching timelines.

Same two-layer strategy as the CAD/3D-Scenes plans (#66, #67), adapted for timelines:

  1. Canonical semantic layer (Muse-owned): a JSON-serializable timeline IR — the "Canonical Model" in §2 — that Muse snapshots, diffs, merges, and queries directly.
  2. Interchange/import-export boundary: OpenTimelineIO (OTIO) as the primary MVP interchange format. OTIO is the closest thing the industry has to "the glTF of editorial timelines": an open-source, actively-maintained project (Pixar/Academy Software Foundation) explicitly designed as a timeline interchange format, with existing adapters for Premiere XML, FCPXML, AAF, and EDL already maintained upstream, and already JSON-shaped — mapping cleanly onto Muse's snapshot-as-JSON model, directly analogous to why glTF was chosen for the 3D Scenes plan. Rather than Muse writing and maintaining N separate editor-format adapters, v0 imports/exports through OTIO's own adapter layer, so Muse's plugin only needs one native integration to reach Premiere XML, FCPXML, AAF, and EDL by extension — a materially different leverage point than the CAD/3D-Scenes plans, which each own their format adapters directly. This is the single most important scope decision in this plan.

Relationship to EDL, XML, FCPXML, OTIO, AAF, Premiere, Resolve, and Final Cut

Format/tool Role in this plan
OTIO v0 primary interchange format and integration point — the adapter targets OTIO's Python object model directly, inheriting its adapter ecosystem rather than reimplementing it.
EDL (CMX3600) Reached transitively via OTIO's EDL adapter. Lowest-common-denominator (no effects, no nested sequences, timecode-only) — a simple fixture source for cut-diff tests, not a rich round-trip target.
Premiere XML Reached transitively via OTIO's adapter. Premiere's native .prproj (proprietary, no public spec) is not a v0 target — only its XML export path, same reasoning as the 3D Scenes plan excluding native .blend.
FCPXML Reached transitively via OTIO's adapter. The richest transitively-supported format (nested sequences, keyframed effects, audio/caption roles) — recommended fixture source for v0's advanced features (§2).
AAF Reached transitively via OTIO's adapter. Post-production-oriented (color metadata, multi-channel audio) — recommended fixture source for stress-testing §2's Color/Audio elements.
Resolve Native OTIO import/export (unlike Premiere/FCP, which require XML first) — recommended secondary reference authoring tool for v0 fixtures.
Final Cut Pro Recommended primary reference authoring tool (richest transitive FCPXML export) — mirrors the CAD plan's FreeCAD choice and the 3D Scenes plan's Blender choice.
CapCut Out of scope in v0 — no public spec, no OTIO adapter. Mentioned only as a category example; addable later without changing the canonical model if an interchange path appears.

What is in scope for v0

  • Projects containing one or more sequences (mirrors OTIO's Timeline/Stack concepts).
  • Video and audio tracks, each holding an ordered, non-overlapping-within-track list of clips (gaps between clips are explicit Gap elements, not implicit).
  • Clips: an in/out time range into a piece of source media, plus per-clip enable/mute state.
  • Source media references: content-identified pointers to external media files (not the media bytes themselves in v0 — see §3).
  • Time ranges expressed in rational frame-rate-aware time (not raw float seconds) — see §3's frame-rate/timecode normalization.
  • Transitions between adjacent clips on the same track (dissolve, wipe, cut — the common cross-editor subset).
  • Effects: a per-clip or per-track ordered effect stack with named parameters, including keyframed parameter values over time.
  • Markers: timeline-anchored, or clip-anchored, annotations with a name/comment/color.
  • Captions/subtitles: time-ranged text cues, modeled as their own track type (mirrors FCPXML's caption roles and OTIO's Marker/subtitle handling).
  • Nested sequences (compound clips): a sequence used as a clip's source media within another sequence — modeled recursively, not flattened.
  • External asset references and their resolution state.

What is explicitly out of scope for v0

  • The actual media essence (video/audio frame data, waveform data, proxy files) — v0 treats source media as externally referenced, content-identified pointers, never as bytes Muse itself stores, transcodes, or renders (§3, Handling large binary media). This is the single largest scope-limiting decision in the plan: this domain is about the edit, not the footage.
  • Multi-camera / multi-cam clip sync groups — deferred to a v1 follow-up.
  • Color grading node graphs (Resolve's full color page) beyond a flattened summary color-metadata record per clip (LUT reference, basic wheels) — same treatment as the CAD plan's stance on freeform surfaces and the 3D Scenes plan's stance on shader graphs.
  • Render/export presets and render-queue state, beyond cache/render metadata (§2) needed for staleness drift detection — no semantic modeling of render settings.
  • Real-time collaborative (CRDT) timeline editing — v0 merge is offline/three-way only (merge_mode: three_way), consistent with every prior domain plan.
  • Multi-hour, thousand-clip broadcast/feature-length timelines and any performance work targeting that scale.
  • Audio mixing automation beyond keyframed volume/pan (mixing-console state, sidechain routing, bus architecture) — treated as opaque effect parameters.

2. Canonical Model

All elements are content-addressed (sha256:<hex>) and referenced by ID, never positional index — the same identity principle as code's AST symbols, midi's notes, the CAD plan's Part/Feature/Body, and the 3D Scenes plan's Node — so that reordering never produces a spurious diff, and time-range shifts are diffed as time- range changes, not delete+insert.

Element Fields Notes
Project id, name, sequences: [Sequence.id], frame_rate (default, rational — §3), metadata One project may hold multiple sequences, since nested sequences reference siblings.
Sequence id, name, tracks: [Track.id] (ordered — stacking order is semantically meaningful, since it affects compositing), frame_rate, duration: TimeRange, metadata A Sequence can be referenced as a clip's source media by another Sequence (nesting) — must remain acyclic (§7).
Track id, kind (video|audio|caption), clips: [Clip.id \| Gap.id] (ordered, non-overlapping), enabled: bool, locked: bool Gaps are explicit elements, not absence — makes gap-finding (§6) a direct query.
Clip id, name, source_ref: SourceMedia.id \| Sequence.id, source_range: TimeRange (in/out within source), timeline_range: TimeRange (position on track), enabled: bool, muted: bool, effects: [Effect.id] (ordered), transition_in/out: Transition.id \| null Splitting source_range from timeline_range is what makes trim vs. move vs. retime distinguishable in diff (§4).
Gap id, duration: TimeRange Explicit absence-of-content on a track; content-addressed by duration.
SourceMedia id, uri_or_path, content_id_at_link_time (hash if locally accessible, else null), frame_rate, sample_rate, duration: TimeRange, resolved: bool, codec_hint Media bytes are never stored (§1) — this is a resolved/content-identified pointer, promoted to first-class since Clip references it directly.
TimeRange id, start_time: RationalTime, duration: RationalTime RationalTime is (value, rate) — e.g. (2400, 24000/1001) — never a raw float (§3).
Transition id, kind (dissolve|wipe|cut|custom), duration: TimeRange, alignment (centered|from_start|from_end), params Sits between two adjacent clips; alignment drives trim-conflict interactions (§5).
Effect id, kind (built_in:<name> or plugin:<vendor>:<name>), params: {key: value \| Keyframe.id list}, enabled: bool Named parameters generically, not a closed effect enum, mirroring the 3D Scenes plan's flattened-PBR approach.
Keyframe id, time: RationalTime, value, interpolation (linear|hold|bezier) Discrete addressable elements, not a binary buffer — timeline keyframe counts are orders of magnitude smaller than mesh/animation data.
Marker id, anchor: {kind: timeline\|clip, ref}, time: RationalTime, name, comment, color Review markers — distinct from Captions, which are time-ranged program content.
Caption id, track_ref: Track.id, time_range: TimeRange, text, language, role (mirrors FCPXML roles) Track-anchored, since captions commonly span underlying clip cut points.
AudioLane id, track_ref: Track.id, channel_layout (mono|stereo|5.1), gain_db, pan Track-level mix state, distinct from per-clip audio Effects.
ColorMetadata id, clip_ref: Clip.id, lut_ref: content_id \| null, color_space, wheels: {lift, gamma, gain} Flattened summary per §1's scope cut — no node-graph modeling.
RenderCacheMetadata id, sequence_ref: Sequence.id, render_hash, output_ref: content_id \| null, stale: bool stale computed by comparing render_hash to current content identity (§7) — mirrors the CAD plan's Measurement.stale.
ExternalReference id, kind (linked_sequence|linked_media|linked_lut), uri_or_path, content_id_at_link_time, resolved: bool Same role as prior plans' ExternalReference — drives drift detection (§7).
Metadata frame_rate, timecode_format (drop_frame|non_drop_frame), author, created_at, source_format (otio|fcpxml|aaf|edl|premiere_xml), source_checksum, schema_version timecode_format is load-bearing for NTSC drop-frame arithmetic (§3).

Design invariant carried over from every prior domain plan: identity is content- and reference-based, never positional, with one exception: Track stacking order within a Sequence is semantically meaningful (compositing order) and is therefore diffed as an order change — the one place this domain departs from the 3D Scenes plan's "child order is never identity-bearing" rule.

3. Snapshot Design

Deterministic timeline normalization

snapshot() walks the live state (an OTIO-loadable project under state/, same pattern as MIDI's .mid walk, the CAD plan's .FCStd/STEP walk, and the 3D Scenes plan's .gltf/.glb walk) and produces the canonical IR in §2 as the SnapshotManifest, applying normalization in a fixed order:

  1. Resolve every element to a stable ID (§3, Stable clip and track identity, below).
  2. Canonicalize all time values to RationalTime (§3, Unit handling, below) — never store or compare raw floating-point seconds.
  3. Canonicalize ordering: Sequence.tracks order is preserved (semantically meaningful, per §2); Track.clips order is derived from timeline_range.start_time (a track's clip list has exactly one canonical order — its position on the timeline — so there is no ambiguity to normalize away, unlike node-child lists in the 3D Scenes plan). Effect stacks preserve authored order (also semantically meaningful — effect application order changes the result).
  4. Compute each Sequence's duration and each Track's occupied-time summary at snapshot time — a cheap invariant used to short-circuit full-track diffs (§4) the same way the CAD plan's topology_summary and the 3D Scenes plan's vertex_count/bounding_box short-circuit their respective expensive diffs.

Stable clip and track identity

Like 3D Scenes nodes, OTIO/FCPXML/AAF clips generally have no globally stable ID across re-exports. Clip identity is derived as a content hash of (track_identity, source_ref content-id, source_range) — "this exact piece of this exact source media, on this exact track" defines identity, independent of timeline position. Directly analogous to the 3D Scenes plan's structural/display identity split:

  • A clip's timeline_range.start_time is explicitly excluded from its identity hash — otherwise every trim/ripple edit (which shifts every downstream clip's start time) would appear as a full delete+insert cascade, the same failure mode the 3D Scenes plan's rename-cascade fix avoids, applied here to temporal shifts.
  • Two clips using the same source media and range at two different timeline points (e.g. a repeated shot) are only disambiguated by track_identity; if on the same track they are genuinely identical by this scheme — same known-limitation pattern as the 3D Scenes plan's identical-sibling-node case, handled the same way: a documented, tested positional tiebreaker, not silently assumed away.
  • Track identity is derived from (sequence_identity, kind, ordinal_position) — tracks have no content-based identity independent of position, since two otherwise- identical video tracks in different stacking positions are genuinely different (compositing order matters, §2). This is why Track order is not excluded from identity the way Node child order is in the 3D Scenes plan.

Content-addressed source media references

Muse does not store media essence (§1). SourceMedia.content_id_at_link_time records a hash of the referenced file's bytes at snapshot time, when the file is locally accessible — a fingerprint for drift detection, not a stored object. If the underlying media file changes on disk after linking (re-exporting a graded master, swapping a placeholder for final VFX), the fingerprint mismatch powers the "missing/ changed source media" drift check (§7). When the file isn't locally accessible at snapshot time (offline/proxy workflows), content_id_at_link_time is null and resolved is false — explicit and visible, never silently assumed resolved.

Handling large binary media

This is the most consequential storage decision in the plan, and differs from every prior one: CAD and 3D Scenes store their binary payloads (B-Rep geometry, mesh/ texture/keyframe buffers) as content-addressed blobs in .muse/objects/. Video essence is orders of magnitude larger — a single hour of ProRes footage can be tens of gigabytes — and Muse has no video-specific storage tier. Therefore: v0 never stores media essence in .muse/objects/ at all. Source media is always an external reference (SourceMedia/ExternalReference, §2), resolved against paths/URIs recorded at commit time — the same way a build system references external dependencies rather than vendoring them. The only binary payloads this plugin writes to the object store are small, timeline-scoped artifacts: LUT files (ColorMetadata.lut_ref, typically kilobytes) and, if populated, an optional low- resolution proxy/thumbnail (RenderCacheMetadata.output_ref) — never a full- resolution export. This boundary is load-bearing, not incidental, and should be revisited only as an explicit, separately-reviewed decision (§10), not relaxed incrementally during implementation.

Frame-rate and timecode normalization

  • Every RationalTime value is stored as (value, rate) — never a float — because standard video frame rates (23.976, 29.97, 59.94, all x*1000/1001 NTSC rates) are not exactly representable in binary floating point, and naive float timecode arithmetic is a well-known source of off-by-one-frame bugs. This parallels the CAD plan's fixed-epsilon rationale, but the fix here is representational (rational arithmetic) rather than tolerance-based, because frame-accurate editing genuinely requires exact equality, not "close enough."
  • Metadata.timecode_format (drop_frame|non_drop_frame) is recorded explicitly, since drop-frame timecode display (NTSC 29.97/59.94) skips certain timecode labels without skipping actual frames — display-layer formatting only, required for correct human-readable diff/conflict output (§4, §5).
  • A sequence's declared frame_rate is authoritative for all RationalTime values within it; a clip whose source rate differs (mixed-frame-rate editing is normal — e.g. slow-motion footage shot at a higher rate) is not an error by itself, but is recorded so the frame-rate-mismatch check (§7) can distinguish intentional design from accidental mismatch.

Unit handling for frames, seconds, samples, and ticks

All units in circulation (frames, seconds, audio samples, and format-specific "ticks" like AAF's edit units) are represented as RationalTime with an explicit rate field, so a value is always unambiguous regardless of source-format unit. Conversion is pure rational arithmetic (no precision loss), performed only at import/export boundaries — the canonical IR itself needs no "current unit mode," eliminating an entire category of unit-confusion bugs by construction (contrast with the CAD/3D- Scenes plans, which must validate unit consistency after the fact, since their source formats don't carry rate information as intrinsically as timecode-based formats do).

Deterministic serialization

OTIO's own JSON serialization is already close to deterministic, but adapter round- trips (e.g. FCPXML → OTIO → EDL) can reorder or drop fields depending on adapter version. Muse never hashes OTIO JSON bytes directly for content-addressing; it always re-serializes through the canonical IR's fixed field order and ID scheme (§2), same discipline as the CAD/3D-Scenes plans' "never hash the interchange format's bytes directly" rule. A golden-snapshot suite (§8) locks this down: re-snapshotting the same fixture twice, or the same project authored in Final Cut vs. round-tripped through Resolve's native OTIO export, must produce byte-identical canonical JSON.

4. Semantic Diff

diff(base, target, *, repo_root=None) returns a StructuredDelta of typed ops, same contract as every prior domain plan.

  • Clip add/remove/move/trim diffs: InsertOp/DeleteOp keyed on clip identity (§3). A move (timeline_range.start_time changed) is reported distinctly from a trim (source_range/timeline_range.duration changed) — "you slid this clip later" vs. "you shortened this clip" read very differently and conflating them would make diffs unreadable, mirroring the CAD plan's value-change vs. structural-change distinction.
  • Track diffs: added/removed tracks are structural; since Track order is identity-bearing (§2, §3), reordering (e.g. moving a video track above another) is reported as a diff — the one deliberate departure from the "order never matters" pattern elsewhere in this plan and the 3D Scenes plan, because compositing order changes the rendered result.
  • Sequence diffs: added/removed sequences; a nested Clip.source_ref pointing at a different Sequence is reported as a retarget, distinct from an ordinary source-media swap, since it can pull in an entirely different sub-timeline.
  • Transition diffs: InsertOp/DeleteOp/ReplaceOp on transition_in/_out, with kind/duration deltas — cheap, fully-diffable, no coarse-summary treatment needed (unlike geometry-scale data in the CAD/3D-Scenes plans).
  • Effect parameter diffs: field-level, same granularity as the 3D Scenes plan's material diffing — different-parameter edits on the same effect are independently reported and mergeable (§5).
  • Keyframe diffs: diffed individually (InsertOp/DeleteOp/ReplaceOp per Keyframe.id), unlike the CAD/3D-Scenes plans' coarse buffer-level treatment — timeline keyframe counts per parameter are orders of magnitude smaller than mesh/ animation data, so per-element diffing is cheap here.
  • Marker/comment diffs: field-level, keyed on Marker.id (text vs. position vs. color changes reported distinctly).
  • Caption diffs: keyed on Caption.id; a text edit is reported distinctly from a time-range edit (re-timing to match a re-cut vs. correcting a typo).
  • Audio edit diffs: AudioLane field-level diffs (gain/pan) reported separately from clip-level audio Effect diffs — track mix state vs. per-clip processing are different layers.
  • Source media reference diffs: a content_id_at_link_time change ("source media changed") is reported distinctly from a resolved flag flip (file went missing/came back) — same distinction the CAD/3D-Scenes plans draw for ExternalReference.
  • Rename/move detection: a rename (identity unchanged, name changed) is a single RenameOp; a move (the track_identity component of a clip's identity changes — cut from one track, pasted onto another) is a MoveOp, not delete+insert — parallels the CAD plan's feature-move and the 3D Scenes plan's node-move detection.
  • Frame-accurate comparisons: every RationalTime comparison is exact rational equality, never epsilon-tolerant — the opposite of the CAD/3D-Scenes plans' floating-point tolerance, because video editing is frame-quantized: a "close but not exact" time value is either a real one-frame edit (must be shown) or a normalization bug (must be fixed, not tolerated).

5. Merge Strategy

merge(base, left, right, *, repo_root=None) — three-way, merge_mode: three_way, consistent with every prior domain plan.

  • Non-conflicting timeline edits (the common case): edits to disjoint clips/tracks/sequences merge automatically via the engine's default address-keyed map merge, driven by schema()'s dimensions (§9).
  • Same-clip conflicts: both branches change the same Clip's source_ref or source_range differently — always a reported conflict (no auto-resolution of "which cut is correct," per every prior plan's rejection of silently picking a value neither author chose).
  • Same-track ordering conflicts: both branches insert a different clip into the same gap — reported conflict (ambiguous, and inserting both would overlap, itself invalid per §7). If the insertions don't overlap in time, both merge, and the track is re-validated as a strict non-overlapping partition post-merge (§7, now a post- merge gate too).
  • Trim conflicts: both branches trim the same in/out point differently — conflict; but trimming different points of the same clip (in vs. out) auto-merges — same field-level-independence principle as the 3D Scenes plan's material merge.
  • Transition conflicts: both branches change the same boundary's transition differently — conflict. A duration change extending past available source handle is flagged as an additional validation failure, not silently clamped.
  • Effect/keyframe conflicts: field-level for parameters (independent edits auto-merge, same-parameter edits conflict); keyframe-level for keyframes (different new keyframes at different times auto-merge; same keyframe edited on both branches conflicts).
  • Marker/comment merges: the one element class where v0 leans toward union-merge by default rather than conflict-on-collision — two editors independently leaving review markers at the same position are not expressing incompatible intent the way conflicting trims are. Only an edit to the same Marker.id's text/color conflicts. A deliberate, documented departure from this plan's default rule, justified by review annotations being qualitatively different from structural edits — call this out explicitly in the implementation and tests (§8).
  • Nested sequence conflicts: recurses — the merge engine runs the same three-way merge on the nested sequence's own tracks/clips rather than treating it as an atomic conflict unit (which would force a conflict on every nested edit). A non-conflicting nested edit auto-merges even if the outer referencing clip was untouched.
  • Missing media conflicts: if a branch's edit relies on SourceMedia that is resolved: false at merge time, the merge is not blocked — offline media is normal, not an error — but the result carries a warning annotation, a third category beyond auto-merge/conflict specific to this domain's offline workflow.
  • Human-readable conflict reports: every conflict carries both branches' values, the base value, and a plain-language cause string in editorial terms (e.g. "Both branches trimmed clip 'Interview_B-roll_04' out point: left=01:23:14:02, right=01:23:15:19, base=01:23:12:00" — formatted timecode, per §3), consistent with the workspace-wide conflict-marker convention and every prior plan's requirement.

6. Query Support

A timeline query DSL mirroring muse code query and the CAD/3D-Scenes plans' muse cad query/muse scene query (muse timeline query "<predicate>" --json), plus purpose-built subcommands for the most common questions.

Example video timeline query DSL

muse timeline query "clip.source_media == 'B-roll_Drone_02.mov'" --json
muse timeline query "effect.kind == 'built_in:gaussian_blur' and effect.enabled == true" --json
muse timeline query "clip.timeline_range.start > timecode('01:00:00:00')" --json

Find clips by source media

muse timeline query "clip.source_media == 'Interview_A_Take3.mov'" --json
muse timeline find-by-source "Interview_A_Take3.mov" --json     # convenience alias

Find gaps

muse timeline query "element.type == 'gap' and element.duration > frames(1)" --json
muse timeline find-gaps --track V1 --json

Find overlaps

muse timeline find-overlaps --json      # should normally return empty (§7 invariant);
                                         # useful during import validation of a
                                         # non-Muse-authored source file

Find muted/disabled clips

muse timeline query "clip.muted == true or clip.enabled == false" --json
muse timeline disabled-clips --json

Find clips using specific effects

muse timeline query "clip.effects contains 'plugin:redgiant:magic_bullet'" --json
muse timeline find-by-effect "plugin:redgiant:magic_bullet" --json

Find timeline regions with unresolved media

muse timeline query "clip.source_media.resolved == false" --json
muse timeline offline-media --json      # mirrors the CAD/3D-Scenes plans' broken/
                                         # missing-reference query pattern

Find edits after a timestamp

muse timeline query "clip.timeline_range.start > timecode('00:45:00:00')" --json
muse timeline edits-after "00:45:00:00" --json

Find all captions in a time range

muse timeline query "caption.time_range intersects (timecode('00:10:00:00'), timecode('00:12:00:00'))" --json
muse timeline captions-in-range --start "00:10:00:00" --end "00:12:00:00" --json

Find high-density edit regions

muse timeline query "region.cut_density(window=frames(150)) > 8" --json
muse timeline edit-density --window 150f --threshold 8 --json     # mirrors `muse code
                                                                    # hotspots` framing —
                                                                    # rapid-cut montage
                                                                    # detection

7. Validation and Drift Detection

Surfaces through muse status (via drift()) and muse check/muse code invariants (via a timeline-specific invariant checker, same pattern as the CAD/3D-Scenes plans' _invariants.py).

  • Missing source media: any SourceMedia with resolved: false, surfaced as a first-class, expected-to-be-common state (unlike the "hard error" framing in the CAD/3D-Scenes plans) — see §5's missing-media-conflict treatment for why this domain treats offline media as routine. Flagged clearly in muse status, not commit- blocking by default.
  • Broken nested sequence references: a Clip.source_ref pointing at a missing Sequence.id, or a nested-sequence cycle (A nests B nests A) — the cycle is commit-blocking (no legitimate editorial meaning); the missing-reference case follows the same "flagged, not blocking" policy as missing source media.
  • Invalid time ranges: overlapping clips on the same track (violates §2's strict- partition invariant), negative/zero-duration ranges, and a source_range extending beyond SourceMedia.duration (checkable only when resolved) — all commit-blocking, since these are genuinely malformed, not a normal offline-media gap.
  • Frame-rate mismatches: a clip's SourceMedia.frame_rate differing from its sequence's is informational by default (mixed-rate editing is normal, §3) but escalated to a warning if the ratio isn't a simple, commonly-intentional one (e.g. 60fps-in-24fps for slow motion is a clean 2.5x; 23.976fps misread as 24fps in a 29.97fps timeline is the subtle confusion this check targets) — specific ratio heuristics are an open question (§10), not guessed at here.
  • Audio sample-rate mismatches: analogous to frame-rate mismatches, but with no "intentional creative choice" analog (no editorial equivalent of slow motion), so always at least a warning, never merely informational.
  • Unsupported effect parameters: an effect whose shape falls outside v0's generic named-parameter model (e.g. a proprietary plugin's binary blob) is flagged as lossy import at snapshot time, same "never silently drop data" principle as the 3D Scenes plan — raw data is preserved as an opaque blob, and the flag makes the fidelity loss visible.
  • Non-deterministic importer output: re-importing the same source project twice must produce identical canonical IR — a hard-error regression gate, same treatment as every prior plan, since it would silently break content-addressing.
  • Schema migration risks: Metadata.schema_version mismatches are flagged before diff/merge across the boundary, never auto-migrated silently, consistent with the workspace-wide rule that muse code migrate requires explicit permission.

8. TDD Plan

Test IDs use the VID_NN prefix, referenced from both this plan and the eventual test files, per the workspace convention (IC_01, the CAD plan's CAD_NN, the 3D Scenes plan's SCN_NN).

Tier Coverage Example test IDs
Unit Canonical model (de)serialization for every element type; RationalTime arithmetic (incl. NTSC drop-frame); clip identity hashing (position-exclusion fix). VID_01–VID_03
Golden snapshot Fixed fixtures (cuts-only sequence, multi-track w/ transitions, keyframed color-correction, nested-sequence project, captioned sequence) snapshotted once, checked byte-for-byte against committed golden JSON. VID_10–VID_14
Round-trip import/export FCPXML/AAF round-trips must reproduce input within the format's fidelity ceiling and produce identical IR on re-import; EDL cross-format must produce a valid subset of the FCPXML-sourced IR — analog of the CAD/3D-Scenes plans' cross-format equivalence tests, adapted for lossy-by-format-choice reality. VID_20–VID_22
Timeline diff One test per §4 diff category (clip add/remove/move/trim, track reorder, sequence retarget, transition, effect, keyframe, marker, caption, audio lane, source media, rename/move, frame-exact-equality). VID_30–VID_41
Merge One test per §5 non-conflicting case and conflict category, incl. trim-field-independence, marker-union-merge, recursive nested-sequence merge, missing-media warning. VID_50–VID_59
Conflict Conflict report shape/content, incl. timecode formatting (§3); muse resolve clears a timeline conflict. VID_60–VID_63
Frame-accurate timecode Property-based RationalTime generation across standard frame rates (23.976–60) asserting rational-equality never false-positives/negatives, and drop-frame display round-trips without altering frame counts. VID_70, VID_71
Validation/drift One test per §7 bullet (missing media informational, broken nested reference, cycle blocking, invalid range, rate-mismatch distinction, unsupported params, non-deterministic import, schema mismatch). VID_80–VID_88
Query One test per §6 query example against a fixture with known results. VID_90–VID_98

Fixture strategy with small canonical timeline projects

  • Fixtures are authored once, by hand, in Final Cut Pro (primary, richest FCPXML export) and DaVinci Resolve (secondary, native OTIO export) — realistic tool output, including exporter non-determinism, is exactly what §3's normalization must absorb — never generated programmatically for the golden/round-trip tiers.
  • A minimal fixture library: a two-clip cuts-only sequence; a multi-track sequence with a dissolve and a keyframed color-correction effect (3-4 keyframes); a nested- sequence project (a compound clip containing its own 2-clip sub-sequence); a captioned sequence with captions spanning a cut point; a sequence with an offline media reference; a sequence with a deliberate frame-rate mismatch (60fps clip in a 24fps timeline); an EDL export of the simplest fixture (lower-fidelity cross-format test).
  • Frame-accurate timecode property-based tests use synthetic generators across the standard frame-rate table, not the hand-authored fixtures, for systematic coverage of rate-conversion edge cases.

9. Backend Integration

Domain plugin interface implementation

muse/plugins/timeline/ following the muse/plugins/midi/ layout (and the CAD/3D- Scenes plans' equivalent layouts):

muse/plugins/timeline/
  __init__.py
  plugin.py             # MuseDomainPlugin implementation: snapshot/diff/merge/drift/apply/schema
  entity.py             # Canonical model dataclasses (§2)
  manifest.py           # SnapshotManifest (de)serialization, normalization (§3)
  rational_time.py       # RationalTime arithmetic, drop-frame timecode formatting
  timeline_diff.py       # Diff algorithms (§4)
  timeline_merge.py       # Merge algorithms (§5)
  _query.py              # Query DSL implementation (§6)
  _invariants.py         # Validation/drift checks (§7)
  _otio_bridge.py         # OTIO <-> IR adapter (the single integration point reaching
                          # FCPXML/AAF/EDL/Premiere-XML transitively, per §1)

schema() declares (mirroring code/midi/identity/the CAD and 3D Scenes plans' domains --json shape):

{
  "schema_version": "<muse_version>",
  "merge_mode": "three_way",
  "description": "Semantic version control for nonlinear video editing timelines...",
  "dimensions": [
    {"name": "structure", "description": "Sequences, tracks, and their stacking order."},
    {"name": "clips", "description": "Clip placement, trims, and source media references."},
    {"name": "transitions", "description": "Transitions between adjacent clips."},
    {"name": "effects", "description": "Effect stacks and their keyframed parameters."},
    {"name": "markers_and_captions", "description": "Review markers and time-ranged caption cues."},
    {"name": "audio", "description": "Per-track audio lanes and mix metadata."},
    {"name": "color", "description": "Per-clip flattened color grading metadata."},
    {"name": "metadata", "description": "Frame rate, timecode format, authorship, schema version."}
  ]
}

Storage implications

Unlike the CAD and 3D Scenes plans, this domain's storage footprint in .muse/objects/ is deliberately small — no media essence is ever stored (§3); only small metadata artifacts (LUTs, optional low-res proxies) use the object store at all. muse count-objects/gc/prune need no timeline-specific changes. This is a positive property of the design that should be preserved, not eroded, as the plugin evolves.

API endpoints needed (MuseHub)

  • GET /:owner/:repo/timeline/:ref/sequences — sequence/track listing at a ref.
  • GET /:owner/:repo/timeline/:ref/diff?base=&target= — structured timeline diff for proposal/review rendering.
  • GET /:owner/:repo/timeline/:ref/offline-media — convenience endpoint surfacing unresolved SourceMedia for a given ref, since this is expected to be a routinely- checked state (§7) rather than an exceptional one.
  • Reuses existing generic muse hub proposal/issue endpoints unchanged, same as every prior domain plan.

CLI implications

New muse timeline subcommand group, structured like muse code/the planned muse cad/muse scene: muse timeline query, muse timeline find-gaps, muse timeline find-overlaps, muse timeline disabled-clips, muse timeline find-by-effect, muse timeline offline-media, muse timeline edits-after, muse timeline captions-in-range, muse timeline edit-density. All accept --json. No changes needed to domain-agnostic commands (status, diff, merge, log, etc.).

MuseHub UI implications

  • Proposal/diff view needs a timeline diff renderer: a horizontal multi-track view (standard NLE metaphor) with changed clips/transitions/effects highlighted and a scrubbable playhead. The structural view needs only timeline structure, not media essence; an actual video preview requires resolving SourceMedia (only possible when locally accessible), so the diff view must degrade gracefully to "structure- only" when media is offline. Conflict panel presented in timecode, not raw frames.
  • Repo browse view needs a sequence/track tree view analogous to the CAD/3D-Scenes plans' part-tree/node-tree views, keyed on Sequence/Track/Clip.

Timeline visualization hooks

  • muse timeline render-thumbnail-strip <ref> --json — server-side filmstrip thumbnail generation, only when source media is locally resolvable — unlike the 3D Scenes plan's equivalent (glTF scenes are self-contained; timelines are not, per §1/§3), this hook doesn't work in every environment. API/UI must handle "media offline, structure-only" as a first-class outcome, not an error state.
  • Diff-aware highlighting: the diff renderer accepts an optional overlay (changed clips/regions highlighted), driven by the StructuredDelta from §4 — same pattern as the 3D Scenes plan's diff-aware viewer hook.

MCP/tooling implications

Expose timeline_query, timeline_diff, timeline_find_gaps as MCP tools (mirroring the CAD/3D-Scenes plans' equivalent exposure), so agentception workers can reason about edit state programmatically — e.g. "are there any gaps or overlaps before this branch merges," "does this cut still reference all its source media," or "how dense are the edits in the cold open" as agent-callable checks before a merge or review is completed.

10. Staging Deliverables

Proposed issue breakdown

  1. VID-1 — Canonical model + snapshot + RationalTime (§2, §3): entity dataclasses, rational-time arithmetic, drop-frame formatting, stable clip/track identity (position-exclusion fix), OTIO import bridge, golden-snapshot tests (VID_01–VID_14). Load-bearing — nothing else starts until this lands.
  2. VID-2 — OTIO export + round-trip adapters (§1, §3): depends on VID-1. FCPXML/ AAF/EDL round-trip via the OTIO bridge. VID_20–VID_22.
  3. VID-3 — Semantic diff (§4): depends on VID-1. VID_30–VID_41.
  4. VID-4 — Merge strategy (§5): depends on VID-1, VID-3. VID_50–VID_63.
  5. VID-5 — Validation & drift (§7): depends on VID-1; parallel with VID-3/VID-4. VID_80–VID_88.
  6. VID-6 — Query DSL + CLI (§6, §9): depends on VID-1, VID-3. VID_90–VID_98.
  7. VID-7 — Frame-accurate timecode property-based suite (§8): depends on VID-1 only (not diff/merge). VID_70–VID_71.
  8. VID-8 — MuseHub API endpoints (§9): depends on VID-1, VID-3.
  9. VID-9 — MuseHub UI: timeline diff viewer + visualization hooks (§9): depends on VID-8. Not scoped in detail until VID-3/VID-4 shapes are proven, and must design for the offline-media degraded case from the start, not retrofit it.
  10. VID-10 — MCP tooling exposure (§9): depends on VID-6.

Milestones

  • M1 (Foundation): VID-1 merged, muse domains --json lists timeline as active, golden-snapshot suite green, RationalTime arithmetic verified across the standard frame-rate table.
  • M2 (Interop): VID-2 merged; FCPXML/AAF/EDL round-trip fixtures pass at their respective fidelity ceilings.
  • M3 (Semantic core): VID-3 + VID-4 merged; a two-branch conflict fixture (both branches trim the same clip's out point differently) produces a correct, timecode- formatted conflict report via muse merge --dry-run; marker-union-merge and nested-sequence-recursive-merge verified working as designed.
  • M4 (Usable): VID-5 + VID-6 + VID-7 merged; muse timeline query/find-gaps/ find-overlaps/offline-media/edit-density all answer correctly.
  • M5 (Ecosystem): VID-8, VID-9, VID-10 merged; a timeline proposal on staging MuseHub renders a real clip-level diff in a multi-track view, degrading gracefully to structure-only rendering when source media is offline.

Acceptance criteria (whole-plan level)

  • muse domains --json shows timeline with active: true and the schema in §9.
  • Every fixture round-trips through OTIO/FCPXML/AAF (§8) with zero diff against itself; the EDL cross-format test produces a valid fidelity-reduced subset, not a divergent result.
  • Trimming a clip produces exactly one TrimOp with no spurious diffs on any other clip on the track — verifies the position-exclusion identity fix (§3).
  • The offline-media fixture is flagged as informational, not commit-blocking (§7); the overlapping-clips and nested-sequence-cycle fixtures are commit- blocking, with no false positives on well-formed fixtures.
  • A merge of disjoint clip/track/effect edits auto-resolves with zero conflicts; a merge where one branch trims a clip's in-point and the other trims its out-point also auto-resolves; two branches adding different markers at the same position merge without conflict.
  • A merge trimming the same out-point to different values reports a conflict with a correctly timecode-formatted cause string (§5).
  • muse timeline query correctly answers all examples in §6 against the fixtures.
  • No test depends on network access, a specific FCP/Resolve export's exact binary output, or wall-clock time (§3/§7).

Risks

  • OTIO dependency-surface risk: routing all editor interop through OTIO's adapter layer (§1) is the plan's central leverage decision, but makes this domain's fidelity ceiling a function of OTIO's own per-format adapter maturity (AAF/Premiere-XML are less complete than FCPXML as of this writing). Mitigation: VID-2's round-trip tests are scoped per-format with fidelity-ceiling assertions (§8), not a blanket claim — a known OTIO gap should surface as a documented, tested ceiling, not a silent failure.
  • Clip identity collision risk: same risk class as the 3D Scenes plan's identical- sibling-node case — a timeline reusing the same shot on the same track (e.g. a repeated flashback insert) hits the positional-tiebreaker path routinely, not as a rare corner case. Must be tested directly.
  • Frame-rate mismatch heuristic risk: the informational-vs-warning distinction in §7 depends on a "commonly-intentional ratio" heuristic this plan declines to fully enumerate (open question below) — risk of false-positive warnings (normal slow motion) or false negatives (missed rate confusion) until real usage informs it.
  • Offline-media-by-design risk: unusually among Muse's domains, a large fraction of normal operation involves data (media essence) the plugin deliberately never accesses. Every downstream feature must degrade gracefully rather than assume media is present (§5, §9) — easy to build a version that works only against an always- online fixture library and breaks on real offline workflows.

Open questions

  • What frame-rate ratio table counts as "commonly intentional" for §7's informational-vs-warning distinction (e.g. 2x, 2.5x slow-motion ratios) — needs a decision, ideally from surveying real fixtures, before VID-5 implementation.
  • Should the clip identity-collision tiebreaker (§3) match the 3D Scenes plan's equivalent for consistency, or does this domain's track-ordinal-position concept (already load-bearing for Track identity) suggest a domain-specific answer?
  • Does RenderCacheMetadata.output_ref's proxy storage need a retention policy beyond ordinary muse gc/prune reachability, given proxies could still accumulate across a long-lived project's history?
  • Should VID-9's diff viewer attempt real frame preview when media is locally resolvable, or should v0 UI stay structure-only everywhere to avoid building two rendering paths before usage shows which is needed first?
  • Is there a meaningful v1 target for multi-cam sync groups (deferred in §1), and would it need a new element type or fit within Clip/Sequence — flagged, not designed now.

Implementation order

VID-1 → (VID-2, VID-3, and VID-7 in parallel) → VID-4 → VID-5 → VID-6 → VID-8 → (VID-9 and VID-10 in parallel). Mirrors §1's dependency structure: nothing can diff, merge, validate, or query state that hasn't been normalized into the canonical model yet. VID-2 and VID-7 run in parallel with VID-3 since neither depends on diff/merge landing first.

This is a plan-only issue. Each deliverable in §10 should be filed as its own staging issue once reviewed, referencing the test IDs and criteria above.

◎ Activity
●
gabriel opened this issue 19 days ago
◎
No activity yet. Use the CLI to comment.