gabriel / muse public
Closed #4 Enhancement
filed by gabriel human · 31 days ago

Stage writes blob to object store at add time (follow git's model)

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Background

Currently `muse code add` records the `object_id` (SHA-256 of file content) in `.muse/code/stage.msgpack` but defers writing the actual blob to the object store until `muse commit`. Git writes the blob at `git add` time — the index is just a pointer map into the object store.

This divergence surfaces as a persistent special case everywhere file content needs to be read: staged files have an `object_id` that doesn't exist in the store yet, so every reader (`muse cat`, `muse code cat`, `muse diff`, etc.) must fall back to disk. That compounds across the whole toolchain.

Discovered while fixing a `muse cat` bug where calling it on an untracked file silently returned disk content with no indication the file wasn't in the snapshot. The correct fix — error on untracked files — is complicated by the staged-but-not-committed state, where a file is tracked but its blob isn't in the store yet.

Proposed change

Write the blob to the object store at `muse code add` time. Stage entry records the `object_id` as today, but the bytes are in the store immediately.

Why this is the right move

  • SHA is already computed at add time (it's in stage.msgpack). Writing the blob is one incremental step that's already paid for.
  • Every tracked file — committed or staged — has a blob in the store. `muse cat` becomes: check manifest → check staged index → not found → `FILE_NOT_TRACKED`. No disk fallback path.
  • Workdir disk fallback in `muse cat` / `muse code cat` can be removed entirely. Cleaner invariant, fewer code paths.
  • Enables `muse cat --staged` (read staged content from store, like `git show :path`).
  • Consistent with 20 years of established VCS idiom. Anyone reasoning about "is this file tracked" has one mental model: in store = tracked.
  • `muse gc` already handles dangling objects from resets — the object store churn argument is already solved infrastructure.

Trade-off acknowledged

Resets and amends leave dangling blobs. `muse gc` handles this, same as git. Acceptable cost.

Implementation plan

Phase 1 — Write blob at stage time

Files: `muse/plugins/code/stage.py`, `muse/cli/commands/code/add.py`

  • In the `muse code add` handler, after computing `object_id` for each file, call `write_object_from_path(root, object_id, disk_path)` before writing the stage entry.
  • Add `has_object` check first — skip the write if the blob is already present (idempotent, same as commit does today).
  • Tests: staged file blob exists in object store immediately after `muse code add`.

Phase 2 — Remove the disk fallback from `muse cat`

Files: `muse/cli/commands/core_cat.py`, `tests/test_cmd_core_cat.py`

  • `_get_file_bytes`: check manifest first, then staged index. If found in either, read from object store. If neither, raise `FILE_NOT_TRACKED`.
  • Remove the `source_is_workdir` disk-read path entirely (symlink guard, path containment, FileNotFoundError fallback — all gone).
  • Keep one carve-out: committed files modified on disk since last commit should still show the working-tree version. That means: if in manifest AND on disk AND not symlink, prefer disk (uncommitted edits visible). If deleted from disk, fall back to store. If untracked, error.
  • Update `TestCatDiskFallback` — those tests now assert the wrong behavior.
  • New tests: untracked file exits 1; staged file readable from store; deleted-from-disk tracked file readable from store.

Phase 3 — Remove the disk fallback from `muse code cat`

Files: `muse/cli/commands/cat.py`, `tests/test_cmd_cat.py`

  • Same change as Phase 2 but for the symbol-level reader.
  • `_get_file_bytes` in `cat.py`: staged files now readable from store — no disk special case.
  • After this phase: `muse code cat file.py::Symbol` on an untracked file errors with `FILE_NOT_TRACKED` regardless of whether the file exists on disk.

Phase 4 — `muse cat --staged`

Files: `muse/cli/commands/core_cat.py`, `muse/cli/commands/cat.py`

  • Add `--staged` flag that reads exclusively from the staged index (ignoring working-tree disk edits made after `muse code add`).
  • Mirrors `git show :path`.
  • Only meaningful after Phase 1 (requires blob in store at add time).

Phase 5 — `muse diff` staged path audit

Files: `muse/cli/commands/diff.py` and related

  • Audit `muse diff --staged` to confirm it reads from the object store rather than disk for staged content.
  • If it falls back to disk anywhere, apply the same fix.
  • This phase may be a no-op depending on how diff is currently implemented.

Acceptance criteria

  • `muse code add src/foo.py` → blob immediately present in `.muse/objects/`
  • `muse cat src/foo.py` on an untracked file → exits 1, `FILE_NOT_TRACKED`
  • `muse cat src/foo.py` on a staged file → exits 0, reads from object store
  • `muse code cat src/foo.py::Symbol` on an untracked file → exits 1
  • All existing cat/code-cat/diff tests pass
  • No disk fallback paths remain in `muse cat` or `muse code cat`
◎ Activity3
●
gabriel opened this issue 31 days ago
○
gabriel 31 days ago

Testing standards

Every phase must ship with full 8-tier test coverage. No phase is done until all tiers are present.

Tier What it covers
Unit Pure functions in isolation — `_get_file_bytes`, `read_stage`, `write_object`, blob ID computation. No I/O, no subprocess.
Integration Two or more components wired together — add → stage index → object store → cat reads it back. Real filesystem, no mocks.
End-to-end Full CLI invocation through `CliRunner` — `muse code add`, `muse cat`, `muse commit` as a user would run them. Asserts on stdout/stderr/exit code.
Stress High-cardinality inputs — 200 files staged at once, 1 MiB blobs, 50-address batch cat. Confirms no quadratic behavior or memory cliff.
State Explicit state-machine coverage — untracked → staged → committed → deleted, reset after stage, re-stage after reset. Every reachable state and every legal transition.
Data integrity Content round-trips exactly — blob written at add time equals blob read at cat time equals bytes on disk. SHA-256 verified end-to-end, not just assumed.
Performance Latency budgets enforced as assertions — single-file cat < 300 ms, 10-file batch < 1 s, `muse code add` on 50 files < 2 s. Prevents silent regressions.
Security Symlink rejection, path traversal guard, ANSI/control-char injection in paths, blob ID tampering (wrong hash rejected by object store).

Code quality

Each phase must also ship with:

  • Module-level docstring on every changed file — states the invariant the module enforces (e.g. "after `muse code add`, the blob is always in the object store"), not just what the file contains.
  • Function docstrings on every public function — contract (preconditions, postconditions), not description. What must be true going in, what is guaranteed coming out.
  • Inline comments only where the why is non-obvious — a hidden constraint, a subtle invariant, a workaround for a specific behavior. Never describe what the code does; well-named identifiers already do that.
  • Error codes as constants — every `_FileError` code string (`FILE_NOT_TRACKED`, `BLOB_NOT_FOUND`, etc.) must be a named constant, not a bare string literal, so tests can assert against the constant rather than a magic string.
○
gabriel 31 days ago

Implementation complete

All acceptance criteria are met as of the commits now on staging/main.

Phases shipped

Phase Commit Status
Phase 1 — write blob at stage time code_stage.py — write_object_from_path called at add time ✅
Phase 2 — remove disk fallback from muse cat 800f3b8e7 fix(muse cat): error on untracked files; read staged blobs from object store ✅
Phase 3 — remove disk fallback from muse code cat 539fe79bb fix(muse code cat): FILE_NOT_TRACKED for untracked files; staged blobs readable from store ✅
Phase 4 — muse cat --staged Not in acceptance criteria — deferred as follow-up if needed ⏭
Phase 5 — muse diff staged audit No-op confirmed: --staged target files use object_id from staged index; Phase 1 guarantees blobs are in store, disk fallback never fires ✅

Acceptance criteria

  • muse code add src/foo.py → blob immediately present in .muse/objects/ ✅
  • muse cat src/foo.py on an untracked file → exits 1, FILE_NOT_TRACKED ✅
  • muse cat src/foo.py on a staged file → exits 0, reads from object store ✅
  • muse code cat src/foo.py::Symbol on an untracked file → exits 1 ✅
  • All existing cat/code-cat/diff tests pass ✅
  • No disk fallback paths remain in muse cat or muse code cat for untracked files ✅
○
gabriel 31 days ago

All 4 phases shipped and on staging.

Phase 1 — muse cat file-level command (core VCS, domain-agnostic) Phase 2 — Untracked file rejection; staged files readable from object store Phase 3 — muse code cat --staged (symbol-level staged reads, muse code cat) Phase 4 — --staged flag on both muse cat and muse code cat; reads staged index (HEAD manifest + stage overrides); mutually exclusive with --at; source_ref = "staged" in JSON output

Tests: 143 passing across test_cmd_core_cat.py and test_cmd_cat.py, including 11 new TestCatStaged tests for muse cat --staged and 8 for muse code cat --staged.