gabriel / muse public
Closed #91 Bug
filed by gabriel human · 41 days ago

semver classifier flags breaking_changes on a reorder-only diff despite its own 'no semantic change' summary

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Summary

While researching Build With Muse Episode 07 (Semantic Diff), found a real inconsistency in classify_delta() (muse/core/semver_classifier.py): a commit that purely reorders two top-level functions within a file — no symbol added, removed, or modified — gets classified with sem_ver_bump: "minor" and a non-empty breaking_changes list, despite the diff's own ops entry explicitly summarizing itself as no semantic change.

Reproduction

  1. Commit a file with two functions, e.g.:
    def helper_a(x): return x
    def helper_b(x): return x * 2
    
  2. Commit again with just the two functions' order swapped (no other edits).
  3. muse diff HEAD~1 HEAD correctly reports zero symbol-level changes:
    M  extra.py
    
    1 modified file
    
  4. But muse read-commit HEAD --json | jq '.structured_delta' shows:
    {
      "ops": [
        {
          "op": "replace",
          "address": "extra.py",
          "old_summary": "extra.py (before)",
          "new_summary": "extra.py (reformatted — no semantic change)"
        }
      ],
      "summary": "1 modified file",
      "sem_ver_bump": "minor",
      "breaking_changes": ["extra.py"]
    }
    

The op's own new_summary says "no semantic change" in plain English, yet the same delta is classified as a breaking, minor-version-bump change one field over.

Hypothesis (not confirmed)

diff()'s fallback path for a file where nothing at the symbol level changed (a pure reorder) apparently produces a top-level replace op keyed by the whole file address ("extra.py") rather than a patch op with an empty child_ops list. classify_delta()'s _classify_replace() branch then treats a file-level replace op the same way it would treat replacing an actual symbol — defaulting to breaking/minor — with no way to distinguish "the file's bytes changed but I found zero semantic content changes" from an actual content replacement it couldn't otherwise interpret.

Ask

  1. Confirm whether this fallback path is intentional conservatism (file-level replace with no visibility into content = assume worst case) or a genuine gap.
  2. If intentional: the new_summary text is misleading and should not claim "no semantic change" while the classification says otherwise — pick one honest story.
  3. If a gap: either emit a patch op with empty child_ops (matching the pattern used elsewhere, e.g. real rename/move ops) instead of a file-level replace, or teach _classify_replace to recognize this specific no-op-content case and classify it as none.
  4. Add a regression test: reorder-only diff must have sem_ver_bump == "none" and breaking_changes == [].

Priority

Not urgent — deliberately not chasing this down mid-recording. Circle back once the Build With Muse Season 1 season is wrapped; this doesn't block any episode script since Episode 07 already avoided making any semver claim about this specific case.

  • Episode 07 script (presentations/build-with-muse/episode-07/semantic-diff.md) — documents choosing not to use this detail on screen, and this ticket, in its own production notes.

Source

Found by Claude Sonnet 5 while researching Episode 07, 2026-09-12.

◎ Activity1
●
gabriel opened this issue 41 days ago
○
gabriel 2 days ago

Fixed — confirmed your hypothesis exactly, and it was a gap, not intentional conservatism.

Confirmation (Ask #1)

build_diff_ops() (muse/plugins/code/symbol_diff.py) does exactly what you suspected: when every symbol's content_id in a modified file is unchanged (pure reorder here — same mechanism behind the comment-only case in #92), it emits a whole-file ReplaceOp labeled "<path> (reformatted — no semantic change)" instead of a patch op with empty child_ops.

_classify_replace() (muse/core/semver_classifier.py) pattern-matches new_summary for "renamed to"/"signature"/"implementation" but had no case for this exact fallback text, so it fell through to the conservative "unrecognised summary" catch-all — classified breaking/minor, confidence 0.4 — directly contradicting the op's own "no semantic change" claim one field over.

Gap, not intentional (Ask #3)

There's no principled reason to assume worst-case when the diff engine already determined, with certainty, that nothing semantic changed — the two fields were just disagreeing with each other. Went with your option 3's second suggestion: taught _classify_replace to recognize this specific case rather than changing the op shape, since it's a smaller, more surgical change than threading a patch/empty-child_ops shape through the whole fallback path. Added a branch checking for "no semantic change" in new_summary, classifying it invisible (no API impact, no bump, confidence 1.0) — placed just before the catch-all, so the existing generic "reformatted" (without the "no semantic change" suffix) catch-all test is untouched and still passes.

Regression tests (Ask #4)

  • tests/test_sem_ver.py::TestClassifyDeltaReplace::test_replace_no_semantic_change_is_invisible — synthetic-op unit test matching the file's existing style.
  • tests/test_reorder_only_semver_classification.py — real-CLI end-to-end test reproducing your exact repro steps (commit two functions, commit again with only their order swapped, assert sem_ver_bump == "none" and breaking_changes == []). This test reproduced your exact contradictory JSON (sem_ver_bump: "minor", breaking_changes: ["extra.py"]) when run against the pre-fix code.

Confirmed genuine RED on both new tests (reverted the fix, both failed for the right reason — same symptom from your ticket body) before GREEN with the fix restored.

Verification

Targeted regression suite — test_sem_ver.py (123 tests, including the untouched catch-all test), the two new tests, test_cmd_commit.py, test_cmd_diff.py (the only two direct callers of classify_delta per muse code impact) — all pass (336 passed), zero regressions.

Merged to dev → main, pushed to local/staging/production for both branches.

✓
closed this issue 2 days ago