gabriel / musehub public
Closed #18 Enhancement
filed by gabriel human · 48 days ago

feat(intel/type): Type Health dashboard — GUI for muse code type

0 Anchors
Blast radius
Churn 30d
0 Proposals

Context

`muse code type` analyses every symbol in the repo for Python type-annotation coverage and produces a ranked health report. The worker already indexes the per-symbol data into `musehub_intel_type` via `TypeProvider` (registered as `intel.code.type`). Only the UI layer and one small DB extension are missing.


CLI output shape (confirmed against musehub repo)

Summary envelope

```json { "mode": "health", "total_symbols": 8467, "fully_typed": 8085, "partially_typed": 365, "untyped": 17, "any_count": 3, "coverage_fraction": 0.9548836659974017 } ```

Per-symbol record

```json { "address": "musehub/api/routes/musehub/ui_intel.py::intel_clones_page", "kind": "async_function", "return_annotation": "HTMLResponse", "return_is_any": false, "params_total": 4, "params_annotated": 4, "params_with_any": 0, "type_score": 1.0 } ```

Score tiers

Tier Condition Count (musehub)
Fully typed `type_score == 1.0` 8,085
Partial `0.5 ≤ type_score < 1.0` 345
Untyped `type_score < 0.5` 37
Any-polluted `return_is_any OR params_with_any > 0` 3

Current DB state

`musehub_intel_type` (created in migration 0004) stores every field except `return_annotation`:

Column Type Stored
repo_id PK
address PK
kind VARCHAR(64)
return_is_any BOOLEAN
params_total INTEGER
params_annotated INTEGER
params_with_any INTEGER
type_score FLOAT
ref VARCHAR(128)
return_annotation VARCHAR(256) ❌ missing

Also missing: `TypeProvider` upserts one row per `await session.execute()` in a Python loop — 8,467 round-trips on the musehub repo. Needs the same batch-upsert fix applied to all other Phase 2 providers.


ASCII art — page layout

``` ╔══════════════════════════════════════════════════════════════════════════════╗ ║ ◈ TYPE HEALTH gabriel / musehub ║ ║ Indexed at sha256:71e4eb32 · 8,467 symbols ║ ╚══════════════════════════════════════════════════════════════════════════════╝

╭──────────────────────────────────────────────────────────────────────────────╮ │ COVERAGE RING STAT CHIPS │ │ │ │ ╭───────────╮ ╔══════════╗ ╔══════════╗ ╔══════════╗ ╔══════════╗ │ │ ╱ ╲ ║ 8,085 ║ ║ 345 ║ ║ 17 ║ ║ 3 ║ │ │ │ 95.5% │ ║ ████████ ║ ║ ██████░░ ║ ║ ░░░░░░░░ ║ ║ ⚠ ANY ║ │ │ │ [spectral │ ║ FULLY ║ ║ PARTIAL ║ ║ UNTYPED ║ ║ POLLUTED ║ │ │ │ gradient │ ║ TYPED ║ ║ ║ ║ ║ ║ ║ │ │ ╲ ╱ ╚══════════╝ ╚══════════╝ ╚══════════╝ ╚══════════╝ │ │ ╰───────────╯ │ ╰──────────────────────────────────────────────────────────────────────────────╯

╭── FILTER ────────────────────────────────────────────────────────────────────╮ │ Tier: [All ●] [Untyped] [Partial] [Any-Polluted] [Fully Typed] │ │ Kind: [All ●] [function] [method] [async_function] [class] │ │ Sort: [Score ▲] [Score ▼] [Address] Show: [20] [50 ●] [100] │ ╰──────────────────────────────────────────────────────────────────────────────╯

╭── SYMBOLS ───────────────────────────────────────────────────────────────────╮ │ │ │ # ADDRESS KIND SCORE PARAMS RETURN BAR │ │ ──── ────────────────────────────── ─────── ────── ─────── ─────── ───── │ │ │ │ 1 deploy/locustfile.py fn 0.00 0/2 — ░░░░░ │ │ ::_check_thresholds ▔▔▔▔▔▔ none │ │ │ │ 2 tests/conftest.py method 0.75 1/1 None ████░ │ │ ::_Asgi24Wrapper.init ⚠ any ▔▔▔▔▔▔ ✓ │ │ │ │ 3 musehub/services/jobs.py async 0.50 1/2 — ██░░░ │ │ ::_process_batch fn ▔▔▔▔▔▔ str │ │ │ │ ··· │ ╰──────────────────────────────────────────────────────────────────────────────╯ ```

Spectral theme mapping

Element CSS class / var
Page title `.intel-subhd-title--spectral` (gradient text)
Coverage ring fill `var(--gradient-spectral)` on SVG arc
Stat chip values `.intel-grav-stat__val` (spectral gradient text)
Score bars `.th-score-bar__fill` → `var(--gradient-spectral)`
Any-pollution badge `.kind-badge--any` → `var(--color-warning)`
Tier pill active `.intel-filter-pill--active` → `var(--color-purple)`

Phase 1 — DB extension + provider fix

1a. Migration 0013: add `return_annotation` column

```sql ALTER TABLE musehub_intel_type ADD COLUMN return_annotation VARCHAR(256); ```

Nullable; existing rows get NULL; populated on next push.

1b. ORM model update

Add to `MusehubIntelType`: ```python return_annotation: Mapped[str | None] = mapped_column(String(256), nullable=True) ```

1c. TypeProvider — batch upsert fix

Replace the per-symbol loop with a single `pg_insert(...).values([...])` call using the existing batch-upsert pattern (same as `ClonesProvider`, `CouplingProvider`, etc.). Eliminates 8k+ sequential round-trips on large repos.

Also wire `return_annotation` into the upsert values.


Phase 2 — TDD (write tests first, confirm red)

All tests in `tests/test_intel_type.py`. Run before any implementation. Every test must fail for the right reason before Phase 3–5 begin.

Layer T1 — DB extension

``` T01 migration 0013 adds return_annotation column T02 MusehubIntelType.return_annotation is nullable VARCHAR(256) T03 TypeProvider stores return_annotation on upsert ```

Layer T2 — Provider batch performance

``` T04 TypeProvider issues one SQL statement per push (not N per symbol) T05 Upserting 500 symbols completes in < 500ms ```

Layer T3 — Route (unit/integration)

``` T06 intel_type_page returns 200 with no rows (empty repo) T07 intel_type_page returns 200 with data rows T08 summary stats (total, fully_typed, partial, untyped, any_count, coverage_fraction) computed from DB match CLI values T09 filter ?tier=untyped returns only symbols with type_score < 0.5 T10 filter ?tier=partial returns only 0.5 ≤ score < 1.0 T11 filter ?tier=any returns only return_is_any OR params_with_any > 0 T12 filter ?kind=function filters by kind column T13 symbols ranked score ASC by default T14 ?top=20 / ?top=50 / ?top=100 limits respected ```

Layer T4 — E2E (HTML body assertions)

``` T15 coverage_fraction rendered as percentage string in HTML body T16 symbol address present in rendered HTML T17 return_annotation present in rendered HTML when non-null T18 Any-pollution warning badge rendered when params_with_any > 0 T19 dashboard card at /intel links to /intel/type ```

Layer T5 — State integrity

``` T20 pushing the same commit twice produces exactly one row per symbol T21 pushing a new commit overwrites type_score for existing addresses T22 repo delete cascades — all musehub_intel_type rows removed ```

Layer T6 — Performance

``` T23 route responds in < 200ms for 10,000 symbol repo T24 DB query uses ix_intel_type_repo index (EXPLAIN confirms) T25 TypeProvider batch upsert for 500 symbols < 500ms wall time ```

Layer T7 — Security

``` T26 XSS payload in address field is escaped in HTML output T27 XSS payload in return_annotation is escaped in HTML output T28 ?tier= with unexpected value returns 200 (treated as "all") T29 ?top= with non-integer returns 422 T30 Unauthenticated request to private repo returns 403/404 ```


Phase 3 — Route handler

File: `musehub/api/routes/musehub/ui_intel.py`

Summary query (no subprocess)

```python

Aggregate stats from DB — mirrors CLI summary envelope

total = SELECT COUNT() FROM musehub_intel_type WHERE repo_id = ? fully = SELECT COUNT() WHERE repo_id = ? AND type_score = 1.0 partial = SELECT COUNT() WHERE repo_id = ? AND type_score >= 0.5 AND type_score < 1.0 untyped = SELECT COUNT() WHERE repo_id = ? AND type_score < 0.5 any_ct = SELECT COUNT(*) WHERE repo_id = ? AND (return_is_any OR params_with_any > 0) cov = fully / total (computed in Python) ```

Symbol list query

```sql SELECT * FROM musehub_intel_type WHERE repo_id = :repo_id [AND type_score < 0.5] -- tier=untyped [AND type_score >= 0.5 AND < 1.0] -- tier=partial [AND type_score = 1.0] -- tier=fully_typed [AND (return_is_any OR params_with_any > 0)] -- tier=any [AND kind = :kind] ORDER BY type_score ASC, address ASC LIMIT :top ```

Context dict

```python ctx = { "repo": repo, "summary": { "total_symbols": int, "fully_typed": int, "partially_typed": int, "untyped": int, "any_count": int, "coverage_fraction": float, # 0.0–1.0 "coverage_pct": str, # "95.5%" }, "symbols": list[dict], # address, kind, type_score, score_pct, # return_annotation, return_is_any, # params_total, params_annotated, params_with_any, # any_polluted (bool) "tier": str, # "all"|"untyped"|"partial"|"any"|"fully_typed" "kind": str, # "all"|"function"|"method"|… "top": int, # 20|50|100 "kinds": list[str], # distinct kinds present in repo "index_meta": …, } ```


Phase 4 — Templates

`musehub/templates/musehub/pages/intel_type.html`

Sections (top → bottom):

  1. Page header — breadcrumb + `.intel-subhd-title--spectral` "Type Health"
  2. Summary band — SVG coverage ring + 4 stat chips (fully typed / partial / untyped / any-polluted)
  3. Filter bar — tier pills + kind pills + top selector
  4. Symbol table — ranked list; each row: rank, address (mono), kind badge, score bar, params ratio, return annotation, any-pollution ⚠ badge
  5. Empty state — shown when no symbols match filter

`intel_dashboard.html` — type card update

The existing placeholder card gains:

  • `coverage_pct` stat rendered if `index_meta` has type intel
  • Link to `/intel/type`

Phase 5 — SCSS

`src/scss/pages/_type.scss` (structural only)

``` .th-page — outer page wrapper .th-summary-band — ring + chips flex row .th-ring-wrap — SVG donut container .th-chips-row — 4-chip grid .th-chip — individual stat chip .th-chip__val — large number .th-chip__lbl — uppercase label .th-symbol-table — list container .th-row — one symbol row .th-row__rank — ordinal column .th-row__addr — monospace address .th-row__kind — badge cell .th-row__score — score bar + number cell .th-row__params — annotated/total fraction .th-row__return — return_annotation text .th-row__any — any-pollution badge cell .th-score-bar — progress track .th-score-bar__fill — gradient fill ```

`src/scss/components/_type.scss` (visual only)

``` .th-chip__val → var(--gradient-spectral) bg-clip text .th-score-bar__fill → var(--gradient-spectral) background .th-ring-arc--fill → var(--gradient-spectral) via SVG linearGradient .intel-subhd-title--spectral (inherited from _intel.scss) .any-badge → var(--color-warning) — ⚠ Any-pollution marker .score-zero → var(--color-danger) text — 0.0 scores .score-full → var(--color-success) text — 1.0 scores ```


Phase 6 — Wire up + final sweep

  • Register route in router (mirrors `intel_clones_page` registration pattern)
  • Add `/intel/type` to `INTEL_SUBPAGES` constant (used by dashboard card visibility check)
  • Confirm dashboard card always visible (even empty repo)
  • Run full test suite: all tiers green

Load-bearing order

``` Phase 1a ← migration 0013 (add return_annotation) Phase 1b ← ORM model update [depends on 1a] Phase 1c ← TypeProvider batch fix [depends on 1b] │ ▼ Phase 2 ← write ALL tests (TDD — confirm red) [depends on 1a–1c for T01–T05] │ ▼ Phase 3 ← Route handler [depends on Phase 2 tests driving the API] │ ▼ Phase 4 ← Templates [depends on Phase 3 context dict] Phase 5 ← SCSS [parallel with Phase 4] │ ▼ Phase 6 ← Wire up + sweep [depends on 3–5] │ ▼ ✅ All 30 tests green → commit → merge dev → push local → push staging → deploy ```


Acceptance criteria

  • Migration 0013 adds `return_annotation VARCHAR(256) NULLABLE` to `musehub_intel_type`
  • `TypeProvider` stores `return_annotation` and uses a single batch-upsert SQL call
  • `/intel/type` returns 200 with an empty repo (no rows)
  • Summary stats (total, fully_typed, partial, untyped, any_count, coverage_fraction) match `muse code type --json` output
  • Tier and kind filters work; default sort is score ASC (worst first)
  • Score bars and coverage ring use `var(--gradient-spectral)`
  • Any-pollution ⚠ badge appears on symbols with `return_is_any` or `params_with_any > 0`
  • Dashboard card always visible; shows coverage_pct when data exists
  • No `muse code` subprocess in the web request path
  • All 30 tests (T01–T30) passing across all 7 tiers
  • Staging verified: `/gabriel/musehub/intel/type` renders with live data

Notes

  • Coverage ring: pure SVG `<circle>` with `stroke-dashoffset` driven by `coverage_fraction`; no JS required
  • `return_annotation` truncated at 48 chars in table display; full value in `title` attribute
  • `params_annotated / params_total` shown as fraction (e.g. `3/4`); 0/0 renders as `—`
  • Summary stats are computed with four COUNT queries; no materialized view needed at this scale
  • TypeProvider batch fix: chunk at 1,000 rows (well under asyncpg 32,767 param limit at 9 params/row)
Activity2
gabriel opened this issue 48 days ago
gabriel 48 days ago

Done

All acceptance criteria met and verified on staging.

Phase 1 — DB + Provider

  • Migration 0013: return_annotation VARCHAR(256) NULLABLE added to musehub_intel_type
  • TypeProvider rewrote to pure DB/Python (no subprocess) — reads snapshot manifest from musehub_snapshots.manifest_blob, fetches source bytes via get_backend().get(), runs extract_annotations() (pure AST), batch-upserts in chunks of 1,000
  • return_annotation stored and included in upsert

Phase 2 — Tests

  • All 30 tests (T01–T30) passing across all 7 tiers

Phases 3–6 — UI

  • Route: summary stats (4 COUNT queries), tier/kind filters, score ASC default, top selector
  • Template: SVG coverage ring, 4 stat chips, filter bar, symbol table with score bars, return_annotation, any-pollution badges
  • SCSS: spectral gradient ring + bars, warning/danger/success tier colors
  • Dashboard card live with coverage_pct and symbol count

Staging

  • /gabriel/musehub/intel/type live with 8,505 symbols at 95.3% typed
gabriel 48 days ago

✅ Issue complete — live type coverage stats

Ran muse code type against the musehub repo:

Metric Value
Total symbols 8,505
Fully typed 8,107
Partial 380
Untyped 18
Any-polluted 3
Coverage 95.32%

All DB models, route handlers, background workers, and the new intel type provider are fully typed. The 380 partial entries are mostly test helpers and internal utilities. 18 untyped symbols are legacy stubs — candidates for the next type-hygiene pass.

The intel/type page is live on staging at /gabriel/musehub/intel/type and consuming this data in real time.