Retrieval, Token Cost, and CLI Reference

This document (1) lists all CLI commands and optional features, (2) explains how they interact to reduce retrieval scope and token cost, (3) shows how each feature helps the retrieval bottleneck, and (4) suggests expansions so agents get the right information at the best price token-wise.

1. The retrieval bottleneck and token cost

Problem: Agents that pull in 2000+ tokens for a simple question waste cost and context. The bottleneck is often over-retrieval — no way to be specific, so the system returns too much.

Approach: Be specific about what the agent wants (filters, time, scope) and control how much comes back (limit, snippet size, fields only). Then the agent can do tiered retrieval: cheap query first (paths or count), then fetch full content only for what’s needed.

2. All CLI commands (single reference)

Global: All commands support --json for machine-readable output.

3. Add-on / optional features (and where they plug in)

4. How they work together (interaction)

Agent needs an answer
        │
        ▼
┌─────────────────────────────────────────────────────────────────┐
│ 1. NARROW SCOPE (filters)                                        │
│    search / list-notes with --project, --tag, --since, --until,   │
│    --chain, --entity, --episode, --limit                         │
│    → Fewer candidates → fewer tokens in the response.            │
└─────────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────────┐
│ 2. CHEAP FIRST (paths or count)                                  │
│    list-notes --limit 5 --json  → get 5 paths + metadata only    │
│    or search with --limit 3 --json → get 3 path + snippet        │
│    → Agent decides from paths/snippets whether to fetch full.     │
└─────────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────────┐
│ 3. CHECK STRUCTURE OR METADATA BEFORE FULL BODY                  │
│    get-note-outline <path> --json for promising notes             │
│    or get-document-tree <path> --json when hierarchy matters      │
│    or get-metadata-facets <path> --json for metadata hints        │
│    → Decide whether the note structure or facets are relevant.    │
└─────────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────────┐
│ 4. FETCH ONLY WHAT’S NEEDED                                      │
│    get-note <path> only for the 1–2 paths that matter             │
│    → Full body only where necessary → minimal tokens.            │
└─────────────────────────────────────────────────────────────────┘

Typical flows:

• “What did we decide about X?” → search "decisions about X" --entity X --limit 5 --json → then get-note for the top 1–2 paths. Filters + limit + delayed full fetch = low tokens.
• “What happened in March?” → list-notes --since 2025-03-01 --until 2025-03-31 --limit 20 --json → then get-note for selected paths. Time + limit = bounded tokens.
• “Which part of this long note matters?” → get-note-outline path/to/note.md --json or get-document-tree path/to/note.md --json → then get-note only if the heading structure confirms the note is relevant.
• “Is this note in the right project, tag, or chain?” → get-metadata-facets path/to/note.md --json → then get-note only if bounded facets confirm it is relevant.
• “What’s the chain that led to Y?” → search "outcome Y" --chain chain-id --order date-asc --limit 10 --json → then get-note for the chain. One chain, ordered, minimal fetch.

5. How each feature helps the bottleneck (and token cost)

6. Token-optimal retrieval (in spec)

The following are in scope and specified in SPEC §4.1–4.2.

Spec (SPEC §4.1–4.2):

• search: --fields path|path+snippet|full (default path+snippet). --snippet-chars <n>. --count-only → { "count": n, "query": "..." }. --keyword → literal text search; JSON includes "mode": "keyword" (and "mode": "semantic" when using the default path in Hub/repo CLI).
• list-notes: --fields path|path+metadata|full (default path+metadata). --count-only → { "total": number }.
• get-note: --body-only or --frontmatter-only for a single note; omit both for full content.
• get-note-outline: --json returns a heading-only knowtation.note_outline/v1 payload for one note.
• get-document-tree: --json returns a nested heading-only knowtation.document_tree/v0 payload for one note.
• get-metadata-facets: --json returns a bounded body-free knowtation.metadata_facets/v0 payload for one note.

7. Single table: commands + filters + token levers

8. Summary

• All commands: search, list-notes, get-note, get-note-outline, get-document-tree, get-metadata-facets, index, write, export, import. Filters (project, tag, folder, since, until, chain, entity, episode) and limit/offset/order apply to search and list-notes.
• Add-ons: Project/tag/time/chain/entity/episode filters, limit/offset, order, optional memory, summary notes/state snapshots, hub API. Each either narrows scope or reduces payload size.
• Interaction: Narrow with filters → cheap first (paths or count) → get-note-outline, get-document-tree, or get-metadata-facets for body-free inspection → get-note only for selected paths. That pattern minimizes tokens.
• Token levers (in spec): --fields, --snippet-chars, --count-only for search/list-notes; --body-only/--frontmatter-only for get-note; get-note-outline --json and get-document-tree --json for heading-only structure; get-metadata-facets --json for bounded metadata hints; tiered retrieval documented in SKILL. These give agents the knobs to get the right information at the best price token-wise.