RETRIEVAL-AND-CLI-REFERENCE.md
file-level
1
files
1
commits
0
hotspots
0
🧊 dead
0
💥 blast risk
| 1 | # Retrieval, Token Cost, and CLI Reference |
| 2 | |
| 3 | This document (1) lists **all CLI commands and optional features**, (2) explains **how they interact** to reduce retrieval scope and token cost, (3) shows **how each feature helps the retrieval bottleneck**, and (4) suggests **expansions** so agents get the right information at the best price token-wise. |
| 4 | |
| 5 | --- |
| 6 | |
| 7 | ## 1. The retrieval bottleneck and token cost |
| 8 | |
| 9 | **Problem:** Agents that pull in 2000+ tokens for a simple question waste cost and context. The bottleneck is often **over-retrieval** — no way to be specific, so the system returns too much. |
| 10 | |
| 11 | **Approach:** Be **specific** about what the agent wants (filters, time, scope) and **control how much** comes back (limit, snippet size, fields only). Then the agent can do **tiered retrieval**: cheap query first (paths or count), then fetch full content only for what’s needed. |
| 12 | |
| 13 | --- |
| 14 | |
| 15 | ## 2. All CLI commands (single reference) |
| 16 | |
| 17 | | Command | Purpose | Key flags | |
| 18 | |--------|--------|-----------| |
| 19 | | **search** \<query\> | **Semantic** search over the indexed vault (default), or **keyword** search with `--keyword` (literal text: path, body, key frontmatter; no index required). | Same filters as list-notes: `--folder`, `--project`, `--tag`, `--since`, `--until`, `--chain`, `--entity`, `--episode`, `--content-scope` (`all` \| `notes` \| `approval_logs`), `--order`, `--limit` (default 10), `--fields`, `--snippet-chars`, `--count-only`, `--json`. **Keyword-only:** `--keyword`, `--match phrase` \| `all-terms`. JSON may include `"mode": "semantic"` \| `"keyword"`. | |
| 20 | | **list-notes** | List notes with filters (no semantic search). | `--folder`, `--project`, `--tag`, `--limit`, `--offset`, `--since`, `--until`, `--chain`, `--entity`, `--episode`, `--order`, `--fields`, `--count-only`, `--json` | |
| 21 | | **get-note** \<path\> | Return full content of one note (frontmatter + body), or subset. | `--body-only`, `--frontmatter-only`, `--json` | |
| 22 | | **get-note-outline** \<path\> | Return a derived Markdown heading outline for one note without note body text. | `--json` required | |
| 23 | | **get-document-tree** \<path\> | Return a derived nested Markdown heading tree for one note without note body text. | `--json` required | |
| 24 | | **get-metadata-facets** \<path\> | Return bounded metadata facets for one note without note body text or full frontmatter. | `--json` required | |
| 25 | | **index** | Re-run indexer: vault → chunk → embed → vector store. | (optional: `--json` for machine output) | |
| 26 | | **write** \<path\> [content] | Create or overwrite a note. | `--stdin`, `--frontmatter`, `--append`, `--json` | |
| 27 | | **export** \<path-or-query\> \<output\> | Export note(s) to a format/directory. | `--format`, `--project`, `--json` | |
| 28 | | **import** \<source-type\> \<input\> | Ingest from external platform (ChatGPT, Claude, etc.). | `--project`, `--output-dir`, `--tags`, `--dry-run`, `--json` | |
| 29 | | **doctor** | Check **self-hosted** vault path (config load, disk readable) and optional **Hub** probes (`KNOWTATION_HUB_URL` health; `KNOWTATION_HUB_TOKEN` + `KNOWTATION_HUB_VAULT_ID` → `GET /api/v1/notes?limit=1`). Prints token discipline context from [TOKEN-SAVINGS.md](./TOKEN-SAVINGS.md). | `--json`, `--hub <url>` | |
| 30 | |
| 31 | **Global:** All commands support `--json` for machine-readable output. |
| 32 | |
| 33 | --- |
| 34 | |
| 35 | ## 3. Add-on / optional features (and where they plug in) |
| 36 | |
| 37 | | Feature | Where it plugs in | What it does for retrieval / tokens | |
| 38 | |--------|--------------------|--------------------------------------| |
| 39 | | **Project / tag / folder filters** | `search`, `list-notes` | Shrink scope to one project, tag, or folder → fewer results, fewer tokens. | |
| 40 | | **Time filters** (`--since`, `--until`) | `search`, `list-notes` | Restrict to a date range → only relevant period, fewer tokens. | |
| 41 | | **Causal / entity / episode** (`--chain`, `--entity`, `--episode`) | `search`, `list-notes` | Return only one chain, entity, or episode → precise set, fewer tokens. | |
| 42 | | **Limit and offset** (`--limit`, `--offset`) | `search`, `list-notes` | Cap how many items return; paginate. Direct token control. | |
| 43 | | **Order** (`--order date \| date-asc`) | `search`, `list-notes` | Get “newest first” or “oldest first” without over-fetching. | |
| 44 | | **Optional memory layer** (Mem0, etc.) | After search/export | Store “last query + result set” or provenance; agent can ask memory for “last run” instead of re-querying. Saves repeat retrieval. | |
| 45 | | **Optional AIR** | Before write/export | Attestation only; no direct token impact on retrieval. | |
| 46 | | **Proposals** | Hub / propose flow | Proposals live outside main vault until committed; agent can list/diff proposals without loading full vault. | |
| 47 | | **Summary notes / state snapshots** | Vault content + filters | Notes with `summarizes` or `state_snapshot` compress a range; agent retrieves one summary instead of many notes. Big token saving for long horizons. | |
| 48 | | **Optional hub API** (shared vault / hub) | Same as CLI | Same filters and limits; same token benefits. | |
| 49 | |
| 50 | --- |
| 51 | |
| 52 | ## 4. How they work together (interaction) |
| 53 | |
| 54 | ``` |
| 55 | Agent needs an answer |
| 56 | │ |
| 57 | ▼ |
| 58 | ┌─────────────────────────────────────────────────────────────────┐ |
| 59 | │ 1. NARROW SCOPE (filters) │ |
| 60 | │ search / list-notes with --project, --tag, --since, --until, │ |
| 61 | │ --chain, --entity, --episode, --limit │ |
| 62 | │ → Fewer candidates → fewer tokens in the response. │ |
| 63 | └─────────────────────────────────────────────────────────────────┘ |
| 64 | │ |
| 65 | ▼ |
| 66 | ┌─────────────────────────────────────────────────────────────────┐ |
| 67 | │ 2. CHEAP FIRST (paths or count) │ |
| 68 | │ list-notes --limit 5 --json → get 5 paths + metadata only │ |
| 69 | │ or search with --limit 3 --json → get 3 path + snippet │ |
| 70 | │ → Agent decides from paths/snippets whether to fetch full. │ |
| 71 | └─────────────────────────────────────────────────────────────────┘ |
| 72 | │ |
| 73 | ▼ |
| 74 | ┌─────────────────────────────────────────────────────────────────┐ |
| 75 | │ 3. CHECK STRUCTURE OR METADATA BEFORE FULL BODY │ |
| 76 | │ get-note-outline <path> --json for promising notes │ |
| 77 | │ or get-document-tree <path> --json when hierarchy matters │ |
| 78 | │ or get-metadata-facets <path> --json for metadata hints │ |
| 79 | │ → Decide whether the note structure or facets are relevant. │ |
| 80 | └─────────────────────────────────────────────────────────────────┘ |
| 81 | │ |
| 82 | ▼ |
| 83 | ┌─────────────────────────────────────────────────────────────────┐ |
| 84 | │ 4. FETCH ONLY WHAT’S NEEDED │ |
| 85 | │ get-note <path> only for the 1–2 paths that matter │ |
| 86 | │ → Full body only where necessary → minimal tokens. │ |
| 87 | └─────────────────────────────────────────────────────────────────┘ |
| 88 | ``` |
| 89 | |
| 90 | **Typical flows:** |
| 91 | |
| 92 | - **“What did we decide about X?”** → `search "decisions about X" --entity X --limit 5 --json` → then `get-note` for the top 1–2 paths. Filters + limit + delayed full fetch = low tokens. |
| 93 | - **“What happened in March?”** → `list-notes --since 2025-03-01 --until 2025-03-31 --limit 20 --json` → then `get-note` for selected paths. Time + limit = bounded tokens. |
| 94 | - **“Which part of this long note matters?”** → `get-note-outline path/to/note.md --json` or `get-document-tree path/to/note.md --json` → then `get-note` only if the heading structure confirms the note is relevant. |
| 95 | - **“Is this note in the right project, tag, or chain?”** → `get-metadata-facets path/to/note.md --json` → then `get-note` only if bounded facets confirm it is relevant. |
| 96 | - **“What’s the chain that led to Y?”** → `search "outcome Y" --chain chain-id --order date-asc --limit 10 --json` → then `get-note` for the chain. One chain, ordered, minimal fetch. |
| 97 | |
| 98 | --- |
| 99 | |
| 100 | ## 5. How each feature helps the bottleneck (and token cost) |
| 101 | |
| 102 | | Feature | Bottleneck it eases | Token effect | |
| 103 | |--------|---------------------|--------------| |
| 104 | | **--project, --tag, --folder** | Reduces candidate set to a subset of the vault. | Fewer results → smaller JSON → less in context. | |
| 105 | | **--since, --until** | Restricts to a time window; no need to scan all time. | Same as above; especially strong for long-lived vaults. | |
| 106 | | **--chain, --entity, --episode** | Returns only one causal chain, entity, or episode. | One logical set instead of many unrelated notes. | |
| 107 | | **--limit** | Hard cap on number of items returned. | Direct: 5 results ≈ 5× (snippet or path) size; 50 results ≈ 10× more tokens. | |
| 108 | | **--offset** | Pagination; agent can ask for “next page” instead of “everything.” | Avoids one huge response. | |
| 109 | | **--order** | Get “newest” or “oldest” first without fetching extra. | Agent can stop after first page when order is meaningful. | |
| 110 | | **Snippet in search** | Return a short snippet per hit, not full body. | Snippet << full note; agent uses snippet to choose which path to get-note. | |
| 111 | | **list-notes (no body)** | Returns path + metadata only, no body. | Very small per note; agent then get-note only for chosen paths. | |
| 112 | | **get-note-outline (no body)** | Shows note structure without body text or snippets. | Lets an agent reject irrelevant long notes before fetching full content. Future nested tree planning lives in [DOCUMENT-TREE-V0-SPEC.md](./DOCUMENT-TREE-V0-SPEC.md). | |
| 113 | | **get-document-tree (no body)** | Shows nested heading hierarchy without body text or snippets. | Helps an agent understand parent/child heading relationships before fetching full content. | |
| 114 | | **get-metadata-facets (no body)** | Shows bounded metadata facets without body text or full frontmatter. | Helps an agent filter by project, tags, date, causal chain, entity, and episode before fetching full content. | |
| 115 | | **Summary notes / state_snapshot** | One note summarizes many or a range. | Retrieve 1 summary instead of N notes → large token saving. | |
| 116 | | **Memory layer** | “Last query + result” or “provenance for export” stored. | Agent can ask memory “what did we last retrieve?” instead of re-running search. | |
| 117 | |
| 118 | --- |
| 119 | |
| 120 | ## 6. Token-optimal retrieval (in spec) |
| 121 | |
| 122 | The following are **in scope** and specified in SPEC §4.1–4.2. |
| 123 | |
| 124 | | Expansion | What it does | Token impact | |
| 125 | |-----------|--------------|--------------| |
| 126 | | **--fields** for search/list-notes | Control what each result contains: `path` only, `path+snippet` (search) or `path+metadata` (list-notes), or `full`. | `--fields path` → minimal payload; agent then get-note only for chosen paths. | |
| 127 | | **--snippet-chars** | Cap snippet length (e.g. 200 chars) for search. | Shorter snippets → smaller search output. | |
| 128 | | **--count-only** for search and list-notes | Return only total count (and optionally first N paths); no snippets or bodies. | Agent asks “how many?” first, then decides whether to run a full search with limit. | |
| 129 | | **Tiered retrieval in SKILL / docs** | Document pattern: (1) list-notes or search with small limit + --json, (2) from paths/snippets pick 1–2, (3) get-note only those. | Teaches agents to use minimal tokens by design. | |
| 130 | | **--body-only / --frontmatter-only** (get-note) | Return only body or only frontmatter for one note. | Saves tokens when the other part is large. | |
| 131 | | **Prefer state_snapshot / summarizes** | When present, search or a dedicated “get summary for range” returns summary notes first. | One summary in context instead of many raw notes (optional/future). | |
| 132 | |
| 133 | **Spec (SPEC §4.1–4.2):** |
| 134 | |
| 135 | - **search:** `--fields path|path+snippet|full` (default path+snippet). `--snippet-chars <n>`. `--count-only` → `{ "count": n, "query": "..." }`. **`--keyword`** → literal text search; JSON includes `"mode": "keyword"` (and `"mode": "semantic"` when using the default path in Hub/repo CLI). |
| 136 | - **list-notes:** `--fields path|path+metadata|full` (default path+metadata). `--count-only` → `{ "total": number }`. |
| 137 | - **get-note:** `--body-only` or `--frontmatter-only` for a single note; omit both for full content. |
| 138 | - **get-note-outline:** `--json` returns a heading-only `knowtation.note_outline/v1` payload for one note. |
| 139 | - **get-document-tree:** `--json` returns a nested heading-only `knowtation.document_tree/v0` payload for one note. |
| 140 | - **get-metadata-facets:** `--json` returns a bounded body-free `knowtation.metadata_facets/v0` payload for one note. |
| 141 | |
| 142 | --- |
| 143 | |
| 144 | ## 7. Single table: commands + filters + token levers |
| 145 | |
| 146 | | Command | Scope filters | Order / limit | Token levers (current) | Token levers (in spec) | |
| 147 | |--------|----------------|---------------|------------------------|--------------------------| |
| 148 | | **search** | folder, project, tag, since, until, chain, entity, episode | limit, order | limit, snippet in JSON | **In spec:** --fields, --snippet-chars, --count-only | |
| 149 | | **list-notes** | folder, project, tag, since, until, chain, entity, episode | limit, offset, order | limit, offset, no body | **In spec:** --fields, --count-only | |
| 150 | | **get-note** | (one path) | — | — | **In spec:** --body-only, --frontmatter-only | |
| 151 | | **get-note-outline** | (one path) | — | no body, no snippets | **In spec:** --json heading outline | |
| 152 | | **get-document-tree** | (one path) | — | no body, no snippets | **In spec:** --json nested heading tree | |
| 153 | | **get-metadata-facets** | (one path) | — | no body, no full frontmatter | **In spec:** --json bounded metadata facets | |
| 154 | | **index** | — | — | — | — | |
| 155 | | **write** | — | — | — | — | |
| 156 | | **export** | project | — | — | — | |
| 157 | | **import** | project, output-dir, tags | — | — | — | |
| 158 | |
| 159 | --- |
| 160 | |
| 161 | ## 8. Summary |
| 162 | |
| 163 | - **All commands:** search, list-notes, get-note, get-note-outline, get-document-tree, get-metadata-facets, index, write, export, import. Filters (project, tag, folder, since, until, chain, entity, episode) and limit/offset/order apply to search and list-notes. |
| 164 | - **Add-ons:** Project/tag/time/chain/entity/episode filters, limit/offset, order, optional memory, summary notes/state snapshots, hub API. Each either narrows scope or reduces payload size. |
| 165 | - **Interaction:** Narrow with filters → cheap first (paths or count) → get-note-outline, get-document-tree, or get-metadata-facets for body-free inspection → get-note only for selected paths. That pattern minimizes tokens. |
| 166 | - **Token levers (in spec):** `--fields`, `--snippet-chars`, `--count-only` for search/list-notes; `--body-only`/`--frontmatter-only` for get-note; `get-note-outline --json` and `get-document-tree --json` for heading-only structure; `get-metadata-facets --json` for bounded metadata hints; tiered retrieval documented in SKILL. These give agents the knobs to get the right information at the best price token-wise. |