> ## Documentation Index
> Fetch the complete documentation index at: https://docs.axiomancer.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Chunking policy

> How Codex chunks records for RAG — structure-aware boundaries on agenda items, speaker turns, and permit scopes with deterministic IDs.

Codex chunks records along structural boundaries (agenda items, speaker turns, permit scopes) rather than fixed token windows. This preserves semantic context and improves retrieval accuracy by 20–40% compared to naive fixed-size chunking on government and legal text.

## Principles

1. **Respect source structure.** Chunk at sections, speaker turns, or clause boundaries. Fall back to token-window chunking only when no structure exists.
2. **One chunk = one contract obligation.** Permit conditions, lease clauses, and blocker tags each become their own chunk.
3. **Parent-child friendly.** Every chunk carries `doc_id` + `section_id` so retrievers can fetch the parent document when a chunk matches.
4. **Bounded size.** Target 300–800 tokens for narrative sections; hard cap 1,500 tokens with intelligent split.
5. **Overlap only when needed.** Zero overlap for structured splits. Fixed 50-token overlap only for token-window fallback.
6. **Deterministic IDs.** `chunk_id` is SHA-256-derived via `make_chunk_id(doc_id, section_id)` — re-running the chunker on unchanged text produces identical IDs, enabling incremental vector-index updates.

## Chunk schema

Every chunk record has this shape, written to a per-dataset `_chunks` Parquet file alongside the main dataset export.

| Field               | Type      | Description                                                               |
| ------------------- | --------- | ------------------------------------------------------------------------- |
| `chunk_id`          | URN       | `urn:aprs:chunk:{16-hex}` — SHA-256 of `doc_id` + section. Deterministic. |
| `doc_id`            | URN       | Parent record's APRS URN                                                  |
| `section_id`        | string    | Stable section identifier within the document                             |
| `chunk_text`        | text      | The chunk content                                                         |
| `chunk_type`        | enum      | See [chunk types](#chunk-types) below                                     |
| `token_count`       | int       | Token count (tiktoken cl100k\_base)                                       |
| `char_count`        | int       | Character count                                                           |
| `seq_index`         | int       | 0-based sequential index within the document                              |
| `parent_chunk_id`   | URN       | Null for top-level chunks; set for subsections                            |
| `evidence_anchor`   | JSON      | `{page, char_span, xpath?}` for source verification                       |
| `embedding_model`   | string    | Embedding model used (e.g. `text-embedding-3-large`)                      |
| `embedding_version` | string    | Model version or fingerprint                                              |
| `chunked_at`        | timestamp | When chunking occurred                                                    |
| `chunking_version`  | string    | Version of the chunking policy applied                                    |

## Per-dataset chunking rules

### Civic Intelligence

Source material: council transcripts, agenda items, meeting minutes.

Chunk at, in priority order:

1. **Agenda item boundaries** — each item is its own chunk tree. `section_id = "item.{agenda_item_number}"`.
2. **Speaker-turn boundaries** — inside an agenda item, chunk per speaker turn. `section_id = "item.{n}.turn.{m}"`. Turns under 50 tokens are merged with the next turn from the same speaker.
3. **Blocker/condition enumeration** — each blocker tag or condition becomes its own chunk with `chunk_type=condition`.
4. **Contingency DAG nodes** — each node becomes a `chunk_type=contingency` chunk.

The parent chunk for an agenda item contains the item summary; each speaker-turn, condition, and contingency is a child with `parent_chunk_id` set.

### Events Timeline

Each event is a chunk. `section_id` is the event's own ID. `chunk_type=event`.

Long event descriptions (e.g. `dark_event` with extensive kinematic detail) split at 800 tokens with 50-token overlap. All splits share the same `parent_chunk_id`.

### Permit Signals

Two-level tree:

1. **Permit parent** — summary plus APRS envelope. `chunk_type=permit_summary`.
2. **Scope sections** — each LLM-extracted scope section (e.g. "Floor 1 addition", "Mechanical — HVAC") becomes its own chunk with `chunk_type=permit_scope_section`.

Each condition in the permit (e.g. "Subject to ADA variance approval") is an independent chunk with `chunk_type=condition` — they are frequently cited in isolation, so joining them with their scope section would hurt retrieval.

### OSHA Safety

Two-level tree:

1. **Case parent** — `chunk_type=osha_case_summary`.
2. **Citation sections** — one per citation, since OSHA citations are independently enforceable. `chunk_type=osha_citation`.

### AIS Maritime Positions

Not chunked. Positions are point-in-time records — one row equals one record. No `_chunks` file for this dataset.

### Urban Signal Grid, LEHD Flows, POI Intelligence

Structured-only datasets. The `llm_text` Markdown-KV view is the text surface and is already compact (under 400 tokens per record). Each record maps to one chunk with `chunk_type=record_view`.

## Chunk types

| Type                   | Description                                              |
| ---------------------- | -------------------------------------------------------- |
| `event`                | Single event row (Events Timeline)                       |
| `record_view`          | Markdown-KV view of a structured record (USG, LEHD, POI) |
| `agenda_item`          | Top-level civic agenda item                              |
| `speaker_turn`         | Civic transcript speaker turn                            |
| `condition`            | Enforceable condition, blocker, or covenant              |
| `contingency`          | DAG node in a civic contingency chain                    |
| `permit_summary`       | Permit root chunk                                        |
| `permit_scope_section` | Scope subsection of a permit                             |
| `osha_case_summary`    | OSHA case root                                           |
| `osha_citation`        | Individual OSHA citation                                 |
| `narrative_window`     | Fallback token-window chunk                              |

## Token-window fallback

When a source has no identifiable structure (rare; mostly raw scraped HTML), Codex falls back to sliding-window chunking:

* **Window:** 600 tokens
* **Stride:** 550 tokens (50-token overlap)
* **Chunk type:** `narrative_window`
* **Section ID:** `"window.{seq_index}"`

Overlap is zero for all structured chunks — duplication across chunks degrades retrieval precision.

## Parent-child retrieval

You can use the chunk tree directly with parent-child retrieval frameworks like LangChain or LlamaIndex `ParentDocumentRetriever`:

```sql theme={null}
-- Retrieve a chunk's parent, then re-score at parent scope
WITH hit AS (
  SELECT chunk_id, doc_id, parent_chunk_id, chunk_text
  FROM codex.civic_intelligence_chunks
  WHERE chunk_id = :hit_chunk_id
)
SELECT parent.chunk_text
FROM hit
JOIN codex.civic_intelligence_chunks parent
  ON parent.chunk_id = hit.parent_chunk_id;
```

## Cross-encoder re-ranking

Initial retrieval over chunk embeddings is coarse. The recommended post-retrieval step is a cross-encoder re-rank over the top 50 results. Score pairs: `(query, chunk_text)`. Drop chunks below a `score_threshold` of 0.3.

<Note>
  Codex does not run the re-ranker — this is a consumer-side pattern. The chunk structure (bounded size, clean boundaries) is designed to be compatible with standard re-ranking pipelines.
</Note>

## Embedding policy

* **Default model:** `text-embedding-3-large` (OpenAI), 3072 dimensions, reduced to 1536 for storage.
* **Re-embedding triggers:** model version bump or `chunking_version` bump. Embeddings are tagged with `embedding_model` and `embedding_version` so you can decide whether to re-embed.
* **Deterministic re-embedding:** because `chunk_id` is deterministic, re-embedding the same text produces the same `chunk_id`. Embedding rows update in place without orphans.