Skip to main content
Codex chunks records along structural boundaries (agenda items, speaker turns, permit scopes) rather than fixed token windows. This preserves semantic context and improves retrieval accuracy by 20–40% compared to naive fixed-size chunking on government and legal text.

​
Principles

  1. Respect source structure. Chunk at sections, speaker turns, or clause boundaries. Fall back to token-window chunking only when no structure exists.
  2. One chunk = one contract obligation. Permit conditions, lease clauses, and blocker tags each become their own chunk.
  3. Parent-child friendly. Every chunk carries doc_id + section_id so retrievers can fetch the parent document when a chunk matches.
  4. Bounded size. Target 300–800 tokens for narrative sections; hard cap 1,500 tokens with intelligent split.
  5. Overlap only when needed. Zero overlap for structured splits. Fixed 50-token overlap only for token-window fallback.
  6. Deterministic IDs. chunk_id is SHA-256-derived via make_chunk_id(doc_id, section_id) — re-running the chunker on unchanged text produces identical IDs, enabling incremental vector-index updates.

​
Chunk schema

Every chunk record has this shape, written to a per-dataset _chunks Parquet file alongside the main dataset export.

​
Per-dataset chunking rules

​
Civic Intelligence

Source material: council transcripts, agenda items, meeting minutes. Chunk at, in priority order:
  1. Agenda item boundaries — each item is its own chunk tree. section_id = "item.{agenda_item_number}".
  2. Speaker-turn boundaries — inside an agenda item, chunk per speaker turn. section_id = "item.{n}.turn.{m}". Turns under 50 tokens are merged with the next turn from the same speaker.
  3. Blocker/condition enumeration — each blocker tag or condition becomes its own chunk with chunk_type=condition.
  4. Contingency DAG nodes — each node becomes a chunk_type=contingency chunk.
The parent chunk for an agenda item contains the item summary; each speaker-turn, condition, and contingency is a child with parent_chunk_id set.

​
Events Timeline

Each event is a chunk. section_id is the event’s own ID. chunk_type=event. Long event descriptions (e.g. dark_event with extensive kinematic detail) split at 800 tokens with 50-token overlap. All splits share the same parent_chunk_id.

​
Permit Signals

Two-level tree:
  1. Permit parent — summary plus APRS envelope. chunk_type=permit_summary.
  2. Scope sections — each LLM-extracted scope section (e.g. “Floor 1 addition”, “Mechanical — HVAC”) becomes its own chunk with chunk_type=permit_scope_section.
Each condition in the permit (e.g. “Subject to ADA variance approval”) is an independent chunk with chunk_type=condition — they are frequently cited in isolation, so joining them with their scope section would hurt retrieval.

​
OSHA Safety

Two-level tree:
  1. Case parent — chunk_type=osha_case_summary.
  2. Citation sections — one per citation, since OSHA citations are independently enforceable. chunk_type=osha_citation.

​
AIS Maritime Positions

Not chunked. Positions are point-in-time records — one row equals one record. No _chunks file for this dataset.

​
Urban Signal Grid, LEHD Flows, POI Intelligence

Structured-only datasets. The llm_text Markdown-KV view is the text surface and is already compact (under 400 tokens per record). Each record maps to one chunk with chunk_type=record_view.

​
Chunk types

​
Token-window fallback

When a source has no identifiable structure (rare; mostly raw scraped HTML), Codex falls back to sliding-window chunking:
  • Window: 600 tokens
  • Stride: 550 tokens (50-token overlap)
  • Chunk type: narrative_window
  • Section ID: "window.{seq_index}"
Overlap is zero for all structured chunks — duplication across chunks degrades retrieval precision.

​
Parent-child retrieval

You can use the chunk tree directly with parent-child retrieval frameworks like LangChain or LlamaIndex ParentDocumentRetriever:

​
Cross-encoder re-ranking

Initial retrieval over chunk embeddings is coarse. The recommended post-retrieval step is a cross-encoder re-rank over the top 50 results. Score pairs: (query, chunk_text). Drop chunks below a score_threshold of 0.3.
Codex does not run the re-ranker — this is a consumer-side pattern. The chunk structure (bounded size, clean boundaries) is designed to be compatible with standard re-ranking pipelines.

​
Embedding policy

  • Default model: text-embedding-3-large (OpenAI), 3072 dimensions, reduced to 1536 for storage.
  • Re-embedding triggers: model version bump or chunking_version bump. Embeddings are tagged with embedding_model and embedding_version so you can decide whether to re-embed.
  • Deterministic re-embedding: because chunk_id is deterministic, re-embedding the same text produces the same chunk_id. Embedding rows update in place without orphans.