> ## Documentation Index
> Fetch the complete documentation index at: https://docs.axiomancer.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Reference notebooks

> Jupyter notebooks for upzoning classification and civic risk mapping using Codex research-tier datasets on Hugging Face — CC-BY-4.0, no paid tier required.

Codex ships two reference notebooks that demonstrate real analytical workflows using the free research-tier datasets on Hugging Face. Both notebooks are licensed CC-BY-4.0, pin to the `2026-04` snapshot, and require no paid tier.

<Columns cols={2}>
  <Card title="Upzoning classifier" icon="brain" href="#upzoning-classifier">
    Fine-tune DistilBERT on council meeting language to predict zoning outcomes, then benchmark against Codex's pre-computed scores.
  </Card>

  <Card title="Civic risk map" icon="map" href="#civic-risk-map">
    Build an H3 choropleth of litigation risk and test whether elevated risk predicts slower permit issuance.
  </Card>
</Columns>

## Prerequisites

Both notebooks require Python 3.10+ and the following core dependencies:

```txt theme={null}
datasets
transformers
pandas
numpy
matplotlib
h3
plotly
scipy
```

The upzoning classifier additionally requires `torch`, `accelerate`, `evaluate`, and `scikit-learn`. The civic risk map additionally requires `geopandas` and `shapely`.

<Tip>
  You do not need a paid Codex tier. Both notebooks load the 100K-record research sample directly from Hugging Face.
</Tip>

## Upzoning classifier

**Goal:** Fine-tune a DistilBERT model (\~66M parameters, \~10 minutes on a single GPU) on the `language_signals` and `summary` fields from Civic Intelligence to predict upzoning outcomes (approved, denied, or continued), then compare against Codex's pre-computed `upzoning_probability`.

**Datasets used:** [Civic Intelligence](/codex/schemas/civic-intelligence) (research tier)

### Workflow

<Steps>
  <Step title="Load the research-tier sample">
    Load the Civic Intelligence dataset from Hugging Face, pinned to the `2026-04` snapshot:

    ```python theme={null}
    from datasets import load_dataset

    ds = load_dataset(
        "axiom-ai/civic-intelligence",
        revision="snapshot/2026-04",
        split="train"
    )
    ```
  </Step>

  <Step title="Filter to zoning records">
    Keep only `zoning_vote` and `rezoning_hearing` records with non-null outcomes for training:

    ```python theme={null}
    zoning = ds.filter(
        lambda r: r["document_type"] in ("zoning_vote", "rezoning_hearing")
        and r["upzoning_probability"] is not None
    )
    ```
  </Step>

  <Step title="Build input features">
    Concatenate the top 30 `language_signals` phrases with the `summary` field to create a text input for the classifier:

    ```python theme={null}
    def build_text(row):
        signals = " | ".join((row["language_signals"] or [])[:30])
        return f"signals: {signals}\nsummary: {row['summary'] or ''}"
    ```
  </Step>

  <Step title="Split by jurisdiction">
    Use a jurisdiction-based train/test split to test generalization — train on Philadelphia, Chicago, and Dallas; evaluate on San Francisco and New York:

    ```python theme={null}
    train_jurisdictions = {"philadelphia-pa", "chicago-il", "dallas-tx"}
    test_jurisdictions = {"san-francisco-ca", "new-york-ny"}
    ```
  </Step>

  <Step title="Fine-tune DistilBERT">
    Train for 3 epochs with batch size 16, learning rate 5e-5, and macro-F1 as the evaluation metric.
  </Step>

  <Step title="Compare against Codex scores">
    Evaluate your trained classifier against Codex's pre-computed `upzoning_probability` using ROC-AUC on the held-out jurisdictions.
  </Step>
</Steps>

### What you learn

* How to load and filter Codex datasets from Hugging Face
* How `language_signals` and `summary` power zoning outcome prediction
* How Codex's pre-computed scores compare to a custom fine-tuned model
* How jurisdiction-based splits reveal geographic generalization gaps

## Civic risk map

**Goal:** Build an H3-cell choropleth of civic litigation risk and test whether elevated risk predicts depressed permit-issuance velocity in the following 6 months.

**Datasets used:** [Civic Intelligence](/codex/schemas/civic-intelligence), Urban Signal Grid, Permit Signals (all research tier)

### Workflow

<Steps>
  <Step title="Load three datasets">
    Load the research-tier samples for Civic Intelligence, Urban Signal Grid, and Permit Signals from Hugging Face:

    ```python theme={null}
    from datasets import load_dataset

    civic = load_dataset("axiom-ai/civic-intelligence", revision="snapshot/2026-04", split="train")
    usg = load_dataset("axiom-ai/urban-signal-grid", revision="snapshot/2026-04", split="train")
    permits = load_dataset("axiom-ai/permit-signals", revision="snapshot/2026-04", split="train")
    ```
  </Step>

  <Step title="Filter to a focus metro">
    Filter all three datasets to a single metro (e.g., `chicago-il`) for tractable analysis.
  </Step>

  <Step title="Aggregate risk to H3 cells">
    Compute a confidence-weighted mean of `litigation_risk_score` per `h3_index` from the Civic Intelligence dataset:

    ```python theme={null}
    import pandas as pd

    civic_df = civic.to_pandas()
    risk_by_cell = (
        civic_df
        .groupby("h3_index")
        .apply(lambda g: (g["litigation_risk_score"] * g["confidence_score"]).sum() / g["confidence_score"].sum())
        .rename("weighted_risk")
    )
    ```
  </Step>

  <Step title="Join to Urban Signal Grid">
    Join the aggregated risk scores to the Urban Signal Grid on `h3_index`:

    ```python theme={null}
    usg_df = usg.to_pandas()
    joined = usg_df.merge(risk_by_cell, on="h3_index", how="left")
    ```
  </Step>

  <Step title="Render an interactive choropleth">
    Materialize H3 hexagon polygons and render a Plotly choropleth colored by litigation risk score.
  </Step>

  <Step title="Test the risk-to-permits hypothesis">
    Compute permit velocity (permits issued per cell in the following 6 months) and run a Spearman correlation test against litigation risk:

    ```python theme={null}
    from scipy.stats import spearmanr

    rho, p = spearmanr(merged["weighted_risk"], merged["permit_velocity"])
    ```
  </Step>
</Steps>

### What you learn

* How all three datasets join on `h3_index` alone — no spatial library required
* How to aggregate record-level scores to the H3 cell grid
* How civic proceedings signal downstream development activity
* How to validate Codex scores against observable permit outcomes

## Key takeaway

Both notebooks demonstrate the core Codex value proposition: datasets that join by construction, carry pre-computed AI labels, and enable cross-domain analysis without custom ETL. Start with the research tier on Hugging Face and upgrade to Commercial when you need full coverage.