Entity resolution

Axiom Overwatch resolves real-world entities — companies, vessel owners, sanctioned parties, permit applicants — across noisy data sources by running probabilistic record-linkage with Splink. High-probability candidate pairs are persisted to entity_resolution_pairs and an analyst either confirms, rejects, or marks them as unsure through the adjudication API before any downstream consumer treats them as the same entity. This is the foundation for investigation case-file consolidation, sanctions screening across naming variants, and UBO chain stitching where a single owner appears under spelling, jurisdiction, or registration-number differences in upstream feeds.

When to use it

Reach for entity resolution when you need to answer “is this the same company / owner / party as that one?” across heterogeneous records — not “does this string match that string.” Typical use cases:

Collapsing duplicate companies across LEI, OpenSanctions, Equasis, and registry feeds before an investigation case file is built
Stitching ownership chains where one party appears under multiple names, transliterations, or registration numbers
De-duplicating permit applicants before sanctions or UBO screening
Producing an adjudication worklist for analyst review when automated matching is not safe to act on alone

If you only need a deterministic join on a known key — IMO, MMSI, LEI, record_id — use the standard Codex join keys instead. Probabilistic resolution is for the cases where deterministic keys are missing, inconsistent, or contested.

How resolution works

For each input table, Splink streams every candidate pair through a settings-driven scorer: blocking rules narrow the candidate space, comparison vectors score the surviving pairs, and an unsupervised expectation-maximisation training step learns the per-field match weights without labelled examples.

Snapshot input

The linker reads from a Parquet snapshot of the source table, never from live Supabase. This keeps training reproducible, lets a model version be re-trained against a fixed input, and prevents long-running linkage jobs from contending with operational reads.

Block candidate pairs

Blocking rules narrow the O(n²) pair space to a tractable candidate set. The companies linker blocks on a 4-character name prefix, jurisdiction plus first letter of the name, and shared non-null LEI code.

Score with comparison vectors

Each surviving pair is scored across the comparison vector — name (Jaro–Winkler at 0.9 and 0.7), jurisdiction (exact match with term-frequency adjustment), registration number (exact match), and legal-form code (Levenshtein at distance 1 and 3 for companies).

Estimate parameters

Unsupervised EM training estimates m and u probabilities per comparison level from the data itself. The two-random-records baseline is anchored on deterministic rules (LEI match, registration-number match, jurisdiction-plus-uppercase-name match) at recall = 0.8, and u is estimated from one million random pair samples.

Persist candidate pairs

Predictions with match_probability >= 0.95 are upserted to entity_resolution_pairs in batches of 500, keyed on (model_version, id_l, id_r). Below 0.95 is discarded — the table is a high-confidence worklist, not a comprehensive scoreboard.

Adjudicate

Each persisted pair starts at adjudication_status = 'pending'. An analyst confirms (match), rejects (not_match), or defers (unsure) through the adjudication API. Only pairs marked match are safe for downstream consumers — investigation case-file merge, dedup on Aleph ingest, UBO chain stitching — to treat as the same entity.

The companies linker

The first linker covers the companies table and runs in Splink’s dedupe_only mode — same source on both sides of the pair. Address deduplication is intentionally out of scope; address normalization is handled separately by libpostal upstream.

Setting	Value
Link type	`dedupe_only`
Unique ID column	`id`
Persistence threshold	`match_probability >= 0.95`
Batch size	500 rows per upsert
Default model version	`splink_companies_v0.1`

Cross-table link_only jobs (gleif × opensanctions × equasis) are a follow-up — the scaffolding supports them, but no production linker for those targets has shipped yet.

Configuring a run

The linker is exposed from packages/python-services/splink and is invoked from a Python worker, not an Edge Function. Pass the path to a Parquet snapshot of the input table:

from supabase import create_client
from companies_linker import (
    train_companies_linker,
    predict_and_persist_companies,
)

linker = train_companies_linker("/path/to/companies.parquet")

supabase = create_client(SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY)
count = predict_and_persist_companies(linker, supabase)
print(f"persisted {count} candidate pairs")

To override the model version or threshold for an experiment, pass them through:

count = predict_and_persist_companies(
    linker,
    supabase,
    model_version="splink_companies_v0.2",
    threshold=0.97,
)

The default splink_<table>_v<major>.<minor> versioning convention is enforced by the upsert key (model_version, id_l, id_r) — a new version produces a fresh set of candidate pairs alongside the old ones rather than overwriting them, so analysts can compare two versions against the same input. When to bump versions:

Minor — thresholds, blocking rules, or comparison settings change but the entity target is the same.
Major — the input table contract, link type, or downstream semantics change.

The `entity_resolution_pairs` table

Every persisted candidate pair lands in public.entity_resolution_pairs with the standard APRS envelope (record_id, schema_version, normalization_version, acl_tier, source_system, ingested_at, modified_at, chunk_id) plus resolution-specific columns:

Column	Meaning
`model_version`	`splink_<table>_v<major>.<minor>` — identifies which trained model produced the pair
`source_table_l`, `source_table_r`	Source tables for the left and right side of the pair. Same value for `dedupe_only` runs; different for cross-table `link_only` runs.
`id_l`, `id_r`	Local IDs from each source table
`match_probability`	Splink-derived probability in `[0, 1]`; only pairs at or above the persistence threshold are stored
`match_weight`	Log-odds match weight from Splink
`blocking_rule`	The blocking rule that surfaced the pair
`adjudication_status`	`pending` / `match` / `not_match` / `unsure`
`adjudicated_by`	UUID of the analyst who decided; `NULL` until decided
`adjudicated_at`	Timestamp of the decision; `NULL` until decided

The table enforces a unique constraint on (model_version, id_l, id_r) so re-running the same model against the same input is idempotent — re-runs update existing rows in place rather than inserting duplicates. Indexes on adjudication_status, match_probability DESC, (source_table_l, id_l), and (source_table_r, id_r) back the common worklist and reverse-lookup queries. Row-level security is configured for service-role writes and authenticated reads — only the Splink workers (and other service-role callers) can persist or update candidate pairs, but any authenticated analyst can read them.

Adjudication API

The adjudication API exposes two endpoints under the entity-resolution tag for analyst tooling — an inbox endpoint and a decision endpoint.

`GET` `/pending`

Returns pending pairs ordered by descending match probability, so the strongest candidates are reviewed first. Parameters

Name	Type	Required	Description
`limit`	number		Number of pairs to return. Default 50, min 1, max 500.

Response

{
  "pairs": [
    {
      "id": 1842,
      "model_version": "splink_companies_v0.1",
      "source_table_l": "companies",
      "source_table_r": "companies",
      "id_l": "co_018af1c2",
      "id_r": "co_0aa39e44",
      "match_probability": 0.987,
      "match_weight": 6.12,
      "blocking_rule": "l.lei_code = r.lei_code and l.lei_code is not null",
      "adjudication_status": "pending",
      "ingested_at": "2026-05-19T09:00:00Z"
    }
  ]
}

`POST` `/decide`

Records an analyst’s decision on a pair. Body

Field	Type	Required	Description
`pair_id`	integer	✓	Primary key of the `entity_resolution_pairs` row
`decision`	string	✓	One of `match`, `not_match`, `unsure`
`adjudicated_by`	UUID		Analyst identifier; optional but recommended for audit

curl -X POST "$OVERWATCH_API_URL/entity-resolution/decide" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "pair_id": 1842,
    "decision": "match",
    "adjudicated_by": "00000000-0000-0000-0000-000000000001"
  }'

The endpoint updates adjudication_status, stamps adjudicated_at and modified_at to the current UTC time, and optionally records adjudicated_by. A 404 is returned when no row matches pair_id.

Gold-set evaluation

Before promoting a model version — that is, before any downstream consumer is allowed to treat its match decisions as authoritative — every model goes through a 200-row gold-set evaluation:

Sample 100 pairs from the high-confidence band (match_probability >= 0.95), 50 from the review band (0.80 <= match_probability < 0.95), and 50 hard negatives below 0.80.
Have an analyst label each pair as match, not_match, or unsure through the adjudication API.
Compute precision, recall proxy, false-positive themes, and threshold sensitivity against the 200 labels.
Record the gold-set date, model version, input snapshot URI, and metrics before enabling downstream consumers.

A model that hasn’t passed gold-set evaluation can still produce candidate pairs and feed the adjudication queue, but its decisions are not yet wired into investigation case-file merge or sanctions screening.

What the linker doesn’t do

No live-table linkage. All inputs are read from Parquet snapshots. Training against operational Supabase tables is explicitly disallowed.
No address deduplication. Addresses are normalized upstream by libpostal; the companies linker doesn’t include any address comparison.
No automatic merge. Persisted pairs are candidates, not decisions. Nothing downstream treats two records as the same entity until an analyst sets adjudication_status = 'match'.
No cross-table link jobs yet. The scaffolding supports link_only mode, but no production cross-table linker (gleif × opensanctions × equasis) has shipped.
No supervised training. The linker relies on unsupervised EM and deterministic anchor rules; no labelled training set is required to produce candidate pairs.

Investigations API — case-file consolidation that consumes confirmed matches
Sanctions API — sanctions screening across name and jurisdiction variants
UBO API — beneficial-ownership chains that benefit from cross-source deduplication
Cross-dataset join keys — deterministic keys to prefer over probabilistic resolution when available

Get started

Configuration

Methodology

When to use it

How resolution works

The companies linker

Configuring a run

The `entity_resolution_pairs` table

Adjudication API

`GET` `/pending`

`POST` `/decide`

Gold-set evaluation

What the linker doesn’t do

​When to use it

​How resolution works

​The companies linker

​Configuring a run

​The entity_resolution_pairs table

​Adjudication API

​GET /pending

​POST /decide

​Gold-set evaluation

​What the linker doesn’t do

​Related references

When to use it

How resolution works

The companies linker

Configuring a run

The `entity_resolution_pairs` table

Adjudication API

`GET` `/pending`

`POST` `/decide`

Gold-set evaluation

What the linker doesn’t do

Related references