Skip to main content
Overwatch continuously ingests AIS vessel positions at full resolution — often multiple reports per vessel per second. To keep the live database performant without losing historical data, Overwatch applies a three-stage lifecycle: hot retention in the live database, tiered downsampling to reduce density over time, and long-term archival to cold storage.

​
How it works

A nightly maintenance job runs in two phases:
  1. Archive — data older than the retention window is compressed and written to cold storage, then removed from the live database.
  2. Downsample — data that remains in the live database is thinned to progressively lower resolution as it ages.
Full-resolution data is always preserved in the archive. If you need to query historical data at original resolution, it can be rehydrated on request.

​
Retention windows

Each data type has a retention window that controls how long it stays in the live database before being archived.
Archived data is retained indefinitely at full resolution in cold storage. Only the live database copy is removed after the retention window.

​
AIS position lifecycle

AIS positions move through four stages as they age. The first three stages keep data in the live database at progressively coarser resolution; the fourth moves raw data to cold storage.
Data older than 90 days is preserved at full resolution in the archive — not downsampled. This means retrospective analyses that need fine-grained position history (for example, ship-to-ship transfer reconstruction or motif detection) can be answered at original resolution by rehydrating from the archive.

​
Resolution impact

At full resolution, a single vessel broadcasting every 2 seconds produces approximately 43,000 position reports per day. After in-place downsampling:

​
What this means for your queries

  • Real-time tracking and alerting (0–7 days) — full resolution, no data loss.
  • Recent investigations (7–30 days) — 1-minute resolution is sufficient for route reconstruction, speed profiling, and anomaly detection.
  • Historical analysis (30–90 days) — 5-minute resolution shows vessel tracks and port visits clearly but may miss brief maneuvers.
  • Long-term and retrospective analysis (90+ days) — full-resolution data is available from the archive on request. Live-database queries do not return data older than 90 days; contact support to rehydrate the date range you need.

​
Archive storage

Archived data is stored in Cloudflare R2 as gzip-compressed JSONL, organized by table and date range. The archive retains full-resolution data indefinitely, so no information is permanently lost when records leave the live database. A daily cron runs at 03:00 UTC and archives any rows older than the retention window. AIS position archival is capped at 200,000 rows per run so that a backlog cannot starve the smaller archives (port events, transit metrics, POI snapshots) that share the same cron budget. At normal ingest volume the cap clears roughly two hours of historical raw positions per day, which is more than enough to keep up once the system reaches steady state. If you need to query archived data at its original resolution — for example, to reconstruct a vessel’s exact track from several months ago — contact support to request a rehydration.

​
Adjusting retention

Retention windows are configured at the platform level and are not user-adjustable. If your use case requires longer hot retention (for example, keeping 90 days of full-resolution AIS positions for ongoing investigations), contact support to discuss options.