Overwatch continuously ingests AIS vessel positions at full resolution — often multiple reports per vessel per second. To keep the live database performant without losing historical data, Overwatch applies a three-stage lifecycle: hot retention in the live database, tiered downsampling to reduce density over time, and long-term archival to cold storage.
How it works
A nightly maintenance job runs in two phases:
- Archive — data older than the retention window is compressed and written to cold storage, then removed from the live database.
- Downsample — data that remains in the live database is thinned to progressively lower resolution as it ages.
Full-resolution data is always preserved in the archive. If you need to query historical data at original resolution, it can be rehydrated on request.
Retention windows
Each data type has a retention window that controls how long it stays in the live database before being archived.
| Data type | Retention window | Notes |
|---|
| AIS positions | 30 days | Largest table by volume. Downsampled before archival. |
| Port events | 60 days | Intelligence is also captured in vessel visits. |
| Dark events | 90 days | AIS gap and dark activity detections. |
| STS events | 90 days | Ship-to-ship transfer encounters. |
| Loitering events | 90 days | Prolonged stationary vessel detections. |
| Ingestion logs | 14 days | Internal pipeline telemetry. |
Archived data is retained indefinitely at full resolution in cold storage. Only the live database copy is removed after the retention window.
Downsampling tiers
AIS positions that remain in the live database are downsampled on a tiered schedule. Each tier keeps one position report per vessel per time bucket, selecting the earliest report in each bucket.
| Tier | Age range | Resolution | What it means |
|---|
| Raw | 0–7 days | Full resolution | Every report is kept as received (often multiple per second). |
| Tier 0 | 7–30 days | 1 per minute | One position per vessel per minute. Sufficient for route replay and behavioral analysis. |
| Tier 1 | 30–90 days | 1 per 5 minutes | One position per vessel per 5-minute window. Useful for historical track review. |
| Tier 2 | Over 90 days | 1 per hour | One position per vessel per hour. Suitable for long-term pattern analysis. |
Tier 1 and Tier 2 apply to data that remains in the live database after archival. In practice, most data older than 30 days has already been archived, so these tiers primarily affect edge cases where archival hasn’t yet completed for a given batch.
Resolution impact
At full resolution, a single vessel broadcasting every 2 seconds produces approximately 43,000 position reports per day. After downsampling:
| Tier | Reports per vessel per day |
|---|
| Raw | ~43,000 |
| Tier 0 (1 min) | ~1,440 |
| Tier 1 (5 min) | ~288 |
| Tier 2 (1 hr) | ~24 |
What this means for your queries
- Real-time tracking and alerting (0–7 days) — full resolution, no data loss.
- Recent investigations (7–30 days) — 1-minute resolution is sufficient for route reconstruction, speed profiling, and anomaly detection.
- Historical analysis (30–90 days) — 5-minute resolution shows vessel tracks and port visits clearly but may miss brief maneuvers.
- Long-term patterns (90+ days) — hourly resolution is suitable for trade route analysis and seasonal patterns. For finer-grained queries on older data, request a rehydration from the archive.
Archive storage
Archived data is stored in compressed format and organized by table and date range. The archive retains full-resolution data indefinitely, so no information is permanently lost when records leave the live database.
If you need to query archived data at its original resolution — for example, to reconstruct a vessel’s exact track from several months ago — contact support to request a rehydration.
Adjusting retention
Retention windows are configured at the platform level and are not user-adjustable. If your use case requires longer hot retention (for example, keeping 90 days of full-resolution AIS positions for ongoing investigations), contact support to discuss options. 
