Planetek · AWS Boardwalk · Plank 5

Colorado Business Data Lake

Every business ever registered with the Colorado Secretary of State, about 3.1 million back to 1864, sitting in an S3 data lake you can query with live SQL. Raw records in one zone, curated partitioned Parquet in another, a Glue catalog describing both, and Athena billing by the byte it scans. No database servers anywhere; the lake costs ~$0 while you're not looking at it.

registrations in the lake
scanned for a full-table GROUP BY
decade partitions
snapshot taken

Sounding 01 · The pipeline

From raw to ready

The classic lake shape, entirely serverless. An ingest job pages the source API into the raw zone; one Athena CREATE TABLE AS SELECT statement then cleans, types, and rewrites it as partitioned Snappy Parquet, registering the result in the Glue catalog. The dashboard below reads aggregates the ETL precomputed, so rendering this page costs zero Athena.

data.colorado.govSecretary of State · CC03.1M rows
Raw zoneS3 · JSONL, gzipped
Glue catalog + Athena CTASone SQL statement
Curated zoneS3 · Parquet + Snappy
This dashboardstatic site + cached SQL$0 to render

Sounding 02 · The dashboard

The state of Colorado business

Rendered from aggregates the ETL precomputed into the analytics zone.

New registrations per year since 1990
What gets formed: entity types, all time
Where they stand today: status
Cities with the most businesses in Good Standing
Still standing: share of each year's cohort in Good Standing today

Sounding 03 · The depth chart

Where the water is deep

Businesses in Good Standing by ZIP, each plotted at its Census centroid. Bubble area is the count; where bubbles crowd and overlap the chart reads darker, the way soundings crowd a harbor. Drawn from the same precomputed analytics zone as the dashboard, so it costs zero Athena to render.

Sounding 04 · Drop a line

Find your boat in the lake

Type the first letters of any business name: yours, your employer's, one you're curious about. The API binds your text to a ? placeholder as an Athena execution parameter, so it never enters the SQL string. A name can surface from any decade, which makes this the anti-pruning exhibit: every partition gets read, but Parquet hands over only the few columns named in the SELECT, and the sweep of all 3.1 million rows still costs a fraction of a cent.

Sounding 05 · Live SQL

Run live SQL on the lake

These queries execute in Athena when you click, against the same 3.1M rows, inside a workgroup that enforces a per-query scan cutoff. The stats underneath are Athena's own numbers: what it scanned, how long the engine ran, and what the query cost at $5 per terabyte scanned. Results cache for a few hours, and live executions are capped per day ().

Sounding 06 · Raw vs curated

The raw-vs-curated race

The whole argument for a curated zone in one click: the identical GROUP BY status aggregation runs twice, once over the raw gzipped JSON and once over the partitioned Parquet. Columnar beats row-text on every number Athena reports.

Sounding 07 · The time machine

The lake keeps its old charts

The fourth zone is an Apache Iceberg copy of the curated table: the format that gives plain S3 files ACID transactions, snapshots, and schema evolution, with no warehouse involved. Each ETL run rebuilds it and then repairs the record with one UPDATE: the most common misspellings of three city names (a few thousand rows, from an enumerated list, not a fuzzy match) get filed under their proper spelling. That leaves two snapshots, and both stay queryable.

Sounding 08 · The rigging

How this is built

S3 raw / curated / analytics zones Glue Data Catalog Athena SQL + CTAS ETL Athena execution parameters query runtime statistics enforced workgroup + scan cutoff Parquet, partitioned by decade Apache Iceberg: ACID + time travel Lambda (Node 22, arm64) DynamoDB result cache + daily caps CloudFront + same-origin API Terraform