The medallion pipeline that built this
This dataset came out of a Medallion Architecture exercise on Databricks, deployed as a Databricks Asset Bundle with CI/CD — the point was to learn and document that machinery on a real question, and the question turned out to have a real answer.
- Bronze. Raw CSVs pulled from the Enverus and
OCC web APIs (plus an S&P export), filtered to
county = 'KINGFISHER'with no other cleaning, each loaded into its own Delta table. - Silver. Identity columns and surface lat/lon standardized per source. S&P reports NAD27, so its coordinates were transformed to WGS84 to match the other two. Points were deliberately not cleaned — the disagreements are the subject, not noise.
- Gold. The three silver tables unioned and
pivoted wide, one row per well, with pairwise distances via
ST_DISTANCESPHEROID. Wells missing any vendor were dropped, leaving 8,339 “triplets.”
The gold table was exported to parquet, and that file — committed to the repo — is what this page serves. A sanity check recomputes every pairwise distance from the snapshot’s own points with DuckDB’s spheroid math: worst disagreement with the stored Databricks values, about 5 microns across ~25,000 pairs.
What the disagreements look like
At low thresholds you see the expected: many S&P points shifted a consistent few dozen meters north-east of their siblings, the signature of a NAD27→WGS84 datum history. Above a few hundred meters the excuses run out: wrong longitude quadrant signs, wells filed in the wrong county, and placements that disagree by kilometers. This is not a data cleaning exercise — it is an argument for never trusting a single vendor’s coordinates without a second opinion.
The Databricks Asset Bundle experiment
The original app was Streamlit, deployed to Databricks as an Asset Bundle with GitHub Actions for tests and deploys. The repo README preserves the full step-by-step — bundle config, service principals, cluster grants, CI secrets — as it worked in August 2025. Highlights from the trenches:
-
Spatial SQL on Databricks was the hard part: Sedona wanted
init scripts, Mosaic wanted old Spark, and the answer was the
then-beta DBR 17.1 with native spatial — which in turn
broke
databricks-connectlocal development. - SQL Warehouses didn’t support spatial functions, so the app had to sit on an interactive cluster. $ouch$.
- Performance on the cluster was surprisingly poor for 30k rows of simple points; DuckDB against local parquet was instant. That observation is, in a sense, why this page is now a static site.
The Databricks workspace is gone; the pipeline, the app, and the walkthrough remain documented in the repo. This page is the museum: the July 2025 snapshot, served flat, with the map rebuilt in MapLibre.