I turn messy financial
data into structured,
analysis-ready assets.
Regulatory filings, market data, and reference data — parsed, cleaned, validated, and shipped as pipelines other people can trust. This page is the deep-dive; the main site has the broader picture.
Most of my work is general backend and infrastructure engineering. This page is the
deep-dive on the slice of it that's specifically financial and regulatory
data — SEC filings, structured extraction from messy real-world documents,
and the ETL discipline needed to turn either into something a research or analytics
team can actually build on.
I don't have a hedge fund or bank on my résumé. What I have instead is a
7-stage pipeline that turned 4,808 SEC 10-K filings into a
regression-ready dataset with a measurable out-of-sample lift, and a second project
where a govtech founder handed me a real, ugly civil-engineering PDF and I had three
days to make it queryable. Both are below in full, not resume-bullet form.
A 7-stage ETL pipeline that pulls annual 10-K filings straight from the SEC's EDGAR system, cleans and parses the embedded iXBRL, isolates risk-factor disclosures, and turns them into a firm-year panel dataset ready for regression analysis — with embeddings alongside the raw structured text for similarity search and downstream modeling.
- Time-based train/test splits (2017–2021 train / 2022–2024 test) — no look-ahead leakage across the panel.
- Firm and year fixed effects with clustered standard errors, so results aren't just picking up firm-level or macro noise.
- Inverse propensity weighting (IPW) and subsample stability checks to pressure-test whether the lift holds up outside the full sample.
- Taxonomy-based similarity search alongside raw embeddings — two independent ways to query unstructured risk-factor text at scale.
The final panel is persisted as partitioned Parquet, keyed by fiscal year — columnar storage so a downstream consumer pulling a single year's cross-section isn't paying the cost of a full-dataset scan. Intermediate stages use DataFusion for the heavier tabular transforms between the cleaning and merge steps.
Neal Agarwal, co-founder of CityVerse AI (a Nebraska-based govtech startup), handed me
a real civil engineering plan set — Plans.pdf,
dense with tables, structural layouts, and domain-specific notation — and one task:
build a system that could answer arbitrary natural-language questions
about it, correctly, including questions about structure and layout, not just text.
I had about three days, while working full-time.
A vision-LLM extraction pipeline (Claude Sonnet + OCR) that converts the unstructured PDF into structured relational data in SQLite — and an LLM-based query router that translates natural-language questions into structured database operations, rather than leaning on naive RAG or keyword matching, which falls apart fast on documents this dense. On top of that: a full-stack Flask app with a Leaflet.js interactive map (toggleable infrastructure layers — manholes, inlets, pipes, flared ends) wired to a real-time chat interface, with structure IDs mentioned in chat answers auto-highlighting on the map.
- Domain ambiguity discovered mid-build: invert elevations needed to be modeled as pipe-level attributes, not structure-level — a distinction the raw PDF didn't make explicit, found only through validation against the real data.
- A sheet-numbering data corruption bug during extraction forced a pipeline correction and a full database rebuild.
- Rather than silently dropping ambiguous or orphaned records, I designed a confidence-flagging approach — bad data gets surfaced, not hidden.
- Lost AWS access mid-project and migrated the deployment strategy to Railway without losing momentum.
The core problem — structured and unstructured data mixed together, where "plausible" isn't good enough — is the same problem financial regulatory data poses. A civil engineering plan set and a 10-K filing look nothing alike, but the extraction discipline transfers directly.
ATLCost unifies Atlanta neighborhood cost-of-living data — rent, groceries, gas, commute, utilities — which normally live in incompatible sources. I architected hybrid batch/streaming ingestion: Airflow DAGs for slower-moving sources (Zillow, BLS, EIA, Census) and Kafka consumers for real-time feeds (GasBuddy, MARTA transit), landing everything into a unified PostgreSQL store that processes 50K+ records daily, with schema validation and automated staleness alerts so data-freshness problems surface on their own rather than getting discovered downstream.
The same store also feeds an XGBoost-based forecasting layer (3–6 month cost projections) and a recommendation engine, served through a REST API deployed on AWS behind a production service boundary with structured logging throughout.
- Owned end-to-end Python pipelines on AWS GPU instances processing 160K+ text and image artifacts per job, sustaining 100-hour continuous runs with fault-tolerant ingestion.
- Shipped LLM-powered validation tooling that improved extraction accuracy from 70% to 99% across 160K+ artifacts, unblocking analysis for 20+ researchers.
- Authored runbooks and operational documentation so the team could recover pipelines and reproduce experiments without engineering involvement.
- Automated weekly data collection across 45+ countries, replacing 30–40 hours of manual researcher work with a pipeline that ran in under 10 minutes.
- Delivered standardized, analysis-ready datasets directly to faculty and PhD researchers.
- Automated invoice and sales report ingestion with Python and SQL across 16 store locations, cutting weekly processing time by 92% while maintaining due diligence on $150K+ in weekly invoice volume.
- Maintained account reconciliations verifying supplier invoice totals against recorded figures, catching discrepancies before they affected procurement decisions.
This is the part that doesn't show up in a skills-tag list but shows up in every project above: validation frameworks that catch anomalies instead of propagating them, confidence-flagging for ambiguous data instead of silent drops, staleness detection on live ingestion, and runbooks written so someone else can recover a pipeline without me in the room.
Let's talk
data.
Open to full-time data engineering roles. If you've got a messy dataset — financial or otherwise — I'd like to hear about it.