Henry Marinho - Projects

From a 100-Call API to a Public Dashboard: A Production Weather ELT Pipeline for Bucaramanga

Wed, 11 Nov 2026 15:38:11 GMT

I wanted a small project that behaves like a real one: data arrives on a schedule, gets transformed and tested, lands in a dashboard people can use, and the whole thing runs on my own server without me babysitting it. The subject is the weather in Bucaramanga, Colombia, the city where I live. The constraint that shaped every decision: a free API that allows 100 calls per month.

This article walks through what I built, the decisions behind it, the bugs I hit, and the data engineering skills each part demonstrates. The live dashboard is embedded at the end.

The stack at a glance

Layer	Tool	What it does
Source	Weatherstack API (free plan)	Current conditions, air quality and astronomy for one city
Ingestion	Python + Apache Airflow 3	Calls the API twice a day and stores the raw response
Storage	PostgreSQL	Raw JSONB payloads and the dbt models, with least-privilege roles
Transformation	dbt (run by Airflow in its own container)	Typed staging model, four marts, data tests and unit tests
BI	Apache Superset 6.1	Dashboard defined as code, branded, public and embeddable
AI access	MCP servers (Superset, Airflow, dbt, Postgres)	Let an AI assistant query and operate the stack through a private gateway
Platform	Docker Compose on a VPS, deployed with Coolify	Merge to GitHub, the server redeploys
Quality	GitHub Actions, Trivy, Dependabot	Lint, tests, dbt parse, security scan and a full end-to-end run on every push

1. Start with the constraint: audit the API before writing code

The first version of the pipeline only stored six fields for New York. Before expanding it, I tested what the free plan really allows with a handful of real requests instead of trusting the pricing page:

The /current endpoint works over HTTPS and returns about 30 fields, including air quality (PM2.5, PM10, ozone, US EPA index) and astronomy (sunrise, sunset, moon phase). We were throwing most of them away.
Bulk queries (several cities in one call) return error 604. Forecasts return 609. Two calls back to back return 106 rate_limit_reached.

That audit turned "we need a paid plan" into "we are wasting 80% of what the free plan already gives us". It also set the budget: one city, twice a day, 7 a.m. and 7 p.m. local time, about 62 calls a month, leaving room for retries.

2. Ingestion: store everything, decide later

Instead of adding 25 new columns to the raw table, the ingestion job now stores the complete API response in a JSONB column, plus a source flag (api or mock). The migration is additive and idempotent, so the existing production table upgrades itself on the next run and the old rows stay valid:

ALTER TABLE dev.raw_weather_data ADD COLUMN IF NOT EXISTS payload JSONB;
ALTER TABLE dev.raw_weather_data ADD COLUMN IF NOT EXISTS source TEXT;

This is schema-on-read: if tomorrow I want visibility or wind direction in a chart, it is a change in dbt, not in the ingestion code, and the history is already there.

Two guards protect the quota:

A monthly budget check. Before each real call, the job counts this month's successful API rows. At 90, it logs a warning and skips the call instead of failing. Failed calls insert nothing, which is why the limit sits below 100.
A pause between calls when more than one city is configured, because the free plan rejects back-to-back requests.

A mock mode returns a real Bucaramanga response with the current timestamp. CI, local development and the first deploy all run the full pipeline without spending a single API call.

3. Transformation with dbt: typed, tested, documented

The staging model turns the JSON into typed columns. Weatherstack sends some numbers as strings ("7.2") and some times as text ("05:41 AM", or "No moonrise" on some days). Two small macros parse them defensively and return NULL instead of breaking the run:

{{ safe_numeric("payload #>> '{current,air_quality,pm2_5}'") }} as aq_pm2_5,
{{ safe_time("payload #>> '{current,astro,moonrise}'") }}      as moonrise,

Rows from before the change have no payload, so the model falls back to the legacy columns and the New York history keeps working. On top of staging there are four marts: observations, daily averages (temperature, humidity, rain, UV, PM2.5), daily air quality and daily astronomy (day length, moon phase).

Testing is where dbt pays off:

Unit tests feed a real JSON payload and a legacy row into the staging model and check the exact output: strings converted to numbers, "yes" to a boolean, "No moonrise" to NULL.
Data tests check ranges (humidity and cloud cover between 0 and 100, EPA index between 1 and 6, UV not negative) and that each city has one row per day in the daily marts.
Source freshness runs after every build and fails if the newest observation is too old.

4. Orchestration and CI: prove it works before it ships

Airflow runs three tasks: ingest, dbt build, then the freshness check. A failed task sends a webhook alert. On every push, GitHub Actions runs:

Ruff and 57 pytest tests (ingestion, budget guard, migration, dashboard definitions).
dbt parse.
A Trivy scan for vulnerable dependencies, committed secrets and container misconfigurations.
An integration job that starts Postgres and Airflow from a fresh checkout, runs the whole DAG in mock mode, checks that every mart has data, then takes a backup and restores it to prove the backup is usable.

CI caught two real bugs on the way. A pair of double quotes inside a YAML description broke dbt parse. A unit test compared 7.13 with the numeric 7.130, which dbt treats as different. When I could not run Docker locally, I reproduced the CI run with an embedded PostgreSQL and dbt in an isolated virtual environment: 43 of 43 dbt nodes passing before pushing again.

5. The dashboard as code, without taking it away from the UI

Superset is configured by a script that registers the database, datasets, charts and dashboard through its REST API. It runs automatically after every deploy as a one-shot container. Three design choices made it practical:

Idempotent and rename-aware. When I translated every title to Spanish, existing charts were renamed in place instead of duplicated.
The UI wins after creation. The script creates what is missing but does not overwrite charts, layout or colors edited by hand. A reset flag restores the repository version when that is what I want.
Labels for humans. Columns get readable Spanish names with units, such as "Lluvia total (mm)" or "PM2.5 promedio (µg/m³)".

6. Data storytelling: the Atlético Bucaramanga theme

A weather dashboard for Bucaramanga should feel local, so the theme borrows the identity of the city's football club, Atlético Bucaramanga. Superset 6 themes are Ant Design tokens, so the brand became a design system rather than a pile of CSS:

Color roles, not just colors. The crest yellow #FFED00 is unreadable as text or as a chart line on a light background, so it became the brand accent (header band, card edges). The away-shirt green #00522C is the primary color for actions and titles. Ink #1A1A2C is for text.
A chart palette that stays legible, with the yellow darkened to gold, plus the crest green, olive and gray.
Sports typography: Bebas Neue for the headline and big numbers, Oswald for chart titles and Montserrat for everything else.
The crest is served from our Directus CMS, and the Content Security Policy allows exactly that host.

7. Public, embeddable and still locked down

To publish the dashboard in this article, I avoided the common shortcut of giving anonymous users the broad "Gamma" role:

Anonymous visitors get Superset's built-in minimal Public role.
Dashboard-level access (DASHBOARD_RBAC) means only dashboards tagged with that role, and the data behind their charts, are visible without logging in. SQL Lab, other dashboards and raw datasets stay private.
A CSP frame-ancestors rule lets only this website embed it.
On the website side, the article HTML is sanitized, and iframes are allowed only from an explicit list of hosts and paths.

8. Running it like production

Least privilege: separate Postgres roles for the pipeline, Airflow's and Superset's own metadata, and a read-only reader that both the dashboards and the AI tools use.
Private by default: only the dashboard has a public domain. Airflow, Postgres and the MCP servers are reachable only through an SSH tunnel. The server accepts SSH keys only, and unused ports are closed.
Recoverable: daily database backups kept for 7 days, plus a restore test in CI.
Maintained: Dependabot opens patch-only image updates, and resource limits keep one service from starving the others.
Debugging from evidence. When the dashboard showed "ChunkLoadError", the server logs proved the browser never requested the failing file. It was a stale cache after a redeploy, not a broken release.

9. Working with AI agents, with a human in charge

I built this with Claude Code as a pair. I used parallel agents for ingestion, dbt and Superset, each working from an explicit contract of column names. MCP servers let the assistant query Postgres, trigger Airflow and edit Superset charts directly. AI sped up the typing. The engineering work stayed the same: define the constraints, write the contracts, review every diff, insist on tests, and refuse to ship until CI was green. When an agent's work broke the build, the tests caught it, not the dashboard's users.

Skills this project demonstrates

Skill	Evidence in this project
API integration under constraints	Audited the free plan with real calls; budget guard, rate-limit pause, mock mode
Data modeling	Raw JSONB plus typed staging and four marts; schema-on-read; backward-compatible migration
dbt	Macros, unit tests, generic and singular data tests, source freshness, documentation
Orchestration	Airflow 3 DAG with retries, timeouts, failure alerts and dbt in an isolated container
Testing and CI/CD	57 Python tests, an end-to-end DAG run and a backup restore test on every push
BI as code and data storytelling	Idempotent Superset setup, design tokens, accessible palette, localized labels
Security	Least-privilege roles, CSP, dashboard-level RBAC, HTML sanitization, Trivy, private services
DevOps	Docker Compose, Coolify deploys from GitHub, VPS hardening, backups
AI-assisted engineering	MCP servers for the data stack, parallel agents, human review and test gates

The live dashboard

This is the real dashboard, updated at 7 a.m. and 7 p.m. Bogotá time. Use the filter to explore, or open it on its own page.

Live Superset dashboard: temperature, humidity, rain, UV, air quality, and sun and moon for Bucaramanga.

The code is open on GitHub, including the dbt models, the Airflow DAG, the Superset setup script and the CI pipeline: henrymarinho90/weather-data-project.

Ecommerce Returns & Profitability Analytics

Sat, 03 Oct 2026 18:56:27 GMT

Ecommerce Returns & Profitability Analytics

I built an end-to-end ecommerce analytics project to understand how revenue, returns, product performance, traffic sources, and fulfillment affect profitability.

The project goes beyond creating a dashboard. I built the data pipeline from PostgreSQL and SQL analysis through Python exploration, reusable analytics marts, Metabase visualization, and final SQL validation.

Interactive Dashboard

http://bi.romahomestore.com/public/dashboard/91e02296-ee2e-41c3-b389-84115d1771c3

Use the filters to explore performance by date, category, brand, and traffic source.

The dashboard is powered by analytics marts built in PostgreSQL and validated directly against SQL before publication.

What I Built

Loaded and modeled ecommerce data in PostgreSQL.
Created relational tables, keys, joins, and an enriched analytical layer.
Used Python, pandas, and Jupyter for exploratory and statistical analysis.
Built reusable analytics marts for business intelligence.
Created an interactive Metabase dashboard with global filters.
Validated dashboard calculations directly against PostgreSQL queries.

Analytics Architecture

Raw Ecommerce Data
        ↓
PostgreSQL
        ↓
Relational / Enriched Data Model
        ↓
SQL + Python Analysis
        ↓
Analytics Marts
        ↓
Metabase Dashboard
        ↓
SQL QA & Reconciliation

Key Results

$5.74M analyzed revenue
66,019 orders
10.16% item return rate
51.91% margin rate proxy

Revenue increased considerably over the analyzed period while the overall return rate remained relatively stable around 10%.

Search generated approximately $4.05M, or about 71% of analyzed revenue. Its return rate, however, was similar to the other major traffic sources, suggesting that its larger return exposure was primarily a result of its much larger sales volume.

I also tested whether longer delivery times were associated with more returns. The item-level correlation between ship-to-deliver time and return behavior was approximately 0.0027, indicating essentially no meaningful linear relationship in this dataset.

A Data Modeling Problem I Had to Solve

One of the most useful challenges came from the Top Products dashboard.

My original product mart was aggregated too early. It worked for a static chart, but once I added dashboard filters for date, category, brand, and traffic source, the BI tool no longer had the dimensions required to correctly recalculate the product rankings.

I redesigned the mart to preserve those dimensions and moved the Top 25 ranking to the dashboard layer so products are ranked after filters are applied.

I also stored additive components such as revenue, margin, returned items, and order items instead of relying on pre-calculated percentages. This allows metrics such as return rate and margin rate to be recalculated correctly for any filtered population.

Quality Assurance

I did not consider the dashboard complete just because the charts looked correct. I reconciled Metabase against PostgreSQL for:

Revenue, orders, return rate, and margin rate
Date, category, brand, and traffic-source filters
Category × brand analysis
Margin burn calculations
Traffic-source metrics
Fulfillment-time bins
Top-product rankings and profitability

The final dashboard values reconciled with the PostgreSQL source of truth.

Skills Demonstrated

SQL · PostgreSQL · Python · pandas · Jupyter · Data Modeling · Data Marts · ETL/ELT · Metabase · Business Intelligence · Data Visualization · Statistical Analysis · Data Validation · Git · GitHub · Data Storytelling

What I Learned

Analytical grain directly affects what a BI dashboard can do.
Ratios should often be recalculated from additive components after filtering.
Updating SQL in Git does not automatically update the database.
Metric definitions must remain consistent across every layer of the stack.
A dashboard should be validated against its database source before it is trusted.

View the project on GitHub →

Beyond Syntax: How I Used 'Vibe Coding' to Build an Enterprise-Grade Data Cleaning Pipeline in Google Sheets

Wed, 14 Jan 2026 18:54:49 GMT

Beyond Syntax: How I Used 'Vibe Coding' to Build an Enterprise-Grade Data Cleaning Pipeline in Google Sheets

Meta Description: Discover how Vibe Coding transforms data analysis. See how I automated a complex CRM data cleaning pipeline in Google Sheets using AI-assisted Google Apps Script, reducing hours of manual work to seconds.

The Business Challenge: CRM Data Normalization Nightmare

Every data analyst knows the pain. You have three different data sources, and they all speak a different language.

For this project, the objective was clear but the path was messy. We needed to consolidate monthly sales reports from three major automotive CRMs into a single, unified format for Meta Conversion API (CAPI) upload.

The Constraints:

Source 1 (VinSolutions): Inconsistent address columns and date formats.
Source 2 (DriveCentric): Missing "Gross Profit" columns and messy headers.
Source 3 (eLead): The header row wasn't even on the first line (it was on row 5), and names were combined in a single cell (Last, First).
The Goal: A clean CSV with 15 specific columns (e.g., event_time, event_name, value, order_id) formatted strictly (dates as YYYY/MM/DD, Zips trimmed to 5 digits, Phones normalized to +1XXXXXXXXXX).

Manual cleaning would take hours per file. Using standard Google Apps Script Automation, I built a tool that does it in seconds.

What is Vibe Coding in the Context of Data Analytics?

Before diving into the code, we must define the methodology. Vibe Coding is continuous iteration with an LLM to produce code that feels right—it handles edge cases, includes error logging, and creates a user-friendly UI.

Unlike the generic snippets found in articles like 5 JavaScript Data Cleaning Patterns, Vibe Coding allows us to handle context. It allows us to say, "The Zip code column keeps reverting to currency format because of ghost formatting; fix it."

The result is not just a script; it is a software application living inside a spreadsheet.

The Solution: A Universal Bulk Automation Tool

The final product was a menu-driven interface inside Google Sheets entitled "⭐ Drive Manager." It allows the user to:

Scan Google Drive folders for the latest files.
Detect which CRM the file belongs to (Vin, DriveCentric, or eLead).
Apply the specific cleaning logic required for that CRM.
Calculate average gross profit to fill in missing data gaps.
Export the clean data to a tracking sheet.

1. The Traffic Controller (CRM Detection)

The core of the script is a function I call the "Traffic Controller." It reads a "CRM" column in our master list and routes the file to the correct cleaning engine.

// Pseudo-code representation of the routing logic
if (crm.includes("vin")) {
   cleanVinSheet(targetSheet, avgGross);
} 
else if (crm.includes("drive")) {
   cleanDriveCentricSheet(targetSheet, avgGross);
} 
else if (crm.includes("elead")) {
   cleanEleadSheet(targetSheet, avgGross);
}

This ensures that we aren't applying VinSolutions logic (which expects headers on row 1) to eLead files (where headers are on row 5).

2. The "Nuclear Option" for Data Hygiene

One of the most persistent issues in Excel to Google Sheets workflows is "ghost formatting." We found that when converting Total Gross (Currency) columns into Zip Code columns during column reordering, the spreadsheet would force the Zip codes into currency format (e.g., turning 65010 into $65,010.00).

Through Vibe Coding, we developed reorderColumnsNuclear.

Instead of simply moving columns, this function:

Reads the data into memory.
Wipes the entire sheet clean (Content and Formatting).
Sets the entire sheet format to @ (Plain Text) to create a "defense shield" against auto-formatting.
Writes the data back.
Selectively applies Date formatting (YYYY/MM/DD) only to the specific date column.

3. Handling Edge Cases: The Logic Injection

The "Avg Gross" Injection: We encountered files where the Value column was empty or contained negative numbers (which breaks Meta ads optimization). The script automatically fetches a calculated "Average Gross" from a separate "Tracker" tab and injects it into any cell with missing or invalid data.
Phone Number Normalization: A Regex cleaner iterates through Cell, Home, and Work phones, stripping non-integers and formatting them to +1... for perfect matching rates.
Zip Trimming: eLead files often included 9-digit zips (e.g., 12345-6789). The script detects this and trims them to the standard 5 digits required for matching.

The Results

Efficiency: Reduced monthly reporting time from ~4 hours to < 5 minutes.
Accuracy: Eliminated human error in column mapping and date formatting.
Scalability: The script handles bulk operations, processing 50+ files in a single run without user intervention.

Conclusion

Data cleaning doesn't have to be a chore. With the right approach to AI-Assisted Google Sheets Automation, we can build tools that are robust, scalable, and incredibly fast. "Vibe Coding" isn't just a buzzword; it's a methodology that allows data analysts to deliver value exponentially faster.

Let's Automate Your Workflow

Are you looking for a Data Analyst who combines technical depth with modern AI workflows to solve business problems?

Connect with me on LinkedIn

Or email me directly at: henrymarinho90@gmail.com

The Rise of Vibe Coding in Data Analytics: Building a Full-Stack Portfolio with AI and Intent

Wed, 14 Jan 2026 03:42:08 GMT

Expert Analysis

The Future of Decision Intelligence: Mastering Vibe Coding to Architect Elite Data Analytics Portfolios

In the rapidly evolving landscape of 2026, the traditional definition of a "Data Analyst" has undergone a radical transformation. The barrier between data insight and technical execution has all but vanished. Today, the most sought-after professionals are Systems Thinkers who can architect entire data ecosystems.

This shift hasn't occurred because analysts suddenly mastered CSS grids or backend management. Instead, we have entered the era of Vibe Coding.

Recruiter's Perspective

This portfolio itself is a living proof of concept. It was architected using Astro, Directus, and Metabase, orchestrated through AI-driven development to demonstrate maximum technical fluency and efficiency.

What is Vibe Coding? A New Paradigm for Data Professionals

Vibe Coding is a philosophy where the professional acts as a Director rather than a typist. It is the art of using high-level AI orchestration—leveraging tools like Cursor, Claude 3.5, and GPT-5—to build complex infrastructures through pure "vibe" or vision.

For a Data Analyst, this means a transition from syntax-focused labor to intent-focused architecture. We focus on the logic of the data and the user experience, allowing AI to handle the boilerplate execution.

Beyond the Static Resume

A "Vibe Coded" portfolio demonstrates that you can manage modern tools at scale:

Headless CMS: Orchestrating dynamic content via Directus.
Static Site Generation: Leveraging Astro for lightning-fast SEO performance.
Interactive BI: Integrating Metabase for real-time data storytelling.

Why Recruiters Care About Your Technical "Vibe"

Top-tier firms are looking for Technical Fluency. When an analyst presents a portfolio built with a modern "Bento Grid" stack, it signals an understanding of the Full-Stack Data Lifecycle.

As an Industrial Engineer (UIS), my approach is rooted in optimization and high-performance leadership. Building through Vibe Coding allows me to apply these principles directly to software and data architecture, bridging the gap between operations and IT.

Interactive Case Study: Used Cars in USA Analysis

The Used Cars in USA analysis project demonstrates a complete automated pipeline: from Python-based data cleaning to interactive deployment.

live_dashboard_v2.sql

Future-Proofing Your Career with AI

The reality of 2026 is simple: Analysts who use AI will take the jobs of those who don't. Vibe Coding allows me to do the work of a three-person development team, focusing my energy on Requirement Engineering and Actionable Insights.

By mastering the "Vibe," we ensure we are not just spreadsheet jockeys, but architects of the modern data stack.

Ready to Elevate Your Data Strategy?

I am currently open to new challenges in Data Analytics, Project Management, and AI-Driven Development.

LinkedIn Profile ↗ Email Me Directly

Used cars in USA Analytis

Tue, 13 Jan 2026 15:30:01 GMT

used-cars-analysis