Moving ingestion off third-party connectors

The problem

A direct-to-consumer brand selling through Shopify, Amazon, TikTok Shop and its own mobile apps had ingestion spread across three generations of tooling: a managed connector service billed per row, a decade of ad-hoc Cloud Functions written by whoever needed the data at the time, and a couple of webhooks nobody owned.

That had three costs. The connector bill grew with volume, not with value. Nobody could say what would break if one piece changed, because the functions shared no pattern and had no documentation. And when a source failed, the first sign was usually a dashboard that looked wrong.

What I decided, and why

• Own the connectors for the sources that matter. Twelve Fivetran datasets moved to in-house Python connectors. The logic is readable and in version control, and the cost is compute, not per-row.
• One pattern for everything new. App Store, Play Store, AppLovin and Amazon replenishment data came in as Gen2 Cloud Functions, which run on Cloud Run, built on the same template as the migrated connectors. There is now one way to add a source.
• One logging and documentation standard across every function and every dbt model that reads from them, so the next person can follow the code and the run history without asking.
• Infrastructure in Terraform, so functions and their configuration are reviewed in pull requests like everything else.

How it was built

Each connector is a small Python function on a shared template: pull from the source API, land the records in BigQuery, and leave typing, deduplication and modelling to dbt downstream. The template carries the logging and documentation standard with it, so a new source gets both for free.

What it changed

• About £1,700 a month in connector and compute costs gone.
• Ingestion the team can read and change: every source on one pattern, with logs and documentation in one place.
• Twenty-plus transformations standardised on the way through.
• Data quality incidents down 54% over the same period, largely because failures now surface in logs and alerts rather than in dashboards.