Quick answer:
A data pipeline is any automated path that moves data from where it’s created to where it’s used, usually with some processing along the way. Source, steps, destination, on a schedule or a trigger. “Pipeline” is the umbrella word; ETL jobs, streaming feeds, dbt runs, and reverse ETL syncs are all pipelines of different shapes.
When data people say “the pipeline broke”, they mean some step in that path failed and now a dashboard, a model, or a customer-facing number is stale or wrong.
What is a data pipeline made of?
Four parts show up in nearly every one:
- A source: a production database, an API, an event stream, a folder someone drops files into (still more common than anyone admits).
- Steps: extract, validate, transform, join, aggregate. Each step consumes the previous step’s output.
- A destination: a warehouse, a lake, a SaaS tool, a model.
- A trigger: a schedule (“every hour”), an event (“when a file lands”), or an upstream completion (“after the load finishes”).
Because steps depend on each other, a pipeline is naturally a graph: extract feeds staging, staging feeds models, models feed dashboards. That’s why orchestrators represent pipelines as DAGs (directed acyclic graphs) and why lineage tools exist to draw the graph back for you when you’ve lost track of it.
What does a typical analytics pipeline look like?
The standard modern arrangement, end to end: Fivetran or Airbyte copies sources into Snowflake or BigQuery; dbt transforms raw tables into clean models; Metabase or Looker reads the models; Hightouch pushes segments back out to marketing tools. Airflow or Dagster conducts the order, and Monte Carlo-style observability watches for silent failures.
That’s 5 or 6 products for one logical flow, which is both the modern stack’s strength (each piece is replaceable) and its running joke (each piece is an invoice).
Batch or streaming?
Batch pipelines run on a clock and process chunks: last hour’s rows, yesterday’s files. Streaming pipelines process events continuously as they arrive, through systems like Kafka. Streaming gives you seconds-fresh data and a much harder engineering problem (ordering, exactly-once processing, late-arriving events).
The grown-up answer for most analytics teams is batch, run as often as the business genuinely needs. Streaming earns its complexity when a machine acts on the data in real time: fraud scoring, personalization, alerts. A dashboard a human checks twice a day does not need a Kafka cluster behind it.
Why do pipelines break so much?
Because both ends move. Sources change (an API adds a field, someone renames a column, a vendor has an outage) and destinations change (a model gets refactored, a warehouse permission shifts). Every pipeline is a contract between 2 systems that never signed anything.
The failure that hurts most is the silent one: nothing errors, the job goes green, and the data is quietly wrong or 3 days stale. That’s the exact gap the data observability category exists to close, and why “did the pipeline run?” and “is the data right?” are different questions.
What are the benefits and drawbacks of pipeline-heavy stacks?
Benefits of well-built pipelines
Humans stop being the transport layer
Before pipelines, “data integration” was someone exporting CSVs on Friday afternoons. Automation removes both the labor and the Friday-afternoon error rate.
Freshness becomes a dial
Hourly, every 15 minutes, streaming: you choose per use case and pay accordingly. Decisions get made on today’s data instead of last sprint’s.
Each stage is inspectable
A staged pipeline leaves evidence: raw landed here, transform produced that, the sync pushed this. When numbers look wrong, you can bisect instead of guess.
Composability keeps vendors honest
Because stages connect through the warehouse, any single tool can be swapped without rebuilding the rest. That’s the modern stack’s actual moat against lock-in.
Drawbacks of pipeline-heavy stacks
Every hop is a failure point
Five tools in a chain means 5 places to break and 4 handoffs to misalign. Reliability engineering arrives whether you scheduled it or not.
Latency accumulates silently
Each staged step adds its schedule to the total. Teams discover their “hourly” dashboard is actually 3 hours behind end to end, because nobody summed the chain.
Cost hides in the seams
Ingestion rows, warehouse compute, orchestrator runs, sync fees: 4 meters for one flow. Pipelines rarely look expensive until you total them per use case.
Sprawl outruns understanding
Pipelines are easy to create and scary to delete, so estates accumulate hundreds nobody fully maps. That’s the gap lineage and catalogs get bought to fill.
What separates a good pipeline from a fragile one?
A few properties, none glamorous: it’s idempotent (running it twice doesn’t duplicate data), it’s replayable (you can rebuild history from raw), it alerts on failure to a place humans look, and its dependencies are declared rather than implied by timing. “Job B starts 30 minutes after job A because A usually finishes in 25” is how 2 a.m. incidents are manufactured.
You don’t need exotic tools for any of that. You need the boring disciplines applied consistently, which is harder than it sounds and cheaper than the alternative.