When a pipeline breaks, you should hear about it before the CFO's dashboard does.
We build ingestion and transformation pipelines on Snowflake, Databricks or Microsoft Fabric, with a test and an alert at every step. Sources come in through Fivetran, Openflow or connectors we write. dbt or Spark transforms them. Airflow runs the schedule, unless the platform's own scheduler is the better fit.
Managed connectorsFivetran · Snowflake OpenflowChange data capturelog-based CDC
03 · Platform
Raw and staging layersSnowflake · Databricks · Fabricdbt or Spark transformsstaging · intermediate · martsOrchestration and CI/CDAirflow · GitHub Actions · Terraform
04 · Outputs
Data martsfinance · sales · membersTests and alertsfreshness · volume · keys
SnowflakeDatabricksdbtMicrosoft Fabric
in short·15-second read
Data engineering builds the pipelines that move data from source systems into a data platform and transform it into usable tables.
Cohash builds pipelines on Snowflake, Databricks and Microsoft Fabric using dbt, Fivetran, Airflow and platform-native tools. We add tests, version control, CI/CD and cost monitoring, so pipelines are reliable and your team can maintain them after we hand over.
you probably need this if
Overnight a pipeline failed, and the first you heard of it was from the people reading the dashboard.
Transformations live in stored procedures and scheduled queries nobody has in version control.
Warehouse or cluster costs grew faster than the data did.
Data engineer (Spark, SQL)Snowflake data engineerDatabricks data engineer
SnowPro Advanced: Data Engineer
Databricks Certified Data Engineer Professional
From the first call to a setup your own team can run.
Here's what usually prompts a call about your Data Engineering, what we do about it and in which order, and where you end up.
01 · where it hurts02 · what we change03 · how it runs04 · where you end up
incident board · data engineering3 open
openOvernight a pipeline failed, and the first you heard of it was from the people reading the dashboard.
recurringTransformations live in stored procedures and scheduled queries nobody has in version control.
blockedWarehouse or cluster costs grew faster than the data did.
exposure
IngestionTransformation with dbtSpark and streaming on DatabricksMicrosoft FabricOrchestration and CI/CDReliability and cost
startrehearsehandover
Assess
Design
Build
Hand over
we are here
ready to hand over
Airflow is on a supported release again, so patches apply.
A pull request now builds only what it touched.
Pipeline architecture and coding standards
dbt or Spark projects in version control
chapter 01 · where it hurts
Why teams reach out
Overnight a pipeline failed, and the first you heard of it was from the people reading the dashboard.
Transformations live in stored procedures and scheduled queries nobody has in version control.
Warehouse or cluster costs grew faster than the data did.
chapter 02 · what we change
How we solve it
Ingestion
Transformation with dbt
Spark and streaming on Databricks
Microsoft Fabric
Orchestration and CI/CD
Reliability and cost
chapter 03 · how it runs
Our approach
Assess
Design
Build
Hand over
chapter 04 · where you end up
What changes
Airflow is on a supported release again, so patches apply.
A pull request now builds only what it touched.
Pipeline architecture and coding standards
dbt or Spark projects in version control
what we do
Where we spend the hours
6 areas of work, each one done by an engineer who has run it in production.
Common sources get a managed connector such as Fivetran or Snowflake Openflow. Databases get change data capture. For awkward systems like Facets or an old ERP, we write custom extracts.
Our dbt projects have staging, intermediate and mart layers. Keys and relationships are tested, and the docs come straight from the code, so they don't drift.
For big or streaming workloads we write Spark pipelines that land Delta tables governed in Unity Catalog.
We set up Fabric lakehouses, warehouses, Data Factory pipelines and mirroring. The Delta tables live in OneLake, which is what Direct Lake reporting reads.
Airflow or native schedulers, plus pull-request checks, environment promotion and Terraform for the infrastructure.
We alert on freshness and volume, triage failed runs, and check warehouse or cluster sizing against real usage.
capability 01 / 06in production
Ingestion
Common sources get a managed connector such as Fivetran or Snowflake Openflow. Databases get change data capture. For awkward systems like Facets or an old ERP, we write custom extracts.
Our dbt projects have staging, intermediate and mart layers. Keys and relationships are tested, and the docs come straight from the code, so they don't drift.
We set up Fabric lakehouses, warehouses, Data Factory pipelines and mirroring. The Delta tables live in OneLake, which is what Direct Lake reporting reads.
We pieced these together from Data Engineering situations that come up a lot. They're patterns. You won't find a specific client in them.
case file 01
A few hundred Airflow 2 DAGs still running after end of life
composite · not a client story
Beforethe situation
A self-managed Airflow 2 cluster runs a few hundred DAGs. Airflow 2 hit end of life on 22 April 2026, so there are no more security fixes. Some DAGs still use SubDAGs. Several custom operators open sessions on the metadata database directly.
What we dothe work
We inventory DAGs by owner and run history and retire the ones nobody needs. Then we fix what Airflow 3 removed: SubDAGs become TaskGroups, direct database access moves to the API client, and dropped context keys like execution_date are replaced. DAGs move in batches to a parallel Airflow 3 environment.
Afterwhere you end up
Airflow is on a supported release again, so patches apply. The custom operators use supported interfaces now. Next upgrade, there's less to rewrite.
case file 02
A dbt project of 1,100 models that rebuilds everything on every pull request
composite · not a client story
Beforethe situation
A dbt project on Snowflake has grown to about 1,100 models. CI rebuilds the whole project on every pull request on an X-Large warehouse, so merges wait over an hour and the CI warehouse is one of the biggest lines on the bill.
What we dothe work
We switch CI to state:modified+ selection with --defer, so only changed models and their children build and unbuilt parents resolve to production. Big fact tables move to incremental models with a merge strategy. The CI warehouse is downsized with a short auto-suspend, and the slowest models are profiled from query history.
Afterwhere you end up
A pull request now builds only what it touched. Merges don't queue, the CI warehouse bill drops, and nobody batches changes just to dodge the wait.
How a tested pipeline moves data from source to marts
Sources land through managed connectors, CDC or custom extracts into raw tables. dbt or Spark builds staging, intermediate and mart layers, Airflow schedules the runs, and tests and freshness alerts fire before bad data reaches reports.
Operational databasesPostgres · SQL Server · Oracle → Change data capture · transaction log
Legacy systemsFacets · legacy ERP extracts → Raw and staging layers · file extracts
02 · Ingestion
Managed connectorsFivetran · Snowflake Openflow → Raw and staging layers · raw tables
Change data capturelog-based CDC → Raw and staging layers · change stream
03 · Platform
Raw and staging layersSnowflake · Databricks · Fabric → dbt or Spark transforms · source tables
dbt or Spark transformsstaging · intermediate · marts → Data marts · build models→ Tests and alerts · test results
Orchestration and CI/CDAirflow · GitHub Actions · Terraform → dbt or Spark transforms · scheduled runs
04 · Outputs
Data martsfinance · sales · members
Tests and alertsfreshness · volume · keys
Read the diagram as text
SaaS apps → Managed connectors: API sync
Operational databases → Change data capture: transaction log
Legacy systems → Raw and staging layers: file extracts
Managed connectors → Raw and staging layers: raw tables
Change data capture → Raw and staging layers: change stream
Orchestration and CI/CD → dbt or Spark transforms: scheduled runs
Raw and staging layers → dbt or Spark transforms: source tables
dbt or Spark transforms → Data marts: build models
dbt or Spark transforms → Tests and alerts: test results
how it runs
A typical engagement
We don't start a step until the last one has produced something you can check.
step 01
Assess
We measure before we change anything. Sources, pipelines, volumes, freshness needs, and the current platform bill.
step 02
Design
Ingestion patterns, layer conventions, naming, testing and deployment standards agreed with your team.
step 03
Build
Pipelines delivered source by source, each with tests and monitoring before it reaches production.
step 04
Hand over
Runbooks, on-call guidance and pairing sessions, until your engineers own the pipelines.
you keep
What you're left with
We write these for your Data Engineering setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Put these to anyone bidding on your Data Engineering work. The ones who've done it answer differently from the ones who've read about it.
Q1
When a pipeline fails at 3 a.m., what alerts, who gets paged and what do they read first?
Every pipeline fails eventually. Ask whether tests stop the bad data before it hits a report, whether the alert finds a real owner and whether anyone wrote down what to do.
Q2
How do you decide, for each of our biggest tables, whether to load incrementally or refresh in full?
Full refreshes are simple and expensive. Incremental models are cheaper but need a reliable change column and a plan for late-arriving rows. It's a per-table decision.
Q3
Somebody renames or drops a source column. How do you catch that before it breaks a metric?
A renamed field in Salesforce or an ERP extract can quietly null a column downstream. Ask about source freshness and schema tests, model contracts and who approves the fix.
Q4
Which jobs run under service principals, and how are their credentials rotated?
Pipelines often run under a departed engineer's account or a key that never expires. Each job should have its own workload identity with scoped grants.
Q5
After the build, can you show us what each pipeline costs?
Snowflake query tags, Databricks cluster tags and Fabric capacity metrics let you pin spend on a particular job. Skip that, and a cost review is mostly guessing.
01Should we use Fivetran or build our own connectors?
For common SaaS sources a managed connector usually costs less than maintaining your own. Custom extracts make sense for odd systems, very high volumes, or data that can't leave your network. We price both before choosing.
02Fivetran and dbt Labs merged. Does that change anything for us?
Existing projects keep working. The deal closed on 1 June 2026, and the company operates as Fivetran + dbt Labs for now. Product and licensing announcements are what we watch. If one affects your setup, you'll hear from us.
03dbt or Spark for transformations?
Use dbt for SQL transformations on Snowflake, Databricks SQL or a Fabric warehouse. Reach for Spark when the job is huge, streaming or Python-heavy. Lots of teams use both.
04Can you reduce our Snowflake or Databricks bill?
Often, yes. We look at warehouse and cluster sizing and auto-suspend settings. We also find slow models and jobs that run more often than anyone needs. Savings depend on what turns up.
05We're still on Airflow 2. Is that a problem?
If you self-host, yes. Apache Airflow 2 hit end of life on 22 April 2026, so it gets no more fixes. Managed services publish their own support dates. Most of an Airflow 3 move is fixing DAGs that rely on removed features, then testing them in a parallel environment.
related
Often paired with
Projects on Data Engineering usually end up touching one of these, so they're worth a look.
