data platform

One Databricks workspace that data, ML and security people can share without fighting over it.

Agents and models now sit right next to your tables, and somebody has to decide who gets to call what. We build Databricks lakehouses, move workloads onto Unity Catalog, and set up Lakeflow pipelines and model serving.

Talk to a Databricks engineer What changed recently

platform dossierDatabricksData platform

Latest runtime: Databricks Runtime 19 (GA 23 Jul 2026); Runtime 18 is now LTS
LTS end of support: 14.3 LTS on 1 Feb 2027; 15.4 LTS on 19 Aug 2027
Genie pricing: Pay-as-you-go; Genie One and Genie Agents free for users through 31 Jan 2027
Recently GA: Domains, external writes to managed Delta tables (Oct 2026)

release watch

Runtime 14.3 LTS and 15.4 LTS end of support1 Feb 2027
Free Genie One and Genie Agents usage ends31 Jan 2027

connects with

in short·15-second read

Cohash builds Databricks lakehouses and keeps them running.

That covers Unity Catalog design, getting tables off the Hive metastore, Lakeflow pipelines, model serving and Agent Bricks. If you're still on Runtime 14.3 LTS, support ends on 1 February 2027. Genie One and Genie Agents are free for users through 31 January 2027, so set budgets before that ends.

you probably need this if

Tables, jobs and mount points still depend on the legacy Hive metastore.
Scheduled notebooks run production, and not one of them has a test or a deployment process behind it.
All-purpose clusters stay up overnight and nobody owns the compute policies.

terms on this page LTSHive metastoreUnity Catalog

Ask an engineer facts checked 2026-11-03

how it runs: 01Workspace and metastore inventory
02Unity Catalog design
03Pipelines as code
04AI governance
05Handover
you keep: An inventory of the Hive metastore: every table, mount and job, who owns it, and which wave it moves in
A Unity Catalog design that shows, domain by domain, which catalogs and schemas exist and who gets which grants
Asset Bundle templates with dev, test and prod targets and CI already wired in, so each new pipeline starts from one
+ 2 more documents
who does it: Databricks data engineer (Lakeflow, Delta Lake)Unity Catalog administratorDatabricks machine learning engineer Databricks Certified Data Engineer Associate Databricks Certified Data Engineer Professional
platforms
proof: 8 case studies on this
background: field note · 14 minYour Databricks AI Functions need Runtime 17.3 LTS by January 2027, even if 15.4 is still supported

databricks · release timeline

16 Jun 2026 Unity Catalog at Data + AI Summit 2026 At the Summit, external lineage went GA, and so did geospatial types in Delta and Iceberg v3. Domains and external access to managed Delta tables went into Public Preview, and Contextual Service Policies into Beta. Domains, and external writes to managed tables, have since reached GA.
16 Jun 2026 Agent Bricks at Data + AI Summit 2026 At the Summit, Document Intelligence went GA with SQL functions like ai_parse_document and ai_extract, and Databricks announced agent memory on Lakebase. Some of it has already moved on. Knowledge Assistant and Supervisor Agent turned legacy on 12 October, so build new code-first agents with the Agent Bricks CLI and no-code ones in Genie Agents.
8 Jul 2026 Genie moved to pay-as-you-go Since 8 July 2026, Genie Agents, Genie Code and Genie One are billed pay-as-you-go, with 150 DBUs of free LLM usage a month. Databricks puts that at about $10.50 in US East. Genie Code usage above the allowance gets a 25% promotional discount through 31 January 2027.
23 Jul 2026 Runtime 19 GA, Runtime 18 moved to LTS Runtime 19 went GA on 23 July 2026, on Apache Spark 4.2.0, and Runtime 18 became LTS that day. That gives 18 LTS three years of fixes, until 10 June 2029, which makes it the obvious landing spot for jobs leaving 14.3 and 15.4. Reach for 19 when you need something like automatic change data feed or ABAC on views, because those need Runtime 19 or later.
4 Aug 2026 Unity Gateway generally available Unity Gateway went GA on 4 August as the part of Unity Catalog that governs AI. Teams use it to decide which models and services people can call, and to keep track of MCP servers and what they cost. Service policies and agent services stayed in Beta, so keep those in development for now.
13 Oct 2026 Domains and external writes to managed Delta tables GA The Discover page and domains went GA on 8 October 2026. Since 13 October, external Spark clients can create and write Unity Catalog managed Delta tables at GA, as long as catalog commits are turned on. Cross-engine ABAC went GA on 1 October too, so row filters and column masks still apply when supported external engines read managed tables.
26 Oct 2026 Lakeflow observability GA, new Beta connectors Lakeflow Jobs now show query performance metrics and data volumes at the job level, with an Optimize button that opens Genie Code. That went GA on 26 October. Lakeflow Connect also got Beta connectors for Microsoft Entra ID, Box Audit Logs, Google Analytics Aggregated, PayPal and QuickBooks between 22 and 26 October. Keep Beta connectors out of pipelines that feed finance reporting.
27 Oct 2026 Unity Gateway Usage Dashboard and GPT-6.1 Terra Version 1.0 of the Unity Gateway Usage Dashboard came out on 27 October. It shows coding-agent adoption and estimated session spend, and it breaks spend down by model, by user and by tag. OpenAI GPT-6.1 Terra was added to Unity Gateway the same day. Check your tags first, because attribution by tag is only as good as the tagging.
28 Oct 2026 Serverless GPU in clean rooms, file editing in volumes Clean room notebooks can now run on serverless GPU compute for inference and training, GA since 28 October. Editing files in Unity Catalog volumes went GA too, so you don't have to download a file and upload it again just to change it. DataFrame caching on serverless has been in Public Preview since 23 October. It's on by default for most workspaces, and admins can switch it off.
today · Nov 09, 2026
31 Jan 2027ahead Free Genie One and Genie Agents usage ends Billing for Genie One and Genie Agents has been paused since 15 July 2026, and usage by users stays free through 31 January 2027 (service principals are still billed). Budget controls don't apply to them until then, so nobody sees what they'd cost. Pull the usage now and set budgets before February. Genie Code is still billed the whole time.
1 Feb 2027ahead Runtime 14.3 LTS and 15.4 LTS end of support Support for Databricks Runtime 14.3 LTS ends on 1 February 2027, and 15.4 LTS follows on 19 August 2027. After that Databricks may stop updating them. Find the jobs and clusters still pinned to either one and test them on 17.3 LTS or 18 LTS before the date, while there's time to fix what breaks.

the story · databricks estate

From the first call to a setup your own team can run.

Here's what usually prompts a call about your Databricks estate, what we do about it and in which order, and where you end up.

01 · where it hurts02 · what we change03 · how it runs04 · where you end up

incident board · databricks estate3 open

openTables, jobs and mount points still depend on the legacy Hive metastore.

recurringScheduled notebooks run production, and not one of them has a test or a deployment process behind it.

blockedAll-purpose clusters stay up overnight and nobody owns the compute policies.

exposure

Unity Catalog design and migrationLakeflow pipelinesPlatform setup and cost controlAI governance in Unity CatalogAgents and model servingInteroperability with Snowflake…

startrehearsehandover

Workspace and metastore…

Unity Catalog design

Pipelines as code

AI governance

Handover

we are here

ready to hand over

Every table has an owner, and its grants live in Unity Catalog.
Finance can now pull up what support's agents cost last month.
An inventory of the Hive metastore: every table, mount and job, who owns it, and which…
A Unity Catalog design that shows, domain by domain, which catalogs and schemas exist…

chapter 01 · where it hurts
Why teams reach out
- Tables, jobs and mount points still depend on the legacy Hive metastore.
- Scheduled notebooks run production, and not one of them has a test or a deployment process behind it.
- All-purpose clusters stay up overnight and nobody owns the compute policies.
chapter 02 · what we change
How we solve it
- Unity Catalog design and migration
- Lakeflow pipelines
- Platform setup and cost control
- AI governance in Unity Catalog
- Agents and model serving
- Interoperability with Snowflake and others
chapter 03 · how it runs
Our approach
- Workspace and metastore inventory
- Unity Catalog design
- Pipelines as code
- AI governance
- Handover
chapter 04 · where you end up
What changes
- Every table has an owner, and its grants live in Unity Catalog.
- Finance can now pull up what support's agents cost last month.
- An inventory of the Hive metastore: every table, mount and job, who owns it, and which wave it moves in
- A Unity Catalog design that shows, domain by domain, which catalogs and schemas exist and who gets which grants

what we do

Our Databricks work

6 kinds of Databricks work, each run by engineers who have done it on live tenants.

We lay out the metastore, catalogs and schemas, then set grants and attribute-based policies. Tables and jobs still on the legacy Hive metastore get migrated across.
Ingestion with Lakeflow Connect and declarative pipelines for transformation, with expectations, retries and alerts defined per table.
Workspaces, compute policies, serverless settings, budgets and system tables for cost reporting, deployed with Terraform or Databricks Asset Bundles.
Foundation model access, MCP servers and agent tools registered and permissioned in Unity Catalog, with Unity Gateway limits and tracing turned on.
Agent Bricks and model serving endpoints built on governed data, evaluated against real questions and monitored after launch.
Delta Sharing, Iceberg and external access to managed tables set up so other engines read the same data without extra copies.

capability 01 / 06in production

Unity Catalog design and migration

We lay out the metastore, catalogs and schemas, then set grants and attribute-based policies. Tables and jobs still on the legacy Hive metastore get migrated across.

talk to an engineer about this →

capability 02 / 06in production

Lakeflow pipelines

Ingestion with Lakeflow Connect and declarative pipelines for transformation, with expectations, retries and alerts defined per table.

talk to an engineer about this →

capability 03 / 06in production

Platform setup and cost control

Workspaces, compute policies, serverless settings, budgets and system tables for cost reporting, deployed with Terraform or Databricks Asset Bundles.

talk to an engineer about this →

capability 04 / 06in production

AI governance in Unity Catalog

Foundation model access, MCP servers and agent tools registered and permissioned in Unity Catalog, with Unity Gateway limits and tracing turned on.

talk to an engineer about this →

capability 05 / 06in production

Agents and model serving

Agent Bricks and model serving endpoints built on governed data, evaluated against real questions and monitored after launch.

talk to an engineer about this →

capability 06 / 06in production

Interoperability with Snowflake and others

Delta Sharing, Iceberg and external access to managed tables set up so other engines read the same data without extra copies.

talk to an engineer about this →

typical situations

Databricks situations we walk into

We pieced these together from Databricks situations that come up a lot. They're patterns. You won't find a specific client in them.

case file 01

A Hive metastore with 3,000 tables and DBFS mounts nobody owns

composite · not a client story

Beforethe situation

Two workspaces still use the legacy Hive metastore, with about 3,000 tables and dozens of DBFS mounts whose storage keys are pasted into notebooks. Jobs run on all-purpose clusters, and lineage stops at the workspace edge.
What we dothe work

We pull the inventory first: tables, mounts, jobs and owners, from system tables and the workspace APIs. Mounts get replaced by external locations and storage credentials. Then tables move to Unity Catalog one domain at a time, and each wave's jobs switch to job compute and three-level names.
Afterwhere you end up

Every table has an owner, and its grants live in Unity Catalog. The storage keys are gone from the notebooks, and lineage now follows the jobs past the old workspace edge into the reports downstream.

case file 02

Six teams calling foundation models with no record of who spends what

composite · not a client story

Beforethe situation

Data science, support and finance each built their own agents on model serving endpoints and external models. Some months the bill jumps and nobody can explain it, and two teams call MCP servers nobody registered.
What we dothe work

We register models, serving endpoints and MCP servers in Unity Catalog. Traffic goes through Unity Gateway with rate limits and hard spend caps per team, and we turn tracing on. Service policies stay in development while they're in Beta.
Afterwhere you end up

Finance can now pull up what support's agents cost last month. A runaway job stops at its team's cap before it ever reaches the invoice, and the platform team keeps the list of MCP servers agents are allowed to reach.

case studies

Databricks case studies

All case studies

migration Data and analytics

Unity Catalog vs Hive metastore on Databricks, and how to move

We move Databricks workspaces off the Hive metastore, DBFS mounts and table ACLs and onto Unity Catalog, ahead of new workspaces dropping legacy features.

read the case study

modernization Industry platforms

AVEVA PI data to cloud analytics on Databricks or Fabric

Getting AVEVA PI System data into Databricks or Microsoft Fabric with asset context, outbound-only paths out of the plant and tag names people understand.

read the case study

AI rollout AI and agents

AI pilot to production with forward deployed engineers

Taking an AI pilot into production with forward deployed engineers who work with the business team, build evals, connect real systems and hand over cleanly.

read the case study

upgrade Data and analytics

Apache Airflow 2 to Airflow 3

Upgrading self-managed Apache Airflow 2, now past end of life, to Airflow 3, and fixing removed features, auth and scheduling changes before DAGs move.

read the case study

implementation Industry platforms

TriZetto Facets upgrades and CMS-0057-F interoperability APIs

Keeping TriZetto Facets current while we build the Provider Access, Payer-to-Payer and Prior Authorization APIs that CMS-0057-F requires by January 2027.

read the case study

migration Data and analytics

Informatica PowerCenter to IDMC or dbt

Moving PowerCenter mappings and workflows to Informatica IDMC Cloud Data Integration or to dbt on the warehouse, decided mapping by mapping.

read the case study

approach

How a Databricks engagement runs

We don't start a step until the last one has produced something you can check.

step 01
Workspace and metastore inventory

We start by listing every workspace, cluster, job, table, mount and grant. Then we read the system tables to see where the compute spend actually goes.
step 02
Unity Catalog design

Catalogs, schemas, grants and attribute-based policies get laid out by domain and environment. After that, Hive metastore tables and their jobs move across in waves, one domain at a time.
step 03
Pipelines as code

We move Lakeflow pipelines and jobs into Databricks Asset Bundles or Terraform and add expectations and CI. Dev, test and prod each get their own target.
step 04
AI governance

We register every model, serving endpoint and MCP server in Unity Catalog. Then each team gets its own rate limits and spend cap in Unity Gateway, and we turn tracing on so you can see what the agents actually called.
step 05
Handover

We hand the compute policies, budgets and deployment templates to your platform team, along with a runbook they'll use the next time a new team needs onboarding.

ai on databricks

Where AI fits on Databricks, and where it needs guardrails.

Databricks keeps AI next to the data. Agent Bricks builds agents, there's model serving, and there are AI functions such as ai_parse_document. Unity Catalog governs the models, agents and MCP servers. Unity AI Gateway adds hard spend caps and cost attribution. A lot of the AI governance is new and some is still Beta, so we check status feature by feature.

AI agentasks for data or an action

Scoped identityonly the access it needs

MCP serverthe one door into the platform

Logged, approveda person signs off on writes

Databricksdoes the work

Databricks publishes managed MCP servers such as Genie One and Databricks SQL, and teams can register external or custom ones. Access follows Unity Catalog permissions. Unity AI Gateway controls credentials and monitors use. We run agents as service principals with narrow grants, trace their calls and cap spend per team.

Agent Bricks Document Intelligence

Document Intelligence in Agent Bricks is GA. ai_parse_document, ai_extract and ai_classify are SQL functions, so the extraction runs in a query. We use them on invoices, contracts and claims, and nothing feeds a downstream table until it scores well against our labelled samples.

Unity AI Gateway

AI Gateway governs traffic to models, agents and MCP servers, with hard spend caps and cost attribution by user and team. Guardrail policies are in Beta. We set rate limits and caps per team, turn on tracing and keep Beta guardrails out of regulated workloads.

Managed MCP servers

Databricks provides managed MCP servers, Genie One for business questions and Databricks SQL for queries among them. It also supports external servers and custom ones built on Databricks Apps. Unity Catalog permissions apply to all of them. We keep a register of every server an agent can reach, and anything not on it doesn't run.

Agent memory on Lakebase

Databricks announced agent memory on Lakebase at Data + AI Summit 2026. Announced isn't shipped. We'd prototype it in development and wait for general availability before any agent depends on it.

architecture

How data reaches Unity Catalog through Lakeflow

Databases, SaaS apps and event streams feed Lakeflow Connect and declarative pipelines, which write Delta or Iceberg tables. Unity Catalog governs those tables whoever reads them, whether that's a SQL user, an agent, or Snowflake reading over Delta Sharing.

use cases

Typical Databricks projects

The work teams usually bring us, and the platforms Databricks has to talk to while we do it.

DAT-01 Your tables and jobs still live on the Hive metastore, mount points included common request
DAT-02 Notebooks run in production with no tests or deployment process common request
DAT-03 Databricks spend is growing and no one owns the compute policies common request
DAT-04 Several teams are building agents, and nobody can say which models and tools they're calling common request
DAT-05 You keep the same data in Snowflake and in Databricks, and pay to store it twice common request
DAT-06 Jobs still pinned to Runtime 14.3 LTS or 15.4 LTS common request

Databricks · connected platforms

Snowflake dbt Terraform Microsoft Fabric OpenAI API Anthropic Claude

you keep

What a Databricks engagement leaves behind

These are working documents, written against your configuration, so your team can run them without us.

inventory01

An inventory of the Hive metastore: every table, mount and job, who owns it, and which wave it moves in
yours to keep
inventory02

A Unity Catalog design that shows, domain by domain, which catalogs and schemas exist and who gets which grants
yours to keep
tests03

Asset Bundle templates with dev, test and prod targets and CI already wired in, so each new pipeline starts from one
yours to keep
report04

Compute policies and a budget for each team, plus cost reports built on system tables so you can see who spent what
yours to keep
document05

Unity Gateway set up with rate limits and a spend cap for each team, so one runaway agent can't eat the month's budget
yours to keep

talent and delivery · databricks

Certified engineers, wherever you need them.

Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.

Onsite

Engineers at your sites in the US

Nearshore

Overlapping hours with US teams

Offshore

Overnight progress on your backlog

All time zones

Coverage handed over across time zones

24-hour coverage · US Central time

OnsiteUS business hours

Nearshoreoverlapping US hours

Offshoreovernight progress

Combinedhandover, not hand-off

certifications our engineers hold

Databricks Certified Data Engineer AssociateDatabricks
Databricks Certified Data Engineer ProfessionalDatabricks
Databricks Certified Machine Learning AssociateDatabricks
Databricks Certified Generative AI Engineer AssociateDatabricks

roles we staff

Databricks data engineer (Lakeflow, Delta Lake)
Unity Catalog administrator
Databricks machine learning engineer
Databricks platform architect
Spark performance engineer
Databricks generative AI engineer

due diligence · databricks

Questions to ask any partner, including us.

Put these to anyone bidding on your Databricks work. The ones who've done it answer differently from the ones who've read about it.

Q1
Before we migrate, how do you find every DBFS mount and every storage key someone pasted into a notebook?
When tables move to Unity Catalog, it's the mounts and the keys pasted into notebooks that break. Security usually flags those same keys, so the inventory does double duty.
Q2
Which of the Data + AI Summit 2026 features you're proposing are actually GA, and which are Preview or Beta?
Only some of it is GA. External lineage and domains are, and service policies are still Beta. Don't put regulated workloads on anything whose behavior can still change.
Q3
How will you attribute compute and model spend to teams?
System tables record usage, but attribution only works when tags, compute policies and Unity Gateway settings are consistent. Without that, cost reviews end in arguments about whose cluster it was.
Q4
What goes into a Databricks Asset Bundle, and what stays in Terraform?
Workspaces, networking and metastores usually live in Terraform, while jobs and pipelines live in bundles. A fuzzy boundary means two tools fight over the same resources.
Q5
How will Snowflake or another engine read our tables without a second copy?
Delta Sharing, Iceberg reads and external access to managed tables aren't equally mature or equally governed. Whichever you pick decides who controls access and where lineage gets recorded.

01Do we have to migrate from the Hive metastore to Unity Catalog?

New governance, lineage and AI features from Databricks are built on Unity Catalog. Stay on the Hive metastore and you miss them. The effort comes down to how many tables, jobs and mount points you have, so we inventory those and then migrate in waves.

02When does Databricks Runtime 14.3 LTS stop getting support?

1 February 2027, going by Databricks' runtime support table, and 15.4 LTS follows on 19 August 2027. Once both are gone you're left with 16.4, 17.3 and 18 LTS, and 18 runs until 10 June 2029. Test your jobs on the version you're moving to before you switch them.

03Is Databricks Genie free to use?

Not all of it, and not for long. Genie One and Genie Agents have been free for users since 15 July 2026 and stay free through 31 January 2027. Genie Code is billed pay-as-you-go, with 150 DBUs of LLM usage free each month and 25% off anything above that until the same date. Budget controls don't cover the free products, so pull their usage now and set budgets before February.

04What drives Databricks cost?

Compute type and runtime, mainly. All-purpose clusters left running, oversized job clusters and unbounded model serving are common causes. System tables show where spend goes, and compute policies and budgets keep it in check.

05Which of the Summit features can we actually run in production yet?

Look up each feature's status. External lineage and geospatial types went GA at the Summit, and domains and external writes to managed Delta tables went GA in October 2026. Service policies are still Beta. Beta is fine in development. Regulated work waits for GA.

06Should we choose Databricks or Snowflake?

It depends on what you run and who runs it. Databricks suits heavy Spark, streaming and ML engineering work. Snowflake suits analytics teams that think SQL-first. Plenty of organizations run both and share the data through open table formats.

07Can our whole Databricks setup live in Terraform from the first day?

Yes. We usually put accounts, workspaces, networking and Unity Catalog metastores in Terraform, and jobs and pipelines in Databricks Asset Bundles. That split keeps the platform team and data teams from editing the same resources.

services

Related services

Projects on Databricks usually end up touching one of these, so they're worth a look.

you are hereDatabricks

fact-checked Checked against vendor and official documentation last checked 2026-11-03 content fingerprint e7ef7fb1

01Book a 30-minute working sessionsenior engineer
02Talk to an engineer about your setupreply within 24 hours
03See how similar projects wentcase studies

Reviewed by the Cohash team Databricks is a trademark of its owner, used here to identify the platform. how we handle facts and data →

next step

Tell us what you're running. We'll tell you what we'd do first.

A 30-minute working session with a senior engineer. No slides.

Book a working session See the case studies

useful to bring

The platform and the version you're on
Where you need to get to, and by when
What's custom: rules, plugins, integrations
Any audit finding or deadline driving it

One Databricks workspace that data, ML and security people can share without fighting over it.

From the first call to a setup your own team can run.

Why teams reach out

How we solve it

Our approach

What changes

Our Databricks work

Unity Catalog design and migration

Lakeflow pipelines

Platform setup and cost control

AI governance in Unity Catalog

Agents and model serving

Interoperability with Snowflake and others

Databricks situations we walk into

Databricks case studies

Unity Catalog vs Hive metastore on Databricks, and how to move

AVEVA PI data to cloud analytics on Databricks or Fabric

AI pilot to production with forward deployed engineers

Apache Airflow 2 to Airflow 3

TriZetto Facets upgrades and CMS-0057-F interoperability APIs

Informatica PowerCenter to IDMC or dbt

How a Databricks engagement runs

Workspace and metastore inventory

Unity Catalog design

Pipelines as code

AI governance

Handover

Where AI fits on Databricks, and where it needs guardrails.

Agent Bricks Document Intelligence

Unity AI Gateway

Managed MCP servers

Agent memory on Lakebase

How data reaches Unity Catalog through Lakeflow

Typical Databricks projects

What a Databricks engagement leaves behind

Certified engineers, wherever you need them.

Onsite

Nearshore

Offshore

All time zones

Questions to ask any partner, including us.

Before we migrate, how do you find every DBFS mount and every storage key someone pasted into a notebook?

Which of the Data + AI Summit 2026 features you're proposing are actually GA, and which are Preview or Beta?

How will you attribute compute and model spend to teams?

What goes into a Databricks Asset Bundle, and what stays in Terraform?

How will Snowflake or another engine read our tables without a second copy?

Related services

Tell us what you're running. We'll tell you what we'd do first.