Unity Catalog vs Hive metastore on Databricks, and how to move
The Hive metastore is a legacy catalog that belongs to one workspace. Unity Catalog sits at account level, with built-in auditing, lineage and access control. Cohash moves you across: the UCX assessment, account groups from your identity provider, tables upgraded with SYNC or DEEP CLONE, then code fixes, and legacy access off once nothing needs it.
- 01UCX assessment
- 02Identity and groups
- 03Storage, catalogs and grants
- 04Table upgrade
- 05Code, jobs and compute
- 06Disable legacy access
The story
Told without the client's name or anything that could identify them.
A retailer's data platform team ran several Databricks workspaces, each with its own Hive metastore and its own copy of the same groups. Jobs read from /mnt paths whose credentials sat in cluster configs nobody owned, and a new business-unit workspace had failed because the provisioning script expected DBFS mounts. We ran the UCX assessment to build the backlog, synced groups to the account from the identity provider, and wrote catalogs and grants in Terraform. Tables moved with SYNC or DEEP CLONE, notebooks and jobs came off mounts and two-part names, and legacy access was switched off once nothing depended on it.
why it was urgentExisting workspaces keep running. The change is for new ones: from 8 October 2026, Databricks provisions new workspaces without the Hive metastore, DBFS root and mounts, no-isolation shared clusters or runtimes older than 13.3 LTS. Accounts created after 18 December 2025 get no legacy features at all, so a team that spins up a fresh workspace can't carry the old patterns into it.
- openEvery workspace has its own Hive metastore, and someone keeps a copy of the same groups in each one.
- recurringJobs read from /mnt paths, and the credentials behind them live in a cluster config nobody owns.
- blockedThe provisioning script for a new business-unit workspace failed, because it assumes DBFS mounts exist.
- openAuditors ask for table-level lineage and access history, and the best anyone can offer is a spreadsheet.
The estate, before and after
Before: a Hive metastore and local groups in every workspace
Each workspace kept its own groups and Hive metastore, and jobs reached storage through DBFS mounts whose credentials lived on clusters. Access rules differed between workspaces and nobody owned the mount credentials.
After: every workspace takes its rules from Unity Catalog
Groups come from the identity provider at account level, and Unity Catalog holds the grants, written in Terraform. Jobs use three-part names and volumes, and tables not moved yet are read through Hive metastore federation.
How we run it, phase by phase.
6 phases take you from Hive metastore, DBFS root and mounts to Databricks Unity Catalog. Each one ends with something you can check before the next starts. As you scroll, the blocks show the estate moving across.
- phase 01
UCX assessment
Run the UCX assessment workflow to inventory tables, formats, mounts, clusters, jobs, grants and groups in each workspace. The dashboard becomes the migration backlog, and we re-run it before each wave.
- phase 02
Identity and groups
Sync users and groups from Entra ID or Okta into the Databricks account through SCIM. Then migrate workspace-local groups to account groups, permissions and all, and remove the local copies.
- phase 03
Storage, catalogs and grants
We create the metastore, storage credentials and external locations, and design catalogs by domain or environment. Unity Catalog grants are written in Terraform and reviewed, never clicked into the UI.
- phase 04
Table upgrade
We upgrade external tables in place with SYNC. For managed tables in the DBFS root it's DEEP CLONE, which only takes Delta or Parquet, so anything else gets converted first. Tables not worth moving yet stay readable through Hive metastore federation.
- phase 05
Code, jobs and compute
Mount paths become volumes or external locations, two-part names become three-part names, and jobs move to Unity Catalog access modes on 13.3 LTS or later. UCX has no code migration workflow yet, only a linter and experimental commands, so our engineers review and finish this part themselves.
- phase 06
Disable legacy access
Turn on the workspace settings that disable Hive metastore access and DBFS root and mounts. Then update provisioning automation and set the metastore to auto-assign for every region.
What the client got
What changed for the client once the work was done. Described, not scored: we don't publish client figures.
- result 01
Groups now come from the identity provider, one set for the account
SCIM now syncs users and groups from Entra ID or Okta to the Databricks account. We deleted the workspace-local copies.
- result 02
Grants are code, not clicks
We put catalogs, schemas, storage credentials and Unity Catalog privileges in Terraform, and nothing changes without review.
- result 03
Jobs no longer depend on mounts
We moved notebooks and jobs to volumes or external locations and three-part table names, running on Unity Catalog access modes with 13.3 LTS or later.
- result 04
Data owners approved the access changes
Some legacy table ACLs and ANY FILE grants had no Unity Catalog equivalent. Data owners signed off the resulting access changes beforehand, so no user found out by hitting an error.
- result 05
New workspaces provision without legacy features
We updated provisioning so new workspaces come up with the Hive metastore and DBFS mounts disabled, and the metastore assigns itself in each region.
Risks we plan for
-
UCX is a Labs project
UCX comes from Databricks Labs without formal support or SLAs. It does most of the inventory and table work well, but plans can't assume it covers everything.
-
Permission drift
Legacy table ACLs, cluster-level credentials and ANY FILE grants don't map one to one to Unity Catalog privileges. Some users will lose access they shouldn't have had, and that needs to be agreed, not discovered.
-
Jobs breaking quietly
A job that writes to a mount can keep writing to the old location after the table moves. We freeze writes per table during its cut-over and compare row counts on both sides.
-
Managed tables that can't clone
DEEP CLONE won't copy CSV, JSON or Avro managed tables in the DBFS root until they're converted. Big ones need a maintenance window.
Where AI helps, and where it needs guardrails
Databricks governs AI through Unity Catalog too. Agent Bricks and Unity AI Gateway (the piece that adds spend caps and cost attribution for models, agents and MCP servers) both depend on data being registered and permissioned there. We let coding agents suggest notebook path and table-name rewrites as pull requests during the migration, and an engineer reviews every one. Agents never hold metastore admin.
forward deployed engineers →What you're left with
We write these for your setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.
- plan01
UCX assessment results, plus a wave plan that tells each workspace what moves and when
yours to keep - inventory02
Catalog, schema, storage credential and grant definitions, all in Terraform so changes get reviewed
yours to keep - document03
Group migration log and an access-change list. Data owners approve that list before anyone loses access.
yours to keep - document04
Updated notebooks and jobs, with before-and-after row counts per table
yours to keep - document05
Provisioning automation that still works once legacy features are gone
yours to keep
Platforms and services on this move
Learn the background
Terms from this page in plain English. If you want the why, the field note behind it is right there.
- 01SCIM
- A way for a company's directory to push users and groups into another system. Auth0 can then keep those groups as their own objects instead of on the profile.in context →
- 02LTS
- Long-term support. Databricks fixes stability and security problems in an LTS runtime for three years after release. Other runtimes get six months, which is why an LTS is the one to build on.in context →
- 03Hive metastore
- The list of tables and file locations each Databricks workspace kept for itself before Unity Catalog. Databricks now calls it legacy.in context →
- 04Unity Catalog
- Databricks' shared catalog for tables, files, models and permissions. Each region gets one metastore, and every workspace attached to it sees the same objects.in context →
- 05DBFS root
- The default storage area of a Databricks workspace. Managed Hive tables created without a location usually ended up there.in context →
- 06SYNC
- A Databricks SQL command that registers your external Hive tables in Unity Catalog. It doesn't copy data, so it's quick, but it won't take DBFS root tables.in context →
Certified engineers, wherever you need them.
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Onsite
Engineers at your sites in the US
Nearshore
Overlapping hours with US teams
Offshore
Overnight progress on your backlog
All time zones
Coverage handed over across time zones
01Unity Catalog vs Hive metastore: what's the actual difference?
Scope comes first. The Hive metastore belongs to one workspace and its policies target workspace-local groups. Unity Catalog is an account-level object and its policies target account groups, so you're not keeping the same rules in several places. Dropped tables differ too: UNDROP brings back a Unity Catalog managed table for 7 days, while dropping a Hive managed table deletes its files immediately.
02What stops working when we move to Unity Catalog?
A few old habits. Unity Catalog has no DENY, no ANY FILE and no READ_METADATA, and anything you haven't granted is denied. Views can only sit on Unity Catalog tables and views, and path-based access to managed tables goes away. Search your grants and code for these before the first wave, so nobody finds out from a failed job.
03Do we have to migrate existing workspaces?
Not by a date. Databricks says existing workspaces aren't affected by the 8 October 2026 change. But any new workspace comes without legacy features, and newer platform features assume Unity Catalog, so the gap grows the longer you wait.
04How much downtime does a table upgrade need?
External tables upgraded with SYNC need very little. Managed tables in the DBFS root are copied, so time depends on size and on how long writers can pause. We schedule those by table and owner.
05Can we keep some tables in the Hive metastore for now?
Yes. Hive metastore federation lets Unity Catalog govern tables still in Hive while you finish. It's a bridge, so don't plan to live on it.
Often on the same roadmap
On-premises Active Directory to cloud-first Microsoft Entra ID
We shrink on-premises Active Directory one application at a time, moving groups, users and app authentication to Microsoft Entra ID as each app is ready.
read the case studyAD FS federation to Entra ID authentication
Converting federated domains from AD FS to Entra ID cloud authentication, moving relying parties app by app and retiring the AD FS farm.
read the case studyAI pilot to production with forward deployed engineers
Taking an AI pilot into production with forward deployed engineers who work with the business team, build evals, connect real systems and hand over cleanly.
read the case studya57ff740 - 01Book a 30-minute working sessionsenior engineer
- 02Talk to an engineer about your setupreply within 24 hours
- 03See how similar projects wentcase studies
Planning this move?
A 30-minute working session with a senior engineer. No slides.
- The platform and the version you're on
- Where you need to get to, and by when
- What's custom: rules, plugins, integrations
- Any audit finding or deadline driving it