Hive metastore to Unity Catalog: the Databricks migration plan, and what breaks on the way
New Databricks workspaces no longer come with a Hive metastore. Here's how to move tables, jobs and permissions to Unity Catalog, what tends to break, and the order we'd do it in.
part 3 of 5 · Data and AI that hold up in productionthe ideaExternal tables just re-register. DBFS root tables and old code are the real work.
after reading, you can- Tell leadership in plain words why new workspaces can't use the Hive metastore and what that costs you
- Work out which Hive tables just re-register and which ones need a full copy
- Spot the code that breaks on Unity Catalog before cutover, not after it
Databricks now creates every new workspace without the old Hive metastoredefinitionThe list of tables and file locations each Databricks workspace kept for itself before Unity Catalog. Databricks now calls it legacy., without DBFS mounts and without runtimes older than 13.3 LTS. If your tables, jobs and permissions still live in the Hive metastore, you can keep running them where they are, but they can't follow you into any new workspace.
Your Databricks tables are registered in an old catalog that new workspaces can't use
Every Databricks workspace used to keep its own list of tables, called the Hive metastore. Unity Catalog replaces that with one shared metastore per region, so every workspace there sees the same tables, with permissions, lineage and auditing in one place. Your existing workspaces keep working for now. But any new workspace you create, for a new region, a new team or a rebuild, comes up without the Hive metastore, so the jobs you'd want to move there have to be on Unity Catalog first.
Think of a company where each office keeps its own paper key register. Head office has moved to one electronic badge system. Old offices can keep their registers for a while, but new offices only get badge readers, and nobody can carry a paper key over to them.
- Teams on workspaces older than Unity Catalog
- Data platform owners
- Heads of data and CDOs
- Security and audit teams
- Anyone opening a new workspace
Why it matters to you
Nothing switches off in your current workspaces this month. The pressure comes from the edges. Databricks' own upgrade pages now call the per-workspace Hive metastore a legacy feature, and the newer governance features are built for Unity Catalog. The day someone asks for a new workspace, a disaster recovery copy or a cleaner network layout, the old setup can't come along.
There's a second, quieter trigger. An admin can turn Hive metastore access off for a workspace, and after 30 days that becomes permanent. Databricks also says it might switch legacy access off on its own, starting in July 2026, for workspaces showing no Hive metastore activity. Check what that page says about your workspaces before someone else decides for you.
-
01 Inventory what's in the Hive metastore
List every table, view, mount, init script, cluster type and job that touches the hive_metastore catalog.
-
02 Set up the account side
Set up the Unity Catalog metastore and account-level groups first. Then add a storage credential and external location for every bucket the jobs read.
-
03 Register or copy the tables
External tables are registered with SYNC. Managed tables in workspace storage have to be copied with DEEP CLONE or CTAS.
-
04 Move the code and the compute
Jobs move to three-part table names and to Standard or Dedicated compute. Anything that read files through a mount gets pointed at a volume.
-
05 Switch the old catalog off
Once nothing reads it, disable Hive metastore access. After 30 days that can't be undone.
For engineers: SYNCdefinitionA Databricks SQL command that registers your external Hive tables in Unity Catalog. It doesn't copy data, so it's quick, but it won't take DBFS root tables. doesn't copy the files under an external Hive table, though by default it adds a few table properties to the source table. Unity Catalog starts tracking the same location, which is why the cloud storage permissions have to be set up first as a storage credential plus an external location. Managed tables whose files sit in the DBFS rootdefinitionThe default storage area of a Databricks workspace. Managed Hive tables created without a location usually ended up there. are different. SYNC refuses them with a DBFS_ROOT_LOCATION status, and the only way across is a copy.
Which way off the Hive metastore? Mostly it depends on your jobs
Most teams end up choosing between three routes. Databricks' current upgrade guide puts federationdefinitionMirroring the Hive metastore as a foreign catalog in Unity Catalog, so Unity Catalog permissions apply to old tables before they move. first, then a move to Unity Catalog tables, and it recommends managed tables as the end state because that's what predictive optimization works on.
| Route | What you do | What you get | What it costs you |
|---|---|---|---|
| Stay for now | Leave existing workspaces on the Hive metastore | No work this quarter | No new workspaces for these jobs, no Unity Catalog governance on these tables, and the risk of a forced switch later |
| Federate first | Mirror the Hive metastore as a foreign catalog in Unity Catalog | Unity Catalog permissions and lineage over the old tables, with a USE CATALOG line or a new default catalog as the main code change | Writes only for the workspace's internal Hive metastore, slightly slower queries, and the real migration still to come |
| Migrate tables | SYNC external tables, DEEP CLONE or CTAS the managed ones, rewrite jobs | Tables fully in Unity Catalog, ready to become managed tables | The most effort up front: code, groups, mounts and compute all change |
- You get Unity Catalog grants and lineage on Hive tables while most job code stays put
- Jobs mostly need a USE CATALOG line or a new default catalog
- Good bridge when hundreds of jobs can't all change at once
- Queries can run a bit slower, Databricks says
- You still have to migrate later
- One move, no bridge to retire later
- Nothing left to retire later, and tables can move on to being managed
- You can't cut over until the jobs, the mounts and the groups are all ready
- DBFS root tables need a full data copy
- Bigger test effort before cutover
Federation is the route people underrate. If the worry is a few hundred jobs nobody wants to touch at once, it puts Unity Catalog permissions over the existing tables while the code moves in waves. Just don't treat it as the destination. Writes only work for the workspace's internal Hive metastore, and Databricks itself says performance can be marginally worse.
What changes, and what usually breaks
The table move is the part everyone plans for. The breakage tends to come from everything around it: how code reads files, which cluster type it runs on, and which groups own what.
Things your jobs did on the Hive metastore that work differently now
Workspaces created from 8 October 2026 don't get one. Existing workspaces keep theirs until it's disabled.
Databricks marks both deprecated. Files move to Unity Catalog volumes, external locations or workspace files, and /mnt paths in code stop being the way in.
Databricks says they've reached end of life. Compute won't start until scripts move, and Unity Catalog volumes are the recommended home.
Not available in new workspaces. We'd move existing ones to Standard or Dedicated access mode early in the project.
Shared is now Standard and single user is now Dedicated. Standard compute can't run RDD APIs or Hive UDFs, and Scala code can't use the Spark context.
Deprecated from Databricks Runtime 15.0 and due to be removed in a later runtime. Storage credentials and external locations take over.
Hive metastore table ACLs are the legacy model. Grants are set again on Unity Catalog objects, for account-level groups.
Tables become catalog.schema.table. Two-part names resolve against the default catalog, so changing that default changes what old code reads.
Unity Catalog doesn't offer it, and from Runtime 17.3 LTS it's unsupported anyway. Jobs that record which file a row came from switch to the _metadata.file_name column.
Two of these catch teams out more than the rest. The first is file access. One user on the Databricks Community forum, back in 2024, had Python code that opened files under /mnt fail with FileNotFoundError once the bucket came under Unity Catalog. A Databricks employee replied that /dbfs paths aren't reachable on shared clusters. Nothing was wrong with the tables. The code was reading files the old way.
The second is the default catalog. Workspaces that were enabled for Unity Catalog by hand kept hive_metastore as their default, so code that says sales.orders still reads the old table. Flip the default too early and the same line reads a different table, or nothing. A Databricks engineer's migration playbook on the community blog warns about exactly this and says to expect some jobs to fail during the transition.
| Hive object | Move it with | What to watch |
|---|---|---|
| External table, supported format | SYNC, or the upgrade wizard in Catalog Explorer | No data is copied. The Unity Catalog table is external and points at the same files |
| Managed table stored outside workspace storage | SYNC ... AS EXTERNAL | Comes across as an external table in Unity Catalog |
| Managed Delta table in the DBFS root | CREATE TABLE ... DEEP CLONE | Shallow clones aren't allowed from the Hive metastore. This is a full data copy |
| Anything else (non-Delta managed, odd formats) | CREATE TABLE AS SELECT | Full copy. Check partitioning and table properties afterwards |
| Views | Recreate them against the new tables | SYNC returns VIEWS_NOT_SUPPORTED. SHOW CREATE TABLE gives you the old definition |
| Grants and ownership | GRANT on the new objects, to account groups | Copied tables are owned by whoever ran the copy unless you set an owner |
Dates that frame a Hive metastore decision
- Mar 2025Shared and single user renamed
Shared becomes Standard and single user becomes Dedicated. Older scripts and forum answers still use the old names.
- Oct 2025Last tagged UCX release, v0.60.1
People still push to the repository. There just hasn't been a tagged release since, so pin whatever you test.
- 18 Dec 2025New accounts start without legacy features
Accounts created after this date never get the Hive metastore, DBFS mounts or no-isolation clusters.
- Jul 2026Automatic disabling may begin
From this point Databricks may switch legacy access off itself for workspaces with no Hive metastore activity.
- 8 Oct 2026All new workspaces start without them
This one applies in every account. Your existing workspaces carry on as they are, but any new one starts without the legacy features.
- +30 daysDisabling becomes permanent
Disable Hive metastore access and you have 30 days to change your mind. After that the workspace is Unity Catalog only for good.
The migration plan, in the order we'd run it
Before: every workspace kept its own list of tables
Each workspace registers its own tables, reads files through mounts and passes credentials through the cluster. Permissions live in table ACLs per workspace, with workspace-local groups.
After: workspaces share one Unity Catalog, so grants are set once
Any workspace attached to the metastore sees the same catalogs and grants. Jobs reach files through external locations and volumes, and permissions go to account-level groups instead of each workspace's own.
From first inventory to switching the Hive metastore off
- 01Take an inventory
Run the UCX assessment workflow, or your own scripts, and list every table with its type and location, every mount, every init script, every cluster's access mode and runtime, and every job that reads hive_metastore. Flag tables under dbfs:/user/hive/warehouse straight away, since those need copying.
- 02Get compute onto supported ground
Move clusters off no-isolation shared mode and onto Runtime 13.3 LTS or later. Move init scripts from DBFS into a Unity Catalog volume. We'd do this early, since init scripts left on DBFS stop compute from starting at all.
- 03Build the account side
Create the metastore, catalogs and schemas by domain and environment. Create account-level groups before you migrate workspace groups, because the UCX group workflow won't create them for you. Then set up storage credentials and external locations for each bucket.
- 04Decide on federation
If code changes have to come in waves, federate the Hive metastore as a foreign catalog first. Teams get Unity Catalog permissions over existing tables while jobs move over one at a time.
- 05Move the tables in waves
SYNC external tables, which only re-registers them. Copy DBFS root tables with DEEP CLONE, and use CTAS for the rest. Recreate views. Re-apply grants to account groups and set owners. Start with the tables your reports depend on most.
- 06Fix the code
Switch to three-part names, replace /mnt and /dbfs paths with volumes or external locations, swap input_file_name() for _metadata.file_name, and rewrite RDD or Spark context calls that Standard compute won't run. The UCX linter flags direct file access, RDD and Spark context calls.
- 07Run old and new side by side
Point a copy of each job at the Unity Catalog tables and compare row counts and totals for a few cycles. Keep SYNC on a schedule for external tables that are still being written through the old names.
- 08Flip the default catalog last, then turn the Hive metastore off
Change the default catalog only when no job relies on two-part names meaning hive_metastore. Watch for failures, then disable Hive metastore access. Use the 30 days before it becomes permanent as your rollback window.
Most migration guides start with UCX, the Databricks Labs toolkit. We'd use it for the assessment, the group migration and the code linter. Know what you're getting, though. Databricks says it doesn't support Labs projects directly, and the last tagged UCX release on GitHub is v0.60.1 from October 2025. Pin the version you tested with and check its dashboards against what you actually find in the workspace.
That last part matters. A user on the Databricks Community forum ran the UCX table migration and found neither the external nor the managed tables in Unity Catalog afterwards. The assessment had flagged data in the DBFS root, legacy table ACL clusters and old runtimes. Another member replied in April 2026, putting it down to managed tables sitting in the DBFS root and suggesting that data move to cloud storage first. The workflow finishing isn't the same as the tables arriving.
What to do now
What to do now, by who you are
- Ask the platform team how many tables, jobs and workspaces still depend on the Hive metastore, and which of them feed regulatory or board reporting.
- Decide whether new workspace requests can wait for the migration, or whether federation goes first.
- Put an owner and a quarter on switching the Hive metastore off, so it isn't decided for you.
- Run the UCX assessment or your own inventory, and list DBFS root tables, mounts and init scripts separately.
- Move clusters off no-isolation shared mode and onto Runtime 13.3 LTS or later.
- Create account-level groups, storage credentials and external locations before the first table moves.
- Search code for /mnt, /dbfs, input_file_name, sc. and RDD calls, and two-part table names.
- Rewrite file reads to volumes or external locations, and test each job on Standard compute.
- Compare outputs between old and new tables for a few runs before switching a job over.
- Map who can read what today in table ACLs and mounts, so you can check the Unity Catalog grants against it.
- Make sure grants go to account-level groups tied to your identity provider, not to individuals.
- Retire instance profiles and passthrough settings once nothing uses them.
Set it up so the next change is smaller
What makes the next platform change less painful
- 01Keep the catalog in code
Put metastores, catalogs, schemas, external locations and grants in Terraform, so a new workspace or region gets the same setup without anyone clicking through it.
- 02Grant to groups, never to people
Use account-level groups synced from your identity provider. When someone leaves, their access goes with the group membership, and audit has one place to look.
- 03Ban new mounts and DBFS paths
Use cluster policies to keep new compute on Standard or Dedicated modes and supported runtimes. Add a check in CI that fails a pull request using /mnt or input_file_name.
- 04Aim for managed tables
Once tables are stable in Unity Catalog, plan the move from external to managed where it makes sense. That's what Databricks' predictive optimization runs on.
If the migration goes wrong
If jobs fail or numbers drift after a cutover
The upgrade wizard leaves the original Hive tables in place, and the Hive metastore stays readable until you disable it. That gives you a way back, as long as you haven't passed the 30-day point.
- 01Stop the bleedingfirst hour
- Point failing jobs back at the hive_metastore tables, or set the old default catalog on their cluster with spark.databricks.sql.initial.catalog.namespace (clusters only, not SQL warehouses).
- Tell the people who own the reports which numbers might be stale, and since when.
- 02Find the causefirst day
- Check the job error first: missing table, permission denied, unsupported API on Standard compute or a file path that no longer resolves.
- Compare row counts and totals between the Hive table and its Unity Catalog copy for the affected dates.
- 03Fix and re-runfirst days
- Fix the grant, path or code, re-run the job against the Unity Catalog table, and compare outputs again.
- If a deep clone went stale because the Hive table kept changing, re-clone or switch writers over first.
- 04Protect the way backbefore day 30
- If you've already disabled Hive metastore access and jobs keep failing, turn it back on before the 30 days run out.
- Don't drop Hive tables or their storage until every consumer has run cleanly on Unity Catalog.
- 05Learn from itnext wave
- Add whatever broke to the code search and the CI check, so the next wave catches it before cutover.
- Write the owner of each failed job into the inventory, so the next wave knows who to call.
What this could look like
Three Databricks estates you might recognise
- the setup
- An analytics team built its tables years ago with plain CREATE TABLE, so most managed tables live under dbfs:/user/hive/warehouse. Reports refresh nightly.
- what happens
- SYNC refuses those tables with a DBFS root status, and the upgrade wizard only takes external tables. The first plan assumed a metadata-only move and has no time set aside for copying data.
- what to do
- Size the DBFS root tables and copy them with DEEP CLONE in waves, largest reports first. Re-point jobs per wave and keep the Hive tables until outputs match.
- the setup
- Underwriting and claims pipelines reference tables as schema.table across a few hundred notebooks and jobs, owned by several teams.
- what happens
- Rewriting every job at once isn't realistic, and changing the default catalog early would make the same names read different tables.
- what to do
- Federate the Hive metastore and point jobs at it with USE CATALOG or a default catalog change. Then move them onto Unity Catalog tables team by team.
- the setup
- Python jobs on a shared cluster read JSON drops from /mnt paths and stamp each row with input_file_name() to trace where it came from.
- what happens
- Once the bucket moves under Unity Catalog, the /mnt reads fail with file-not-found errors and input_file_name() isn't available.
- what to do
- Create an external location or volume for the bucket, change reads to volume paths, and use _metadata.file_name for the source file.
How Cohash helps
We help teams run Databricks migrations from the inventory to the day the Hive metastore goes off: workspace and catalog design, group and grant mapping, table waves, code fixes and side-by-side checks. See our Databricks services, the Unity Catalog migration case study, hire Databricks engineers for your own team, or start with a free assessment of what still depends on the Hive metastore.
What to take from this
- 01
New Databricks workspaces have had no Hive metastore since 8 October 2026. Existing ones still work, but new workspaces can't use the old catalog.
- 02
SYNC only re-registers external tables. Managed tables in the DBFS root need DEEP CLONE or CTAS, and views have to be recreated.
- 03
Most breakage is in the code around the tables: /mnt paths, input_file_name(), RDD calls and two-part names.
- 04
Change the default catalog last, and treat the 30 days after disabling the Hive metastore as your only rollback window.
01Do we have to migrate from the Hive metastore to Unity Catalog?
Existing workspaces aren't affected by the October 2026 change and still run. But new workspaces come up without a Hive metastore, Databricks calls it legacy, and access can be disabled for good. If you'll ever want a new workspace for these jobs, plan the move.
02What happened on October 8, 2026?
From that date Databricks creates new workspaces in every account without the Hive metastore, DBFS root and mounts, no-isolation shared clusters, or runtimes older than 13.3 LTS. Accounts created after December 18, 2025 never had those features.
03Does SYNC copy our data?
No. SYNC registers external Hive tables in Unity Catalog and leaves the files where they are. Managed tables stored in the DBFS root can't be synced, so they're copied with DEEP CLONE or CREATE TABLE AS SELECT.
04Do Hive metastore permissions carry over to Unity Catalog?
Plan as if they don't. Databricks' upgrade steps tell you to grant account-level users or groups access to the new tables, and copied tables are owned by whoever created them unless you set an owner. Map today's table ACLs first and re-apply them to account groups.
05Is UCX supported by Databricks?
UCX is a Databricks Labs project on GitHub, and Databricks says it isn't supported directly. Its last tagged release is v0.60.1 from October 2025. It's still useful for the assessment, group migration and code linting, but check its results yourself.
06What code breaks most often after the move?
File reads through /mnt or /dbfs paths, input_file_name(), RDD and Spark context calls on Standard compute, Hive UDFs, and two-part table names once the default catalog changes. Search for these before cutover.
07Can we undo disabling the Hive metastore?
Databricks says it becomes permanent after 30 days. From then on the workspace is Unity Catalog only and direct Hive metastore access can't be re-enabled, so if you need it back, turn it on before then.
Three questions, about a minute
Pick an answer and we'll tell you if it's right, and why it is or isn't. The score stays in this browser.
-
Q1SYNC skips a set of managed tables with a DBFS root status. What's the realistic way to get them into Unity Catalog?
show the answer
B. Copy them with DEEP CLONE or CTAS into Unity Catalog SYNC can't upgrade managed tables stored in workspace storage, and shallow clones from the Hive metastore aren't allowed. A deep clone or CTAS copies the data into Unity Catalog.
-
Q2Jobs refer to tables as sales.orders. You switch the workspace default catalog away from hive_metastore. What happens?
show the answer
A. Those names now resolve against the new default catalog, so jobs may read other tables or fail Two-part names resolve against the default catalog. That's why the default should change last, once no job relies on it meaning hive_metastore.
-
Q3Hive metastore access was disabled 20 days ago and a key report is now failing. What's your safest move?
show the answer
D. Turn Hive metastore access back on, fix the job, disable it again later Databricks makes the switch permanent only after 30 days. Inside that window, turning access back on while you fix the job is still an option.