Start a project
Menu
ServicesAll servicesCybersecurityCloud EngineeringApplication ServicesEnterprise PlatformsData & AnalyticsAI & Machine LearningInternet of ThingsMobilize Talent
PlatformsAll platformsIdentity and accessPrivileged access and secretsERP, HCM and financeCRM, ITSM and automationHealthcareInsurance and bankingData and integrationCloud and AIIndustrial, utilities and mainframeSkills matrix
Case studiesAll case studiesIdentity and accessPrivileged accessSecurity operationsCloud and infrastructureApplicationsERP and HCMCRM, ITSM and automationIndustry platformsData and analyticsAI and agents
IndustriesAll industriesEnergy, oil & gasHealthcare payersBanking & financial servicesInsurancePublic sectorManufacturingLocations
InsightsField notesLearning tracks
CompanyAbout CohashPortfolioSkills matrixCareersTrust and securityContact
Start a project
delivered for clients · names withheldmigration Data and analytics

Databricks Hive metastore to Unity Catalog

Cohash upgrades Databricks workspaces from the Hive metastore to Unity Catalog. We run the UCX assessment, move workspace-local groups to account groups fed by your identity provider, set up storage credentials and external locations, and upgrade tables with SYNC or DEEP CLONE. Then we fix the notebooks and jobs that still use mounts or two-part table names, and turn off legacy access once nothing depends on it.

Plan this move with us See how we did it
  1. 01UCX assessment
  2. 02Identity and groups
  3. 03Storage, catalogs and grants
  4. 04Table upgrade
  5. 05Code, jobs and compute
  6. 06Disable legacy access
case-study / hive-metastore-to-unity-catalog delivered
beforeHive metastore, DBFS root and mounts afterDatabricks Unity Catalog
01 · IDENTITY AND CODE 02 · COMPUTE AND CODE 03 · GOVERNANCE 04 · STORAGE Entra ID or Okta: SCIM to the Databricks account Entra ID or Okta SCIM to the Databricksaccount Account groups: service principals for jobs Account groups service principals forjobs Terraform: catalogs · grants · credentials Terraform catalogs · grants ·credentials Notebooks and jobs: three-part names · 13.3 LTS+ Notebooks and jobs three-part names ·13.3 LTS+ Unity Catalog metastore: catalogs by domain · privileges Unity Catalogmetastore catalogs by domain ·privileges Hive metastore federation: bridge for tables not moved yet Hive metastorefederation bridge for tables notmoved yet External locations: storage credentials External locations storage credentials Volumes: replace /mnt paths Volumes replace /mnt paths
case file

The story

Told without the client's name or anything that could identify them.

what happened

A retailer's data platform team ran several Databricks workspaces, each with its own Hive metastore and its own copy of the same groups. Jobs read from /mnt paths whose credentials sat in cluster configs nobody owned, and a new business-unit workspace had failed because the provisioning script expected DBFS mounts. We ran the UCX assessment to build the backlog, synced groups to the account from the identity provider, and wrote catalogs and grants in Terraform. Tables moved with SYNC or DEEP CLONE, notebooks and jobs came off mounts and two-part names, and legacy access was switched off once nothing depended on it.

why it was urgent

Existing workspaces keep running. The change is for new ones: from 30 September 2026, Databricks provisions new workspaces without the Hive metastore, DBFS root and mounts, no-isolation shared clusters or runtimes older than 13.3 LTS. Accounts created after 18 December 2025 get no legacy features at all, so a team that spins up a fresh workspace can't carry the old patterns into it.

problem statement
  • openEach workspace has its own Hive metastore and its own copy of the same groups.
  • recurringJobs read from /mnt paths, and the credentials behind them live in a cluster config nobody owns.
  • blockedThe provisioning script for a new business-unit workspace failed, because it assumes DBFS mounts exist.
  • openAuditors ask for table-level lineage and access history, and the best anyone can offer is a spreadsheet.
architecture

The estate, before and after

before
before

Before: a Hive metastore and local groups in every workspace

Each workspace kept its own groups and Hive metastore, and jobs reached storage through DBFS mounts whose credentials lived on clusters. Access rules differed between workspaces and nobody owned the mount credentials.

Before: a Hive metastore and local groups in every workspace Each workspace kept its own groups and Hive metastore, and jobs reached storage through DBFS mounts whose credentials lived on clusters. Access rules differed between workspaces and nobody owned the mount credentials. 01 · IDENTITY 02 · COMPUTE AND CODE 03 · METASTORE 04 · STORAGE table ACLs run on two-part names managed tables external tables mount credentials Workspace-local groups copied in each workspace Notebooks and jobs two-part names · /mntpaths Shared clusters credentials in clusterconfig Hive metastore perworkspace table ACLs DBFS root managed tables Cloud storage viamounts /mnt paths
Read the diagram as text
  1. Workspace-local groups → Hive metastore per workspace: table ACLs
  2. Notebooks and jobs → Shared clusters: run on
  3. Notebooks and jobs → Hive metastore per workspace: two-part names
  4. Hive metastore per workspace → DBFS root: managed tables
  5. Hive metastore per workspace → Cloud storage via mounts: external tables
  6. Shared clusters → Cloud storage via mounts: mount credentials
after
after

After: Unity Catalog governs every workspace from the account

Groups come from the identity provider at account level, and Unity Catalog holds the grants, written in Terraform. Jobs use three-part names and volumes, and tables not moved yet are read through Hive metastore federation.

After: Unity Catalog governs every workspace from the account Groups come from the identity provider at account level, and Unity Catalog holds the grants, written in Terraform. Jobs use three-part names and volumes, and tables not moved yet are read through Hive metastore federation. 01 · IDENTITY AND CODE 02 · COMPUTE AND CODE 03 · GOVERNANCE 04 · STORAGE SCIM privileges grants as code three-part names federated tables governed access governed paths Ex Entra ID or Okta SCIM to theDatabricks account Account groups service principals forjobs Terraform catalogs · grants ·credentials Notebooks and jobs three-part names · 13.3LTS+ Unity Catalogmetastore catalogs by domain ·privileges Hive metastorefederation bridge for tables notmoved yet External locations storage credentials Volumes replace /mnt paths
Read the diagram as text
  1. Entra ID or Okta → Account groups: SCIM
  2. Account groups → Unity Catalog metastore: privileges
  3. Terraform → Unity Catalog metastore: grants as code
  4. Notebooks and jobs → Unity Catalog metastore: three-part names
  5. Hive metastore federation → Unity Catalog metastore: federated tables
  6. Unity Catalog metastore → External locations: governed access
  7. Unity Catalog metastore → Volumes: governed paths
results

What the client got

What changed for the client once the work was done. Described, not scored: we don't publish client figures.

  1. result 01

    One set of account groups fed by the identity provider

    SCIM now syncs users and groups from Entra ID or Okta to the Databricks account. We deleted the workspace-local copies.

  2. result 02

    Grants are code, not clicks

    We put catalogs, schemas, storage credentials and Unity Catalog privileges in Terraform, and nothing changes without review.

  3. result 03

    Jobs no longer depend on mounts

    We moved notebooks and jobs to volumes or external locations and three-part table names, running on Unity Catalog access modes with 13.3 LTS or later.

  4. result 04

    Data owners approved the access changes

    Some legacy table ACLs and ANY FILE grants had no Unity Catalog equivalent. Data owners signed off the resulting access changes beforehand, so no user found out by hitting an error.

  5. result 05

    New workspaces provision without legacy features

    We updated provisioning so new workspaces come up with the Hive metastore and DBFS mounts disabled, and the metastore assigns itself in each region.

what can go wrong

Risks we plan for

  1. UCX is a Labs project

    UCX comes from Databricks Labs without formal support or SLAs. It does most of the inventory and table work well, but plans can't assume it covers everything.

  2. Permission drift

    Legacy table ACLs, cluster-level credentials and ANY FILE grants don't map one to one to Unity Catalog privileges. Some users will lose access they shouldn't have had, and that needs to be agreed, not discovered.

  3. Jobs breaking quietly

    A job that writes to a mount can keep writing to the old location after the table moves. We freeze writes per table during its cut-over and compare row counts on both sides.

  4. Managed tables that can't clone

    DEEP CLONE won't copy CSV, JSON or Avro managed tables in the DBFS root until they're converted. Big ones need a maintenance window.

ai in this move

Where AI helps, and where it needs guardrails

Databricks governs AI through Unity Catalog too. Agent Bricks and Unity AI Gateway (the piece that adds spend caps and cost attribution for models, agents and MCP servers) both depend on data being registered and permissioned there. We let coding agents suggest notebook path and table-name rewrites as pull requests during the migration, and an engineer reviews every one. Agents never hold metastore admin.

forward deployed engineers →
you keep

What you're left with

We write these for your setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.

  • plan01

    UCX assessment results with a wave plan per workspace

    yours to keep
  • inventory02

    Catalog, schema, storage credential and grant definitions in Terraform

    yours to keep
  • document03

    Group migration log and access-change list approved by data owners

    yours to keep
  • document04

    Updated notebooks and jobs, with before-and-after row counts per table

    yours to keep
  • document05

    Provisioning automation that works without legacy features

    yours to keep
what's involved

Platforms and services on this move

platforms in play
Microsoft Entra IDOktaTerraform
services

Services involved

talent and delivery · databricks unity catalog

Certified engineers, wherever you need them.

Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.

Onsite

Engineers at your sites in the US

Nearshore

Overlapping hours with US teams

Offshore

Overnight progress on your backlog

All time zones

Coverage handed over across time zones

24-hour coverage · US Central time
OnsiteUS business hours
Nearshoreoverlapping US hours
Offshoreovernight progress
Combinedhandover, not hand-off
01Do we have to migrate existing workspaces?

Not by a date. Databricks says existing workspaces aren't affected by the 30 September 2026 change. But any new workspace comes without legacy features, and newer platform features assume Unity Catalog, so the gap grows the longer you wait.

02How much downtime does a table upgrade need?

External tables upgraded with SYNC need very little. Managed tables in the DBFS root are copied, so time depends on size and on how long writers can pause. We schedule those by table and owner.

03Can we keep some tables in the Hive metastore for now?

Yes. Hive metastore federation lets Unity Catalog govern tables still in Hive while you finish. It's a bridge, so don't plan to live on it.

04Do you also handle the identity side?

Yes. Group sync from Entra ID or Okta, service principals for jobs and access reviews on catalogs are part of the work, and they're often where most of the decisions sit.

related

Often on the same roadmap

All case studies
migration Identity and access

On-premises Active Directory to cloud-first Microsoft Entra ID

We shrink on-premises Active Directory one application at a time, moving groups, users and app authentication to Microsoft Entra ID as each app is ready.

read the case study
migration Identity and access

AD FS federation to Entra ID authentication

Converting federated domains from AD FS to Entra ID cloud authentication, moving relying parties app by app and retiring the AD FS farm.

read the case study
AI rollout AI and agents

AI pilot to production with forward deployed engineers

Taking an AI pilot into production with forward deployed engineers who work with the business team, build evals, connect real systems and hand over cleanly.

read the case study
sources 4 sources linked last checked 2026-10-20 content fingerprint 45da3070
  1. 01Microsoft Learn: Migrate your account to UC-only workspaces (Azure Databricks)learn.microsoft.com ↗
  2. 02Databricks docs: Use the UCX utilities to upgrade your workspace to Unity Catalogdocs.databricks.com ↗
  3. 03Databricks: AI governance with Unity AI Gatewaydatabricks.com ↗
  4. 04Databricks: What's new with Unity Catalog at Data + AI Summit 2026databricks.com ↗
Reviewed by the Cohash team Product names are trademarks of their owners. how we handle facts and data →
next step

Planning this move?

A 30-minute working session with a senior engineer. No slides.

Book a working session See the case studies
useful to bring
  • The platform and the version you're on
  • Where you need to get to, and by when
  • What's custom: rules, plugins, integrations
  • Any audit finding or deadline driving it