AVEVA PI data to cloud analytics on Databricks or Fabric
Cohash gets AVEVA PI System data into Databricks or Microsoft Fabric without opening the plant network. Interface gaps and messy tag naming get fixed first. Then we agree an asset model with process engineers, replicate to a DMZ or AVEVA CONNECT, and land time-series data with its context in Delta tables or a Fabric Eventhouse. Every feed runs outbound, under its own identity.
- 01Historian health check
- 02Asset model and naming
- 03Outbound path design
- 04Land and model in the target
- 05Backfill, reconcile and hand over
The story
Told without the client's name or anything that could identify them.
A chemical manufacturer's data scientists kept asking for direct connections into plant historians, and cloud teams copied PI data with scripts that broke whenever a tag was renamed. Every site named tags differently, and a PI subscription renewal was coming without a clear list of the components analytics needed. We read interface logs, buffering settings and archive gaps by site and fixed the collection problems first, then agreed a tag naming and asset hierarchy standard with process engineers. Data now leaves the plant outbound through a DMZ PI replica and lands in Delta tables under Unity Catalog with asset context attached.
why it was urgentOn 3 April 2024 AVEVA and Databricks announced a strategic collaboration, with data from AVEVA CONNECT shared through Delta Sharing. AVEVA also says PI System is moving to subscription licensing. That change forces a decision on scope. Microsoft Fabric is the other common target: its Real-Time Intelligence keeps time-based data in Eventhouses that you query with KQL.
- openData scientists ask for a direct connection into the plant historian.
- recurringEvery site names tags differently, so any cross-site analysis starts in a spreadsheet.
- blockedInterface gaps show up weeks later as flat lines in reports.
- openThe PI subscription renewal is close, and nobody can say which components the analytics actually use.
- recurringCloud teams copy PI data with scripts that break when a tag is renamed.
The estate, before and after
Before: site historians and fragile copy scripts
Each site's PI Data Archive collected through interfaces with buffering gaps and its own tag names. Cloud scripts and spreadsheet exports pulled data out, and broke whenever a tag changed.
After: outbound PI data in Databricks with asset context
Data flows one way. Site PI archives replicate to a PI server in the industrial DMZ, and that server pushes outbound to Databricks, where Delta tables under Unity Catalog join time-series values to the agreed asset model for analysts and quality checks.
How we run it, phase by phase.
5 phases take you from AVEVA PI System site historians to Databricks or Microsoft Fabric. Each one ends with something you can check before the next starts. As you scroll, the blocks show the estate moving across.
- phase 01
Historian health check
Go through interface logs, buffering settings, stale tag counts and archive gaps site by site. Fix the collection problems now, or analytics inherits them.
- phase 02
Asset model and naming
Agree a tag naming and asset hierarchy standard with process engineers, starting with the units and equipment analysts ask about most.
- phase 03
Outbound path design
Pick DMZ PI replication with an outbound feed or AVEVA CONNECT with Delta Sharing. Either works, as long as nothing in the cloud opens a connection into the control network.
- phase 04
Land and model in the target
Land time-series values and asset context in Delta tables under Unity Catalog, or in a Fabric Eventhouse with OneLake availability. Sort out units and time zones once, when the data lands.
- phase 05
Backfill, reconcile and hand over
Backfill history tag by tag, reconcile counts and values against PI, then hand over data quality checks to the site and data teams.
What the client got
What changed for the client once the work was done. Described, not scored: we don't publish client figures.
- result 01
Nothing in the cloud connects into the control network
PI data reaches the cloud through an outbound feed from a DMZ replica. No inbound firewall rules to the control network were needed.
- result 02
Tags follow one naming and asset standard
Process engineers agreed a tag naming and asset hierarchy standard, starting with the units and equipment analysts ask about most, so cross-site analysis no longer starts in a spreadsheet.
- result 03
Collection gaps are fixed at the source
Interface, buffering and stale tag problems were fixed site by site before analytics leaned on the data. Otherwise they'd have turned up later as flat lines on someone's chart.
- result 04
Analysts get time-series data with asset context
Analysts get the reading and the asset it came from in the same Delta table, under Unity Catalog. Units, time zones and sampling rules are written down once.
- result 05
Site and data teams own the quality checks
History was backfilled and reconciled against PI tag by tag, and the data quality checks are handed over to named site and data teams.
Risks we plan for
-
Opening a path into OT
A design that needs inbound firewall rules to the control network will stall with the OT team, and it should. All feeds run outbound from the DMZ or CONNECT.
-
Losing asset context
Raw tag streams without the asset hierarchy are hard to use. Context travels with the data, or analysts rebuild it badly in each notebook.
-
Compression and interpolation surprises
PI stores compressed values. Treat archive values as evenly sampled and your averages come out wrong. We document the sampling rules in the target.
-
Licence scope after the subscription change
Subscription licensing changes what each component costs. Finish the inventory of interfaces, CONNECT services and sites before renewal talks start.
Where AI helps, and where it needs guardrails
Fabric Real-Time Intelligence has data agents that reason over live and historical Eventhouse data, and Databricks Agent Bricks runs agents on governed tables. We point them at the curated cloud copy only. Any MCP server or agent that answers plant questions gets read-only access to cloud tables. Never write access to PI, and never a path into the control network.
forward deployed engineers →What you're left with
We write these for your setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.
- document01
Historian health check with collection gaps and causes by site
yours to keep - guide02
Tag naming and asset hierarchy standard agreed with process engineers
yours to keep - design03
Outbound data path design through the DMZ or AVEVA CONNECT
yours to keep - design04
Time-series and asset model in Databricks or Fabric
yours to keep - report05
Backfill and reconciliation report per site
yours to keep - document06
Data quality checks owned by site and data teams
yours to keep
Platforms and services on this move
Learn the background
Terms from this page in plain English. If you want the why, the field note behind it is right there.
- 01Rules
- The JavaScript snippets Auth0 used to run at login. Someone would write one to add a role to the token or block a sign-in, and each one ran after the last.in context →
- 02Unity Catalog
- Databricks' shared catalog for tables, files, models and permissions. Each region gets one metastore, and every workspace attached to it sees the same objects.in context →
- 03OneLake
- The one data lake each Fabric tenant gets, and you can't create a second. It sits on Azure Data Lake Storage and keeps tables as Delta Parquet or Iceberg.in context →
Certified engineers, wherever you need them.
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Onsite
Engineers at your sites in the US
Nearshore
Overlapping hours with US teams
Offshore
Overnight progress on your backlog
All time zones
Coverage handed over across time zones
01Should we use AVEVA CONNECT or build our own feed from PI?
It depends on your licence, how many sites you have and whether you want AVEVA to run the cloud side. CONNECT with Delta Sharing suits Databricks users. A DMZ replica with your own outbound feed gives more control. We cost both against your sites.
02Databricks or Microsoft Fabric for plant data?
Use whatever your data team already runs. If they build models on Delta tables, that's Databricks. If they live in Power BI and want KQL over streaming data, it's Fabric. PI data works with either.
03Do we have to clean up tag names first?
Not for every tag. Start with the equipment and units your first analytics use cases need, and extend the standard site by site.
04How do you keep the plant network safe?
Nothing in the cloud connects into the control network, ever. Data leaves on an outbound path through a DMZ copy or CONNECT, under a service identity your OT security team has reviewed.
Often on the same roadmap
AI pilot to production with forward deployed engineers
Taking an AI pilot into production with forward deployed engineers who work with the business team, build evals, connect real systems and hand over cleanly.
read the case study
Apache Airflow 2 to Airflow 3
Upgrading self-managed Apache Airflow 2, now past end of life, to Airflow 3, and fixing removed features, auth and scheduling changes before DAGs move.
read the case studyNasdaq Calypso upgrade to V19
Upgrading Nasdaq Calypso to V19, with server consolidation, properties moved to the database, the WebUI and deployment changes tested on your own trades.
read the case studyb67a6994 - 01Book a 30-minute working sessionsenior engineer
- 02Talk to an engineer about your setupreply within 24 hours
- 03See how similar projects wentcase studies
Planning this move?
A 30-minute working session with a senior engineer. No slides.
- The platform and the version you're on
- Where you need to get to, and by when
- What's custom: rules, plugins, integrations
- Any audit finding or deadline driving it