The need emerged to use operational data not just for dashboards and incident notifications, but to identify patterns over longer periods of time. The existing environment, based on Splunk, reached its limits as data volumes grew and analyses became more complex. Increasingly, the focus shifted to timeseries data from machines, including status information, events, performance metrics and failure data, which need to be analyzed in combination. This marked the point where a platform was required that better fits these types of challenges.
Choosing Databricks created the opportunity to collect operational data centrally and organize it in a fundamentally different way. At Itility, we bring extensive Databricks expertise, which we applied throughout the migration. This was not just about moving data, but about redesigning pipelines, storage and data flows so they remain reusable and manageable over time. One of the most challenging aspects was migrating existing data streams while systems remained in production. At the same time, new standards were introduced to keep future growth under control.
By working with a clear step by step approach and making deliberate choices, the project maintained momentum without compromising stability. The combination of platform knowledge, experience with complex data environments and a pragmatic delivery approach made it possible to continue building in a controlled manner.
With the new data foundation in place, the organization now has a solid starting point for further development. In practice, this means that:
• Operational data is centrally available and scales with the size of the installations
• Quality, governance and security are structurally embedded
• Analyses across larger data volumes and longer time horizons are possible
In addition, the move to Databricks has resulted in a more efficient data platform setup. Based on initial experience, this translates into a significant cost reduction compared to the previous solution, estimated at around forty percent.
This phase lays the foundation for next steps, where operational data is increasingly used for insight, optimization and continued development.
During the transition to the new data platform, the focus was not only on technology, but on how architecture and execution reinforce each other. In an environment where scale and reliability are constantly under pressure, focus, experience and a clear migration path make the difference. The result is a data foundation that was not designed as an end state, but as a working foundation that grows alongside day-to-day operations.