Databricks predicts AI tipping point as ANZ firms fix data issues

Solving the data silo problem

Craig Wiley, senior director of product management at Databricks, suggested that while every industry stands to benefit from AI and “the way we work is changing,” many companies are struggling to take advantage of the technology.

A common hurdle is data locked in multiple, often proprietary, systems with fragmented security and governance frameworks. This prevents employees from finding the data they need and hampers automation.

Databricks addresses this by consolidating data into low-cost storage using open formats, overlaid with a unified, granular governance approach. The platform also supports composable agents, simplifying the deployment of prebuilt agents, custom organisational agents, and classical models used for tasks like churn prediction.

Several Databricks customers shared their experiences at the conference.

Telstra: From crimes against data to AI-first

Telstra has set a goal to become an AI-first organisation. However, its data and AI executive Dayle Stevens admitted that the telco’s long history, technical debt, and past “crimes against data” mean they must first correct their foundations.

Progress includes separating compute and storage to reduce costs and improve scalability, establishing data lineage, and enabling data sharing across multiple platforms and clouds.

Stevens highlighted Telstra’s commitment to responsible AI. The company was an early adopter of the federal government’s AI ethics principles and the first Australian company to join Unesco’s Business Council to promote ethical AI implementation.

“Every AI model at Telstra goes through an oversight committee,” she said, with objectives focused on privacy protection and preventing AI from entering “creepy states.”

Culturally, Stevens noted a tension between early tech adoption and maintaining the trust of customers and employees. The priority is retaining that trust through transparency, data quality, and governance. To support this, Telstra established a data and AI academy, with 20,000 employees having already completed at least one course.

Fonterra: Data-driven dairy

Helius Guimaraes, chief data and AI officer at New Zealand dairy co-operative Fonterra, explained how the organisation uses Databricks to underpin its transformation.

With much of its historical data held in legacy systems, Fonterra adopted a data products approach. These reusable products are combined with AI-powered self-service capabilities to deliver greater efficiency than traditional reporting. Fonterra now has more than 6,000 employees using generative AI functions within various applications.

Alinta Energy: Retiring fragile models

Alinta Energy’s spot trading team previously relied on a fragile machine learning (ML) model developed over a decade ago in the Matlab programming language and analytics environment on a desktop PC.

Andrew Davis, data and AI platform delivery manager, explained that the old model “took days for the business to train,” meaning it was only updated annually. By moving to Databricks, ingestion and processing tasks that took hours are now completed in minutes.

Andrew Gorkic, principal AI consultant at Fujitsu Australia (which services Alinta), noted that Databricks also offers ML operations support. This automates “challenger versus champion” testing and warns when model output drifts from reality.

The shift has relieved stress for the trading team, allowing them to focus on market trends. The new model is both faster and more accurate, as it can be retrained every month or two depending on weather and market conditions.

Prospa: Mapping fraud with graphs

Small and medium-enterprise (SME) lender Prospa uses Databricks to approve small loans in under an hour. However, mapping complex relationships between businesses and individuals required moving beyond conventional databases.

Principal AI scientist Jin Foo explained that adopting Neo4j’s graph database allowed Prospa to better visualise risk. For example, if ten companies with the same owner apply for loans simultaneously, or if a fraud ring uses dozens of shell companies, the graph database reveals these connections.

Prospa plans to implement retrieval-augmented generation by connecting the graph database to a large language model, allowing operations teams to query data using natural language.