How TCS is supporting Australia’s electricity market settlement

Tata Consultancy Services (TCS) played a major role in the design and implementation of the Australian Energy Market Operator’s (AEMO) new system for settling the National Electricity Market (NEM) every five minutes.

AEMO operates Australia’s electricity and gas markets. In the case of electricity, that includes the NEM which, despite its name, connects electricity producers and electricity retailers in the southern and eastern states.

Until late 2021, NEM settlements occurred in 30-minute blocks even though producers dispatched electricity in five-minute blocks. One of the effects of this mismatch was disorderly bidding, TCS’s ANZ manufacturing and utilities business unit head, RP Sathpathy, told Computer Weekly.

Disorderly bidding is the term used for the situation where network constraints – such as the capacity of interconnectors – prevent NEM from accepting the lowest bids that would put the required amount of electricity into the market, and then producers rebid capacity at different prices or even declare themselves unavailable.

On 1 October 2021, AEMO switched to five-minute settlement, which aligns dispatch and settlement, presents better price signals, and provides an incentive to faster responding producers, such as those operating batteries, Sathpathy said, adding: “This is the whole aim of five-minute settlement.”

The result is a more reliable power system and increased investment in fast-response technology. This is expected to gradually lower wholesale prices, which will be passed on to consumers through competition in the retail market.

The main reason for 30-minute settlement had been AEMO’s limited capacity for handling the data from the meters that measure how much electricity providers are suppling to the grid. Going from 30 minutes to five minutes theoretically meant six times more data, and even more was expected as additional providers join the market.

So, a completely new system was needed to run the NEM efficiently and transparently. AEMO searched the market for an off-the-shelf product but found nothing suitable as no other energy market was operating on five-minute settlement.

TCS, which was not an incumbent supplier to AEMO, responded to the resulting request for proposal and worked with AEMO to design a system that could meet the current requirements and accommodate expected changes.

TCS already understood the Australian market, but realised a completely new approach was required, making the project “an interesting challenge”, said Sathpathy.

The company brought in industry experts and an engineering team from Microsoft, and in around three and a half months, the 25 people involved came up with a design that used Azure, Kubernetes, Cosmos DB, Azure Data Lake, Databricks and other technologies to handle the required volume of data while being easy to manage.

Having AEMO subject matter experts in the same room as the rest of the design team was important, and “there was learning for everyone”, Sathpathy said. Spending enough time on the early stages of the project was also an important contributor to the project’s success, he added.

Agile was a new way of working for AEMO staff, so TCS ran workshops for them. “It was a culture change, it was a mindset change for all of us,” said Sathpathy.

Using Microsoft Azure technologies as much as possible helped keep things relatively simple, as they support each other, but some custom code was needed due to the market-specific nature of the project.

A proof of concept demonstrated the soundness of the design, which was then fully implemented and put into action. At the peak, some 250 people – 110 from TCS – were working on the project.

“There were lots of unknowns,” Sathpathy observed. For example, this was the first use of Cosmos DB in Australia.

Other practices used in the project include DevOps and automated testing, and the system incorporates application programming interfaces (APIs) to connect to market participants’ systems. It also makes provision for further automation, potentially including the application of machine learning.

Where 30-minute settlement involved around seven million meter reads a day, the switch to five-minute settlement resulted in an almost immediate jump to some 20 million a day. The reason this was an approximately threefold rather than sixfold increase was that not all electricity providers were equipped with smart meters, let alone those that support five-minute intervals.

This is expected to increase to two trillion reads a day by 2028, by which time all supplies will be smart metered at five-minute intervals and there will be many more market participants. Consequently, the system is designed to scale automatically as the data volume grows. A profiling and allocating engine monitors the data ingestion from the meters and creates the required profiles for the platform, so as more data is ingested, more resources are automatically allocated to process it.

Because the system is completely microservices based, it could be easily, quickly and cheaply adapted for the gas market if required, according to Sathpathy.

It could also be applied in other countries. Net zero targets mean there will be lots of distributed sources of electricity, he said, and even if market operators decide they do not need five-minute settlement they will need something similar to accommodate the operators of smaller generators and batteries that are unable to bid to supply in 30-minute blocks.

“The system is running like a well-oiled machine from day one,” said Sathpathy, noting that the market has never stopped, no settlement has failed, and no priority one or two tickets have been raised.

This represents “a great team outcome” for the largest market transformation by AEMO in 20 years, he said, noting that the trust between the people involved was a key success factor.