bakhtiarzein - Fotolia

Ofcom raps BT over Openreach governance failures

Ofcom says it is disappointed to learn that as the result of an internal error, BT plans to let Openreach use the BT Enterprise Cloud to temporarily run some of its systems

Ofcom has rapped BT’s knuckles after it said it would have to run Openreach’s operational support systems (OSS) on its own BT Enterprise Cloud (BTEC), breaching section 5.44.2(a) of the undertakings to keep its infrastructure arm functionally separate from the rest of the business.

This comes in the wake of what the telco described as a “regrettable” lapse in its internal governance processes.

BT group director of regulatory affairs Mark Shurmer reported the upcoming breach to Ofcom in a letter dated 24 September 2015.

Shurmer said that for operational reasons BT wanted to temporarily run a small number of Openreach’s OSS on BTEC while it constructs a separate Openreach-only compute platform, scheduled for completion by 31 December 2015.

The five OSS in question are: ESB, BTB and Portal, which are key to Openreach’s View My Engineer service and have collectively outgrown the current platform; Service Flow, which enables Openreach to ensure compliance with its service-level agreements; and Or Smarts, it’s fault management system for Openreach next-generation access, fibre-to-the-cabinet (FTTC), fibre-to-the-premises (FTTP), Ethernet and optical spectrum products.

“Enhanced versions of these OSS have been erroneously developed to be run live on an existing computing platform called BT Enterprise Cloud which is shared across the whole of BT,” wrote Shurmer.

“In all five cases, as soon as these development errors were discovered all migration plans were halted in January.”

Level 2 separation

BT said it had discovered that three of the OSS – ESB, BTB and Portal – were wrongly categorised as being Level 2 separated, meaning they were virtually separated from BT but not physically, and therefore it would be allowable under the undertakings to move them to BTEC where they would still be Level 2 separated.

The original versions were hosted on shared virtual datacentres (VDCs) that were Level 2 compliant and had been shared across BT’s lines of business, although the OSS were hosted on a dedicated Openreach VDC, it claimed.

The other two, Service Flow and Or Smarts, were inappropriately targeted to run on BTEC “because they needed additional computing resource that could not be added to their existing platforms”.

BT argued there was nothing in the undertakings that forbade a physically separate system being moved back to Level 2 separation temporarily.

Lapse in governance

Shurmer said that all five of the affected OSS were critical elements when it came to enabling Openreach to meet its obligations to communications provider customers and consumers alike, and the high volume of transactions running through them meant that new platforms were urgently needed.

However, because of BT’s categorisation errors, alternative physically separated Openreach platforms had not been build and needed to be.

“The mistakes that resulted in this problem were not intentional but were the result of regrettable lapses in internal governance and errors made in categorising existing Openreach computer systems as to whether they could be implemented on shared hardware. Corrective action is being taken,” wrote Shurmer.

BT attempted to reassure the regulator that if allowed to run the five Openreach OSS on its own systems until the end of 2015, it would run them in virtualised computer environments to ensure both applications and data were not accessible to the rest of BT.

BT insisted there would be no adverse consequences for competition to either internet service providers using the Openreach network – such as Sky and TalkTalk – or their customers.

Not the first time

Furthermore, said BT, earlier in 2015 five other OSS were inadvertently deployed on BTEC in breach of the undertakings.

Two of those OSS had also been wrongly categorised, the other three were Openreach-only instances of existing OSS that had been used elsewhere in BT, and were put on BTEC in the “belief that these instances were not new OSS being designed specifically for Openreach”.

This breach was notified to Ofcom and has been described in the 2014/15 annual report of BT's equality of access board.

BT claimed that no inappropriate sharing of Openreach’s confidential customer or commercial information with the rest of BT occurred at the time.

Regulator responds

In its response to BT’s request, Ofcom said it was very disappointed at the circumstances, and reiterated that section 5.44.2(a) was very clear in requiring BT to keep all of Openreach’s systems physically separate from its own.

“We take these breaches very seriously and consider it important that BT commits to appropriate remedial action in relation to them,” said Jonathan Oxley, Ofcom group director of competition.

However, he said, as BT had self-reported the breaches and set out remedial actions it planned to take around both the construction of an Openreach-only platform, an audit of its systems estate, and improvements to internal governance, the regulator would not look to further punish BT.

Read more about Openreach

Additionally, because the affected OSS were essential to the smooth running of Openreach and that to stop all use of them for the next three months would seriously damage the market, BT should be permitted to run the OSS for the time-limited period.

“We do not on balance consider it would be appropriate for Ofcom to take enforcement action against BT in result of its proposal to run the five specified OSS on the BTEC until the end of December,” said Oxley.

“BT accepts that it made mistakes in developing these Openreach OSS with the expectation that they would go live on the shared BTEC platform rather than on an Openreach-only physically separated platform,” said Shurmer.

“BT is committed to moving these five OSS to physically separated hardware as soon as possible. Allowing these OSS to be run for a short term on shared hardware ... would ensure that both BT’s CP customers’ and end users’ experience is not compromised.”

However, even though BT self-reported the breaches, the major lapse in governance, coupled with recent accusations from rivals that BT is currently trying to present its best face to Ofcom as the regulator considers whether full structural separation of Openreach should happen, is likely to add fuel to the anti-BT fire.

Read more on Telecoms networks and broadband communications