Newcastle University finds data integration advantage with Talend

Newcastle University is reaping the benefits of a data integration programme based on open source technology, Talend Integration Suite, a weapon in this user’s arsenal.

Newcastle University is reaping the benefits of a data integration programme based on open source technology, Talend Integration Suite.

Among the IT services the data integration project has made possible are the personalised timetables Newcastle students now have.

“It is all about improving the student experience,” said Cal Racey, middleware team manager at the university, which has 17,000 students.

The government’s December 2010 decision to raise the yearly fee to £9,000 has been a “game changer," Racey said. "It has sharpened up minds. The change in the funding model really does underline how we are now in a very competitive market.” The new funding pressures also make it important to support researchers around the Research Excellence Framework, he said. “There are enormous amounts of reporting to be done there. It is a challenge to ship data to the right systems.”

Newcastle began to address its data integration problem in 2008. The university realised it need a more efficient data management tool to cope with demands for faster response times and more data about users. The major challenge was being able to input data relating to its staff members, students, the library and the university as a whole whilst ensuring data consistency.

It won funding to carry out a national exemplar trial project for the Joint Information Systems Committee (JISC), which supports networking and IT services in UK higher education. Newcastle chose Talend as the institutional data management part of that project, ID-MAPs – Institutional Data Management for Personalisation and Syndication, Racey said.

“Previously, we had self-written Perl and Shell scripts," he said. "That didn’t scale and was not supportable. If there was a problem with what data had gone where, we couldn’t see easily when things gone wrong, especially if someone had gone on holiday. The problem had got so large that we had to address it in order to offer other IT services.”

The programme is now a fully supported service internally, not funded by JISC. “The real value is now showing, with 30 projects and 60 data feeds into different systems,” Racey said.

Why did Racey and his team select Talend? They also looked at Pentaho, an open source supplier of business intelligence and at an IBM WebSphere add on.

“Talend Integration Suite met our requirements on scalability, being able to fulfil real-time capabilities to ensure data is kept up-to-date and accurate, as well as providing large volume handling capabilities. [And] being open source, costs can be kept extremely low,” said Richard James, systems architect at Newcastle.

“We do like open source,” Racey said. “But we’ve also got a large SAP install for our ERP, and it’s a question of the best tool for the job. In this space, the open source nature [of Talend] is important because it is data integration and it sits in the middle of a whole load of systems. If we wanted to swap it out, it would be a huge project. We are happy having Talend to support us, but if needs be, we could run with it ourselves.

“What we want to avoid in contract negotiations is what can happen to universities – the gun put to the head with data integration suites. That is less likely to happen with Talend.” He confirmed that the university is also using Talend’s data quality tool and contemplating the vendor’s service oriented architecture enterprise service bus (SOA ESB) with a view to real time delivery of data, where it is currently nightly or hourly.

Read more on Database management