NYSE improves data management with datawarehousing

The New York Stock Exchange (NYSE), which trades shares in Europe, the US and Asia, is using off-the-shelf technology to cut the time taken to access business-critical data on its network from as long as 26 hours to just over two minutes.

NYSE, which runs the Paris stock exchange Euronext, has replaced an Oracle IO relational database with a datawarehousing appliance from Netezza, allowing it to conduct rapid searches of 650 terrabytes of data.

The move forms part of a wider trend for financial services firms to move away from complex legacy technology towards off-the-shelf products, which gives businesses greater flexibility to meet changing market demands.

The NYSE technology - which was installed in December - collects, stores and manages trading data, collected by its New York exchange, its Paris operation Euronext and its Asian online exchange Archipelago.

The new appliances combine a database search engine and storage in the same unit and use multiple processors in parallel to find the data quickly.

Brian Clark, chief architect at NYSE, said, "It was painful to put data in and get it out with the previous system."

The previous system had to send large quanities of data from storage devices to the database over the network, and trawl through large amounts of irrelevant information to complete searches.

Clark said the appliances, which carry out server, storage and database management activities, will pay for themselves on the storage benefits alone. "The financial analysis we did was just on storage volume and we were able to get a return on investment in one year," he said.

He said the Oracle system used up too much storage capacity because IT staff had to build indexes that were used to help search for information, which meant some data was repeated.

Ralph Silva, analyst at TowerGroup, said if the exchange can source data faster it can cut the costs associated with answering questions from the regulators. "If you have good data [management] you can answer the regulator's questions in seconds without the need to put three or four people on the job," he said.

Silva said that NYSE will also be able to sell more data. "The big market for data currently is from the algorithm writers who use it to test algorithms against [historical marker conditions] to see if they work."