Can storage systems be number crunchers too?

A research has been project launched to find out whether storage systems can be used for crunching numbers.

A research has been project launched to find out whether storage systems can be used for crunching numbers.

Silicon Graphics (SGI) is helping to fund a year-long Pacific Northwest National Laboratory (PNNL) project to write software that will let storage devices do calculations directly, rather than simply serving up files to other computers for processing.

The software, which is based on the open-source Lustre file system, could eventually greatly improve the processing capabilities of large-scale computing clusters, researchers said.

In most file systems data is broken up for storage on the hard drive and must be recombined before it can be processed, said Scott Studham, manager of computer operations with the PNNL's Molecular Science Computing Facility.

Because of the structure of Lustre, however, recombination is not necessary. PNNL researchers are developing software that takes advantage of this fact to harness the processors on storage arrays so that they can perform calculations whenever they are otherwise unoccupied, a process Studham calls "active storage".

The researchers will use this software to speed up proteomic research being done at PNNL by off-loading certain calculations, called Fourier Transforms, to the file system, Studham said.

Active storage could eventually be useful in any area that involves data mining, including the chemical industry, law enforcement, marketing databases, and the insurance industry, said Phil Schwan, chief executive of the company that develops Lustre, Cluster File Systems.

Existing data mining applications will need some modification in order to take advantage of active storage, he said, although he does not anticipate that this will be extremely hard to do. "I don't think there's a lot of very special application programming that needs to be done," he said.

A larger challenge is the fact that Lustre itself is not widely used outside of the high performance computing space. Schwan said his company is working to develop a broader set of Lustre users.

Studham declined to say how much money SGI is contributing towards the active storage project, but it will be enough to fund at least two full-time software developers, he said.

Robert McMillan writes for IDG News Service

Read more on Business applications