HP will supply hardware and engineering expertise to help tune Cluster File Systems' Lustre software in a three-year agreement with the DOE's National Nuclear Security Administration (NNSA), which already has clusters of low-cost servers in place.
The Lustre file system was designed to tackle the problem of reading and writing data across thousands of servers and storage systems in a cluster.
"The customers for tightly coupled clusters are in high-performance computing," Wheeler said. "But already given the price performance of these systems, there is growing interest in the enterprise world to adopt and use this type of system."
Building clusters by linking thousands of relatively low-cost servers makes it possible to generate the type of computing power found in a more expensive supercomputing system. But managing the movement of data across the clustered servers and storage systems, which sometimes number in the thousands, has proved challenging.
The Lustre file system addresses some of these management concerns with its use of OBFS (object-based file system) technology, which is designed to speed up the way clustered server and storage systems read and write data.
The technology requires the use of intelligent disk drives that, for example, know when to back up a file or send its contents to another drive.
In addition, data would be spread across all of the hardware in a cluster instead of being locked into individual servers.
This means that changes made to data in one server are reflected immediately in the entire cluster, thus keeping the data uniform for all concurrent users.
Work on Lustre was started at Carnegie Mellon University but has been further developed by several organizations over the past three years. Cluster File Systems, HP, Seagate Technology LLC, and the NNSA have all contributed to the software, Wheeler said.
An early version of the file system, called Lustre Light, will roll out across 100 servers next month.
The organisations involved will then improve the technology over the next three years. The software will be released as open source.
HP, Cluster File Systems and the NNSA will work to add more failover and security features to the file system along with advanced management tools.
Eventually, companies could use the technology for large-scale data mining tasks or even for rendering animation.