Google launches Parallelstore file storage at cloud AI training

Saturate server bandwidth

“This efficient data delivery maximises goodput to GPUs [graphics processing units] and TPUs [tensor processing units], a critical factor for optimising AI workload costs,” said GCP product director Barak Epstein in a blog post. “Parallelstore can also provide continuous read/write access to thousands of VMs [virtual machines], GPUs and TPUs, satisfying modest-to-massive AI and high-performance computing workload requirements.” 

He added that for the maximum Parallelstore deployment of 100TB (terabytes), throughput can scale to around 115GBps, three million read IOPS, one million write IOPS, and a minimum latency of near 0.3 milliseconds.

“This means that Parallelstore is also a good platform for small files and random, distributed access across a large number of clients,” said Epstein.

According to Epstein, AI model training times can be speeded up by nearly four times compared to other machine learning data loaders.

GCP’s idea is that customers first put their data in Google Cloud Storage, which can be used for all use cases on GCP and in software-as-a-service applications via virtual machines. That part of the process would allow the customer to select data suited to AI processing via Parallelstore from among all its data. To help here, GCP offers its Storage Insights Dataset service, part of its Gemini AI offer, to help customers assess their data.

Once data is selected as training data, its transfer to Parallelstore can take place at 20GBps. If files are small – less than 32MB, for example – it’s possible to achieve a transfer rate of 5,000 files per second.

Beyond the AI training use cases targeted by GCP, Parallelstore will also be accessible to Kubernetes clusters – such as via GCP’s Google Container Engine (GKE) – through dedicated CSI drivers. In practice, administrators will be able to manage the Parallelstore volume like any other storage attached to GKE.

DAOS is an open source effort object storage system that decouples the data and control planes while also segregating I/O metadata and indexing workloads from bulk storage.

DAOS stores metadata on fast, persistent memory and bulk data on non-volatile memory express (NVMe) solid-state drives (SSDs). According to Intel, DAOS read/write I/O performance scales almost linearly with an increasing number of client I/O requests – to approximately 32 to 64 remote clients – to make it well suited for the cloud and other shared environments.