The Avere hardware has allowed Framestore to avoid production bottlenecks that occurred near the end of movie production cycles and also allowed it to survive an electrical brownout without ill effect.
Soho-based Framestore, which employs more than 600 artists, developers and engineers, underwent a full revamp of its storage systems to deal with the demands of Gravity. This included deployment of around 1PB of Hitachi Data Systems’ BlueArc clustered NAS as bulk online storage with DotHill SANs behind open-source ZFS servers for nearline storage and tape for long-term archiving.
But a key challenge faced by Framestore was dealing with massive demand on the BlueArc storage systems as projects culminated, including large movie files with tens of Gigabytes per frame, said CTO Steve McPherson.
“The scenario we face at the end of every production cycle is that the systems get swamped with people working off the servers on sophisticated final renders at full resolution. The first warning signs are when the phone calls start coming that the network is slow,” he said.
More on NAS
Initially, Framstore began testing the Linux open-source caching tool CacheFS, but this proved adequate for reads but insufficiently rapid on writes, said McPherson.
He said that in the first stages of the project Framestore rejected Avere. “Why would anyone need this, we thought, and it didn’t help that they sent a sales person and not a technical person to see us,” said McPherson. “But the thing that kept them in the running was their [Avere CEO Ron Bianchini’s] history at Spinnaker [a NAS clustering specialist acquired by NetApp] and the more we looked at the problem the more it made sense.”
Eventually, Framestore deployed three Avere FXT 4500 NAS acceleration devices. These cache NAS traffic to speed up access to hot data with multiple devices operating in clustered fashion to give a single pool of storage to all devices accessing them. Each FXT 4500 has 144GB of DRAM, 2GB of NVRAM, 3TB of flash storage capacity and a combination of 10Gbps and 1Gbps Ethernet ports.
Main movie files are initially accessed via Avere and held there while staff work on them, said McPherson. Meanwhile, other staff can work on smaller sub-projects with no disruption of performance either way.
“Avere protects the render farm,” said McPherson, referring to Framestore’s high performance compute and storage cluster used to create CGI effects. “It extracts sets of frequently used data and isolates them as cache.”
The most extreme test of the Avere cluster’s usefulness came when Framestore suffered an electrical brownout that knocked out most of its systems. “Normally we’d have problems when we fired things up after an event like this,” said McPherson. “But because the working set was there in Avere it went straight to peak capacity and the majority of jobs fired straight back up from cache.”
Is there anything Avere could improve in future versions? Only the cost, said McPherson.
“It’s an expensive solution. I imagine if it catches on that the cost might come down, but as it is you have to have a well-defined project in mind to justify it.”