Facebook has open sourced its Zstandard compression algorithm. The technology itself is said to outperform ‘zlib’, which has previously been considered to the reigning standard in this field.
What is a compression algorithm?
A compression algorithm works to reduce the size of data being handled — lossless and lossy compression are terms that describe whether or not, in the compression of a file, all original data can be recovered when the file is uncompressed.
In terms of usage, lossless compression would suit (for example) text or spreadsheet files and lossy compression is better suited for (for example) video and sound, where a certain amount of information loss will not be detected by most users.
According to facebook.github.io/zstd/, “Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder. It also offers a special mode for small data, called dictionary compression and can create dictionaries from any sample set. Zstandard library is provided as open source software using a BSD license.”
In this case, Zstandard compression is a lossless compression technology.
Zstandard in action
Zstandard works in a manner which directs the data in hand to be as ‘branchless’ as is physically and mathematically possible — in doing so we can say that Zstandard is able to reduce the number of potential ‘pipeline flushes’ which can occur (during un-compression) as a result of incorrect branch predictions.