TPC-E: New IT benchmarks for OLTP database servers

Benchmarks are a critical part of the IT industry. They allow us to measure technological progress, enable buyers to compare offerings from multiple vendors and help vendors improve their products. In the world of database servers, the Transaction Processing Performance Council Benchmark C (TPC-C) has had a starring role in the online transaction processing (OLTP) space for more than 15 years. Few vendors publish data server specifications without TPC-C numbers. Now, TPC offers a new, rich and carefully designed benchmark -- TPC-E -- intended to become, like TPC-C, a standard measure of data server performance. TPC-E is designed to be broadly representative of contemporary OLTP systems, challenging today's fast processors with a rich data model and demanding calculations.

The TPC-E model
TPC-E is modeled on a brokerage with its database of customers, accounts and securities. In a typical brokerage setting, customers generate data research requests, account inquiries and trade requests. The brokerage house receives trade requests, sends them to the market and returns the results to the customer. When developing TPC-E, TPC consulted with well-known brokerage houses to develop a model that is as realistic as possible. TPC-E uses multiple trade types -- market orders, limit orders, stop orders and more. There are several tiers of simulated customers, each with distinct trading behaviors. Simulated companies issue both preferred and common stock. When customers make research requests, the system responds with large text documents, which include line-of-business (LOB) data as well as historic financial and market data.

The resulting model comprises 33 tables, 22 check constraints and referential integrity constraints, including 33 primary and 50 foreign keys. TPC-E has three dimensions of data, posing realistic partitioning challenges. The model implements business logic using 10 transactions, many with various modes of execution. The rich schema and diverse transactions of TPC-E provide new opportunities for vendors to optimize system performance.

Modern database features
TPC-E is designed to match several additional features of contemporary databases:

• Some transactions in TPC-E cannot be implemented as a single stored procedure, enforcing multiple round trips for certain data.
• Strict database isolation requirements in TPC-E require improved database locking performance.
• Constraint checking, now a standard feature of all database management systems, is enforced in TPC-E.
• TPC-E provides a business intelligence-style transaction (broker-volume) that requires the processing of various-sized data sets, including volatile data.
• TPC-E requires database disks to be RAID-protected, reflecting typical real-world configurations. In addition, sponsors must report the time interval between catastrophic disk failure as achieving 95% of pre-failure throughput. Buyers will be able to better judge the tradeoffs between performance and reliability, and vendors will be rewarded for maximizing both together.

Low barriers to TCP benchmark adoption
Benchmark sponsors will find that running TPC-E is relatively easy and affordable. TPC-E has been carefully designed and tested to balance CPU demands with physical I/O, minimizing the number of disk drives required to run it. In addition to shrinking the cost of running TPC-E, lower I/O requirements will result in a closer resemblance between benchmark configurations and those used in real data centers. Also, TPC-E is focused on benchmarking the database server and requires fewer front-end systems to emulate user actions. Finally, TPC has provided code that generates test data for the initial database as well as implementations of key parts of the workload logic. This lets test sponsors focus on optimizing their products rather than developing and optimizing the benchmark application. All of these design choices reduce cost and complexity, putting TPC-E within reach of a broader spectrum of hardware vendors. The more broadly a benchmark is used, the more helpful it is for both buyers and manufacturers.

Transitioning to TPC-E
In many industries that use OLTP, TPC-E will deliver highly relevant performance measurements and drive practical system improvements. While lower cost and simplified infrastructure will allow TPC-E to be adopted rapidly, TPC-C remains a crucial part of the technology industry. Vendors can continue maximizing the value of their investment in TPC-C while preparing to transition to or add TPC-E to their environments. Customers can gain expertise in interpreting TPC-E results and shift their decision frameworks gradually to include the new benchmark. Just as TPC-C has done, TPC-E will help the transaction processing industry achieve new innovations and new heights of performance.

ABOUT THE AUTHOR: Matthew Emmerton is the chairperson of the TPC-E Maintenance Subcommittee. He has been involved in the development and maintenance of the TPC-E benchmark standard since 2005. He is also a member of the TPC's Technical Advisory Board.