How to establish QoE benchmarks that make sense in your environment

Our exploration of Quality of Experience continues with a look at how to design benchmarks for your business.

Benchmarking for QoE immediately presents a challenge from any number of perspectives. There are many network technologies that, in most environments, function together. These support a variety of protocol types and in the end enable the delivery of multiple application types. I don't think EMA has worked with a single IT or service provider that had a simple, monolithic set of requirements for delivering application services over the network.

Figure 1: Which of the following technologies does your network currently incorporate?

Which of the following technologies does your network currently incorporate?

© 2007 Enterprise Management Associates

Let's look first at the network transport technologies present in most IT organizations today, as in Figure 1. Old technologies such as frame relay and ATM cohabit with wireless, VPN and MPLS networks, many of which function as virtual overlays to core Ethernet transports. WAN, LAN and VLAN technologies exist together, along with the many flavors of WiMAX for last-mile wireless complements. And while the answers for QoE benchmarking reside beyond all of these transport challenges, they are far from immune from them. If you care about QoE, you need to have real visibility into the design and performance of your network, ideally with technologies that can begin to effectively link the user "experience" with real network transport issues.

Figure 2: Which applications are you delivering or planning to deliver over a network?

Which applications are you delivering or planning to deliver over a network?

© 2007 Enterprise Management Associates

More to the point for QoE itself, there is a wide array of application technologies and types, as in Figure 2. As you can see from this data from last year -- and EMA has revisited this discussion on several occasions with virtually identical results -- Web applications clearly dominate when it comes to managing the performance of applications over the network. Many of these application types come with unique protocols -- such as AJAX, Java or ActiveX, and SOAP -- and are increasingly becoming componentized so that more pieces of different applications are residing in different locations, escalating the complexity of the user's end station (laptop, desktop, mobile device) with virtually the rest of the outside world. This complexity will only become greater as service-oriented architectures (SOAs) become more and more pervasive. Data from mid-2007 shows that SOA is already beginning to challenge VoIP as a protocol of concern to operations managers.

More on application benchmarking for quality of experience (QoE)

What is Quality of Experience?

QoE: Why technical benchmarking is not enough

Given the limits of space, I have chosen to focus on benchmarks targeted at this increasingly mainstream, Web-based application world, as opposed to VoIP or video, which do, of course, present unique challenges of their own and which deserve their own, separate discussion. The first thing to stress is that QoE telescopes the problem beyond the network itself into the true end-user experience. You can't benchmark the network if you're blind to these transaction-driven details, which often require complementary and more application- oriented monitoring than most network management solutions offer. As a result, I am going to focus primarily on those technologies and solutions targeted at capturing transactional levels of awareness.

One of the more hotly debated topics over the last few years has been the difference between synthetic transactions and observed transactions in testing application response. Another part of this discussion involves where the transactions are monitored by location -- e.g., in the data center, at the edge of the data center, or from end stations themselves. And if the end station, is it sufficient to have instrumentation on a per-branch office basis, or do you need to strive for blanket coverage across a wide variety of end-user points?

As for synthetic versus observed, the truth is that both are valuable. Synthetic tests are proactive, can give you more consistent data suitable for SLA requirements, and can let you know if availability is lost, which observed transactions typically cannot. Many synthetic tests also offer diagnostic value, especially when the scripts are optimized to look at certain types of transactional behaviors that occur on an ad-hoc basis in the real world. On the other hand, synthetic tests occur at specified intervals and therefore may fail to capture any number of real problems that occur in finite time frames. They may create overly simplified pictures of real-world experiences by a wide variety of the customers you may care about most. Moreover, many observed capabilities have become increasingly rich in function and are beginning to offer much of the granularity of insights once available only in synthetic tests. So the truth is that both synthetic and observed should be in place -- if you really care about QoE.

Placement is also a choice that is more both/and than either/or, but one where you should apply common sense based on business needs and cost versus a monomaniacal urge to achieve technical perfection. Data-centric transactional monitoring can provide back-office detail that is quite useful in diagnostics, but it can also provide rich insights into certain issues surrounding QoE -- in some cases, playing back actual transactions in cinematic fashion. These solutions can catch Web-application design issues, and even problems with layout and graphics, to which -- from a network perspective -- you may feel immune. But your immunity can be challenged when you're blamed for degraded network performance that's really caused by poor application design. However, there are also solutions that provide full-service playback from the user end station and which capture a different set of dynamics more thoroughly -- fully aligned with how the end user sees the world. And while many of these, especially those optimized for rich transactional detail, are optimized for branch office rather than blanket deployments, both are available and, to be frank, offer largely complementary values.

And then many of the more network-centric solutions for QoE benchmarking sit at the edge of the data center and calculate end-user experience, in some cases in conjunction with insights into the back office transaction as well. Some solutions also offer probes, or are probe-based, with a focus on network segments and are optimized to assess bandwidth/per application usage. While many of these function at Layers 3 and 4, almost all at least support HTTP monitoring, or detailed capture of Session Layer (Layer 5) transactions, and some offer more granular transaction analysis through Layer 7. Though most of these are not the most "heavy hitting" in the true QoE sense, having their insights can allow you to execute far more quickly on actually diagnosing the cause of the problem, and some will suffice for proactively anticipating most performance degradations in remote locations.

I'm going to close this tip with a partial list of vendors that offer strong insights into QoE from one or more of these perspectives. These include services such as Keynote and Gomez, which can offer valuable insights across the board, but which have come out of the tradition of synthetic test suites. Data-center transaction-oriented monitoring tools such as Empirix (which combines Web and VoIP call center interaction) and TeaLeaf (for granular transaction replay and analysis) can be complemented by strong end-station visibility through solutions such as AlertSite, Coradiant, Symphoniq and Xangati. Multi-purpose capabilities for QoE are available from larger vendors such as Compuware, Fluke Networks, OPNET and Quest. And finally, these complement some of the more network-centric approaches for QoE such as those from Apparent Networks, NetScout, NetQoS and Shunra.

How you choose to approach this thorny problem is, once again, a matter of pragmatism, in which cost and overall resources should be balanced with the need to know all the gory details. But do realize that benchmarking for QoE is an operations-wide endeavor and should ideally even touch application development and QA/Test as well. You can't  optimise the network in a vacuum when the experience you're trying to support lies beyond it.

This tip first appeared at

Read more on Network monitoring and analysis