• Two different approaches to measuring throughput, one of which often fails in a HetNet environment
• How client buffers and performance can impact network measurements
• How HetNets and technology development will force us to find alternatives to measuring capacity and throughput

Throughput measurements are a cornerstone in any network testing toolbox, designed to stress the capacity and bandwidth of the network. What’s important to remember is that, in order for throughput to carry any real value the network path (number of hops per IP packet) should be minimized.

When you want to test the speed of your network and your network alone it is important that the packets never leave the network so that the measurement becomes influenced by a third party, such as an operator or content provider.

This case also illustrates one of the challenges MNOs face when it comes to OTT applications. Since the service can originate or terminate outside of the controlled part of the network, guaranteeing performance and control over a quality test is nigh impossible. Content caching and compression are common techniques to enhance fast retrieval of common objects outside the core network.

ETSI has standardized throughput measurements for well-known services (HTTP / FTP). It is essential that these KPI’s are used in reporting in order to allow KPI comparison across tools and capture points.

There are two main options when doing throughput measurements using file or content transfers.

• Complete transfers (the entire file is transferred successfully)
• Time based measurements (transfer aborted after a pre-defined time, even if there is data left to transfer)

I’d recommend time based measurements for the following reasons.

• Makes the transfer time same regardless of underlying network technology
• Total measurement execution time is more predictable because of ‘1’
• Avoid network technology changes to influence a single measurement sample, i.e. switch from GSM to LTE
• Does not exhaust network to the same degree as a complete download

Regardless of complete or time based measurements ETSI defines a series of triggers used to drive the KPI calculation. For IP based measurements these triggers are defined on the packet level. In order to have full compliance with the standard the triggers must be evaluated by deep inspection.

ETSI KPI Triggers

Dissecting an IP based test (in this case a HTTP Download) we get the following picture.

Note: The ETSI trigger markers are in bold under the UE/Client column.

Measurement window and abort criteria

Each service goes through various stages/states, such as:

• Connect – Setup of connection (TCP establishment, including DNS)
• Authenticate/Negotiate
  • Authentication to service (logging in – if required)
  • Negotiation (selection of which file to download for FTP)
• Transfer – The actual transfer of content
• Close – Tear down of connection

The various stages are not required for all services and could be optional, for example HTTP Download does not usually require authentication. There might as well be additional steps required for other services, the negotiation phase for e-mail is very different from FTP.

Each stage can be subject for time-out handling. In many measurement solutions this has been simplified to two operational windows, both subject to time-out handling:

• Session (or Activity) timeout
• Transfer timeout (if time limited measurements are enabled)

The session timeout controls the whole activity (HTTP/FTP Download, etc..). While the transfer timeout is triggered only during the data transfer phase. This simplification is set to mimic user behavior. Most applications will use the host OS default values for this. As such an aborted activity due to timeout is therefore an indication that something is not quite right in the network. In case of an activity timeout it is possible to track in which state the service was aborted, the state is tracked as an additional KPI.

Complete transfers allow the test to fully complete the transfer. This makes it very easy to estimate the throughput. It is as simple as dividing the file size with the time it took to transfer the file. Presto! It is also possible to achieve high ETSI compliance even without deep packet inspection. ETSI triggers can be transformed to API-level function calls and the delay introduced thereof can be quantified.

As such, complete transfers are the preferred way of measuring on hosts that does not allow for deep packet inspection. However, as mentioned previously complete transfers are not suitable for throughput measurements.

Imagine testing your LTE network, pushing maybe 60-80 Mb/s throughput, which requires a quite large file in order for the network to reach it’s full throughput potential. If the file is too short the test will be over before max throughput has been reached and your test data will consist mainly of throughput values that are ramping up but which are not representative of the network performance. So, setting up your transfer with a sizable file, what happens if you’re handed down to GSM or EDGE? You’ll be waiting for that transfer to end for ever, and ever, and ever…

In my next post I will delve deeper into time-based measurements, the impact of buffers and device performance on KPI measurements.

Until next time!

Fredrik