Y.156sam

ITU-T Y.156sam Ethernet Service Activation Test Methodology is a draft recommendation under study by the ITU-T describing a new testing methodology adapted to the multiservice reality of packet-based networks.

Key objectives

ITU-T Y.156sam is designed around three key objectives:

• To serve as a network service level agreement (SLA) validation tool, ensuring that a service meets its guaranteed performance settings in a controlled test time
• To ensure that all services carried by the network meet their SLA objectives at their maximum committed rate, proving that under maximum load network devices and paths can support all the traffic as designed
• To perform medium- and long-term service testing, confirming that network element can properly carry all services while under stress during a soaking period

Test methodology

ITU-T Y.156sam defines an out-of-service test methodology to assess the proper configuration and performance of an Ethernet service prior to customer notification and delivery. The test methodology applies to point-to-point and point-to-multipoint connectivity in the Ethernet layer and to the network portions that provide, or contribute to, the provisioning of such services. This recommendation does not define Ethernet network architectures or services, but rather defines a methodology to test Ethernet-based services at the service activation stage. In particular, it is aimed at solving the deficiencies of RFC 2544 listed below.

Existing test methodologies: RFC 2544

The Internet Engineering Task Force RFC 2544 is a benchmarking methodology for network interconnect devices. This request for comment (RFC) was created in 1999 as a methodology to benchmark network devices such as hubs, switches and routers as well as to provide accurate and comparable values for comparison and benchmarking.

RFC 2544 provides engineers and network technicians with a common language and results format. The RFC 2544 describes six subtests:

• Throughput : measures the maximum rate at which none of the offered frames are dropped by the device/system under test (DUT/SUT). This measurement translates into the available bandwidth of the Ethernet virtual connection.
• Back-to-back or burstability: measures the longest burst of frames at maximum throughput or minimum legal separation between frames that the device or network under test will handle without any loss of frames. This measurement is a good indication of the buffering capacity of a DUT.
• Frame loss: defines the percentage of frames that should have been forwarded by a network device under steady state (constant) loads that were not forwarded due to lack of resources. This measurement can be used for reporting the performance of a network device in an overloaded state, as it can be a useful indication of how a device would perform under pathological network conditions such as broadcast storms.
• Latency : measures the round-trip time taken by a test frame to travel through a network device or across the network and back to the test port. Latency is the time interval that begins when the last bit of the input frame reaches the input port and ends when the first bit of the output frame is seen on the output port. It is the time taken by a bit to go through the network and back. Latency variability can be a problem. With protocols like voice over Internet protocol (VoIP), a variable or long latency can cause degradation in voice quality.
• System reset: measures the speed at which a DUT recovers from a hardware or software reset. This subtest is performed by measuring the interruption of a continuous stream of frames during the reset process.
• System recovery: measures the speed at which a DUT recovers from an overload or oversubscription condition. This subtest is performed by temporarily oversubscribing the device under test and then reducing the throughput at normal or low load while measuring frame delay in these two conditions. The different between delay at overloaded condition and the delay and low load conditions represent the recovery time.

Drawbacks of RFC 2544

From a laboratory and benchmarking perspective, the RFC 2544 methodology is an ideal tool for automated measurement and reporting.

From a service turn-up and troubleshooting perspective, RFC 2544, although acceptable and valid, does have some drawbacks:

• Service providers are shifting from only providing Ethernet pipes to enabling services. Networks must support multiple services from multiple customers, and each service has its own performance requirements that must be met even under full load conditions and with all services being processed simultaneously. RFC 2544 was designed as a performance tool with a focus on a single stream to measure maximum performance of a DUT or network under test and was never intended for multiservice testing.
• With RFC 2544’s focus on identifying the maximum performance of a device or network under test, the overall test time is variable and heavily depends on the quality of the link and subtest settings. RFC 2544 test cycles can easily require a few hours of testing. This is not an issue for lab testing or benchmarking, but becomes a serious issue for network operators with short service maintenance windows.
• Packet delay variation is a key performance indicator (KPI) for real-time services such as VoIP and Internet Protocol television (IPTV) and is not measured by the RFC 2544 methodology. Network operators that performed service testing with RFC 2544 typically must execute external packet jitter testing outside of RFC 2544 as this KPI was not defined or measured by the RFC.
• Testing is performed sequentially on one KPI after another. In multiservice environments, traffic is going to experience all KPIs at the same time, although throughput might be good, it can also be accompanied by very high latency due to buffering. Designed as a performance assessment tool, RFC 2544 measures each KPI individually through its subtest and therefore cannot immediately associate a very high latency with a good throughput, which should be cause for concern.

Service definitions

The ITU-T Y.156sam defines test streams with service attributes linked to the Metro Ethernet Forum (MEF) 10.2 definitions.

Services are traffic streams with specific attributes identified by different classifiers such as 802.1q VLAN, 802.1ad and class of service (CoS) profiles. These services are defined at the UNI level with different frame and bandwidth profile such as the service’s maximum transmission unit (MTU) or frame size, committed information rate (CIR), and excess information rate (EIR).

Test rates

The ITU Y.156sam defines three key test rates based on the MEF service attributes for Ethernet virtual circuit (EVC) and user-to-network interface (UNI) bandwidth profiles.

• CIR defines the maximum transmission rate for a service where the service is guaranteed certain performance objectives. These objectives are typically defined and enforced via SLAs.
• EIR defines the maximum transmission rate above the committed information rate considered as excess traffic. This excess traffic is forwarded as capacity allows and is not subject to meeting any guaranteed performance objectives (best effort forwarding).
• Overshoot rate defines a testing transmission rate above CIR or EIR and is used to ensure that the DUT or network under test does not forward more traffic than specified by the CIR or EIR of the service.

Color markings

These rates can be associated to color markings:

• Green traffic is equivalent to CIR
• Yellow traffic is equivalent to EIR
• Red traffic represents discarded traffic (overshoot – CIR or overshoot – EIR).

Subtests

The ITU-T Y.156sam is built around two key subtests, the service configuration test and the service performance test, which are performed in order:

Service configuration test

Forwarding devices such as switches, routers, bridges and network interface units are the basis of any network as they interconnect segments. If a service is not correctly configured on any one of these devices within the end-to-end path, network performance can be greatly affected, leading to potential service outages and network-wide issues such as congestion and link failures.

The Service configuration test measures the ability of DUT or network under test to properly forward in three different states:

• In the CIR phase, where performance metrics for the service are measured and compared to the SLA performance objectives
• In the EIR phase, where performance is not guaranteed and the services transfer rate is measured to ensure that CIR is the minimum bandwidth
• In the discard phase, where the service is generated at the overshoot rate and the expected forwarded rate is not greater than the committed information rate or excess rate (when configured).

Service performance test

As network devices come under load, they must prioritize one traffic flow over another to meet the KPIs set for each traffic class. With only one traffic class, there is no prioritization performed by the network devices since there is only one set of KPIs. As the number of traffic flows increase, prioritization is necessary and performance failures may occur.

The service performance test measures the ability of the DUT or network under test to forward multiple services while maintaining SLA conformance for each service. Services are generated at the CIR, where performance is guaranteed, and pass/fail assessment is performed on the KPI values for each service according to its SLA.

Service performance assessment must also be maintained for a medium- to long-term period as performance degradation will likely occur as the network is under stress for longer period of times. The service performance test is designed to soak the network under full committed load for all services and measure performance over medium and long test time.

Metrics

The Y.156sam focuses on the following KPIs for service quality:

• Bandwidth: this is a bit rate measure of available or consumed data communication resources expressed in bits/second or multiples of it (kilobits/s, megabits/s, etc.).
• Frame transfer delay (FTD): also known as latency, this is a measurement of the time delay between the transmission and the reception of a frame. Typically this is a round-trip measurement, meaning that the calculation measures both the near-end to far-end and far-end to near-end direction simultaneously.
• Frame delay variations: also known as packet jitter, this is a measurement of the variations in the time delay between packet deliveries. As packets travel through a network to their destination, they are often queued and sent in bursts to the next hop. There may be prioritization at random moments also resulting in packets being sent at random rates. Packets are therefore received at irregular intervals. The direct consequence of this jitter is stress on the receiving buffers of the end nodes where buffers can be overused or underused when there are large swings of jitter.
• Frame loss: typically expressed as a ratio, this is a measurement of the number of packets lost over the total number of packets sent. Frame loss can be due to a number of issues such as network congestion or errors during transmissions.

Vendor implementation

The ITU-T Y.1564 (previously Y.156sam) has gained momentum in the test and measurement industry. EXFO was the first test vendor to implement the ITU-T Y.1564, and today other companies like JDSU, Albedo Telecom and Veex are also offering Ethernet and multiservice test equipment via the IntelliSAM and EtherSAM test methodologies.

References

• RFC 1242 (1996), Benchmarking Terminology for Network Interconnection Devices
• RFC 2544 (1999), Benchmarking Methodology for Network Interconnect Devices

External links

• ALBEDO Telecom eSAM
• EXFO EtherSAM
• Metro Ethernet Forum EPL
• Metro Ethernet Forum MEF 10.1 ^{[ permanent dead link ]}