Endace

Last updated

Endace Ltd
Company type Private
Industry Network monitoring
Founded2001
Headquarters Auckland,
New Zealand
Key people
Stuart Wilson: CEO
Website www.endace.com

Endace Ltd is a privately owned network monitoring company, based in New Zealand and founded in 2001. [1] It provides network visibility and network recording products to large organizations. The company was listed on the London Stock Exchange in 2005 and then delisted in 2013 when it was acquired by Emulex. [2] In 2016 Endace was spun out of Emulex and is currently a private company. [3]

Contents

In October 2016, The Intercept revealed that some Endace clients were intelligence agencies, including the British GCHQ (known for conducting massive surveillance on network communications) and the Moroccan DGST, likewise known for mass surveillance of its citizens.

Background and history

Endace was founded after the DAG project at the School of Computing and Mathematical Sciences at the University of Waikato in New Zealand. [1] [4] The first cards designed at the university were intended to measure latency in ATM networks. [5]

In 2006, Endace transitioned from component manufacturer to appliance manufacturer to managed infrastructure provider. The company now sells network visibility fabrics, based on its range of network recorders, to large corporations and government agencies. [6]

Endace was the first New Zealand company to list on London's Alternative Investment Market when it floated in mid-June 2005 [7] a move which was not without controversy. [8] Poor share price performance in the early years and a seeming failure to attract a broad enough shareholder base lent weight to the criticism that Endace should have focused initially on developing its local profile (via NZX) rather than pushing for overseas investment (via London AIM).

Endace is headquartered in Auckland, New Zealand, and has an R&D centre in Hamilton, New Zealand, and offices in Australia, United States and Great Britain.

Key innovations of the DAG

The DAG project grew from academic research at Waikato University. Having found that software measurements of ATM cells (or packets) were unsatisfactory, both for reasons of accuracy and lack of certainty about packet loss, the research group set about developing their own hardware to generate better quality recordings. [5] This hardware and its subsequent iterations introduced two fundamental innovations: hardware timestamping and hardware accounting for packet loss.

Hardware timestamping

Conventionally, each packet or cell is given a timestamp by the host machine's kernel (i.e. in software) when the kernel driver is notified that a new packet has arrived. This approach results in poor quality timestamps for several reasons, among them the considerable latency and jitter between the packet arriving at the network interface and receipt by the kernel driver and uncertainty caused by interrupt coalescing wherein one host interrupt signifies the arrival of several packets. Such poor quality limits what research can usefully be done on network performance and related fields.

To solve this, the DAG generates timestamps in the hardware as close to the network interface as possible. Not only does this obviate latency, jitter and problems caused by interrupt coalescing, the hardware is capable of much greater accuracy and precision than software-generated timestamps. Precision comes from the freedom of custom hardware to assign as many bits to the timestamp as required and accuracy is assured by reference to an external time source such as GPS which is accurate to ± 40 nanoseconds. [9] In contrast, the accuracy of NTP (by which kernel clocks can be corrected over the Internet) is in the order of milliseconds (about 100,000 times less accurate), depending on the conditions involved.

The DAG produces 64 bit timestamps in fixed-point format with 32 fractional bits, giving a potential precision of seconds or 233 picoseconds. The actual precision offered varies with the particular model of DAG, the oldest giving 24 fractional bits (60 nanoseconds) and better precisions offered in DAGs for higher bandwidth networks. [10]

The timestamp is derived from a free-running clock provided by a crystal oscillator but the accuracy of crystals drift with both temperature and age. The DAG's solution is to use direct digital synthesis using the 1 Hz pulse-per-second output that many GPS receivers provide as its reference clock. This mechanism is described in §5.5.3 of Stephen Donnelly's PhD thesis [11] which also describes in detail the pre-commercial era models of DAG.

Crucially, and an academically significant contribution of the DAG, the ability to use an external reference such as globally synchronised GPS makes it possible to do one-way time-of-flight measurements. This is of immense interest to academic researchers because packets flowing between two points on the Internet are neither guaranteed to follow the same path in each direction nor guaranteed to have the same timing characteristics in each direction.

Outside of the academic world, timestamp accuracy has commercial applications in the enforcement and compliance with law such as the EU Markets in Financial Instruments Directive 2004.

Packet loss

Almost as important as timestamp accuracy is guaranteeing 100% cell or packet capture and, where loss is unavoidable, knowing not only that packets have been lost but where. The "where" is important because, when analysing a packet trace, it's important to be able to compensate for lost packets when calculating inter-arrival times.

Most commercial NICs keep a count of dropped packets, but they can't indicate where packets were lost. The DAG prepends a header [12] which, amongst other things, indicates how many packets were dropped between that packet and the previously accepted packet.

The DAG is also engineered to deliver recorded packets to the host with the greatest possible efficiency. That, together with the interstitial loss counter, is what makes the DAG so appealing for surveillance applications. The interstitial loss counter also finds application in forensics; a prosecutor needs to be able to prove that the record is complete or, if it is not, where it is not.

Controversy and surveillance

In October 2016, The Intercept published an article showing that Endace customers include intelligence agencies, including the GCHQ, Canadian and Australian intelligence agencies, and the DGST (Morocco's domestic surveillance agency). [13] Edward Snowden documents have shown that the GCHQ has installed massive surveillance of network communications in UK, using the over-sea cable between Europe and North America.

Related Research Articles

The Transmission Control Protocol (TCP) is one of the main protocols of the Internet protocol suite. It originated in the initial network implementation in which it complemented the Internet Protocol (IP). Therefore, the entire suite is commonly referred to as TCP/IP. TCP provides reliable, ordered, and error-checked delivery of a stream of octets (bytes) between applications running on hosts communicating via an IP network. Major internet applications such as the World Wide Web, email, remote administration, and file transfer rely on TCP, which is part of the Transport layer of the TCP/IP suite. SSL/TLS often runs on top of TCP.

In electronics and telecommunications, jitter is the deviation from true periodicity of a presumably periodic signal, often in relation to a reference clock signal. In clock recovery applications it is called timing jitter. Jitter is a significant, and usually undesired, factor in the design of almost all communications links.

Time and frequency transfer is a scheme where multiple sites share a precise reference time or frequency. The technique is commonly used for creating and distributing standard time scales such as International Atomic Time (TAI). Time transfer solves problems such as astronomical observatories correlating observed flashes or other phenomena with each other, as well as cell phone towers coordinating handoffs as a phone moves from one cell to another.

<span class="mw-page-title-main">Network Time Protocol</span> Standard protocol for synchronizing time across devices

The Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks. In operation since before 1985, NTP is one of the oldest Internet protocols in current use. NTP was designed by David L. Mills of the University of Delaware.

<span class="mw-page-title-main">Radio clock</span> Type of clock which self-synchronizes its time using dedicated radio transmitters

A radio clock or radio-controlled clock (RCC), and often colloquially referred to as an "atomic clock", is a type of quartz clock or watch that is automatically synchronized to a time code transmitted by a radio transmitter connected to a time standard such as an atomic clock. Such a clock may be synchronized to the time sent by a single transmitter, such as many national or regional time transmitters, or may use the multiple transmitters used by satellite navigation systems such as Global Positioning System. Such systems may be used to automatically set clocks or for any purpose where accurate time is needed. Radio clocks may include any feature available for a clock, such as alarm function, display of ambient temperature and humidity, broadcast radio reception, etc.

Clock synchronization is a topic in computer science and engineering that aims to coordinate otherwise independent clocks. Even when initially set accurately, real clocks will differ after some amount of time due to clock drift, caused by clocks counting time at slightly different rates. There are several problems that occur as a result of clock rate differences and several solutions, some being more acceptable than others in certain contexts.

Network performance refers to measures of service quality of a network as seen by the customer.

Professional video over IP systems use some existing standard video codec to reduce the program material to a bitstream, and then use an Internet Protocol (IP) network to carry that bitstream encapsulated in a stream of IP packets. This is typically accomplished using some variant of the RTP protocol.

The Precision Time Protocol (PTP) is a protocol for clock synchronization throughout a computer network with relatively high precision and therefore potentially high accuracy. In a local area network (LAN), accuracy can be sub-microsecond – making it suitable for measurement and control systems. PTP is used to synchronize financial transactions, mobile phone tower transmissions, sub-sea acoustic arrays, and networks that require precise timing but lack access to satellite navigation signals.

Capacity management's goal is to ensure that information technology resources are sufficient to meet upcoming business requirements cost-effectively. One common interpretation of capacity management is described in the ITIL framework. ITIL version 3 views capacity management as comprising three sub-processes: business capacity management, service capacity management, and component capacity management.

A pulse per second is an electrical signal that has a width of less than one second and a sharply rising or abruptly falling edge that accurately repeats once per second. PPS signals are output by radio beacons, frequency standards, other types of precision oscillators and some GPS receivers. Precision clocks are sometimes manufactured by interfacing a PPS signal generator to processing equipment that aligns the PPS signal to the UTC second and converts it to a useful display. Atomic clocks usually have an external PPS output, although internally they may operate at 9,192,631,770 Hz. PPS signals have an accuracy ranging from 12 picoseconds to a few microseconds per second, or 2.0 nanoseconds to a few milliseconds per day based on the resolution and accuracy of the device generating the signal.

In computer science and computer programming, system time represents a computer system's notion of the passage of time. In this sense, time also includes the passing of days on the calendar.

<span class="mw-page-title-main">Atomic clock</span> Clock that monitors the resonant frequency of atoms

An atomic clock is a clock that measures time by monitoring the resonant frequency of atoms. It is based on atoms having different energy levels. Electron states in an atom are associated with different energy levels, and in transitions between such states they interact with a very specific frequency of electromagnetic radiation. This phenomenon serves as the basis for the International System of Units' (SI) definition of a second:

The second, symbol s, is the SI unit of time. It is defined by taking the fixed numerical value of the caesium frequency, , the unperturbed ground-state hyperfine transition frequency of the caesium-133 atom, to be 9192631770 when expressed in the unit Hz, which is equal to s−1.

ITU-T Y.156sam Ethernet Service Activation Test Methodology is a draft recommendation under study by the ITU-T describing a new testing methodology adapted to the multiservice reality of packet-based networks.

ITU-T Y.1564 is an Ethernet service activation test methodology, which is the new ITU-T standard for turning up, installing and troubleshooting Ethernet-based services. It is the only standard test methodology that allows for complete validation of Ethernet service-level agreements (SLAs) in a single test.

PTPd is an open source implementation of the Precision Time Protocol for Unix-like computers.

White Rabbit is the name of a collaborative project including CERN, GSI Helmholtz Centre for Heavy Ion Research and other partners from universities and industry to develop a fully deterministic Ethernet-based network for general purpose data transfer and sub-nanosecond accuracy time transfer. Its initial use was as a timing distribution network for control and data acquisition timing of the accelerator sites at CERN as well as in GSI's Facility for Antiproton and Ion Research (FAIR) project. The hardware designs as well as the source code are publicly available. The name of the project is a reference to the White Rabbit appearing in Lewis Carroll's novel Alice's Adventures in Wonderland.

RTP-MIDI is a protocol to transport MIDI messages within Real-time Transport Protocol (RTP) packets over Ethernet and WiFi networks. It is completely open and free, and is compatible both with LAN and WAN application fields. Compared to MIDI 1.0, RTP-MIDI includes new features like session management, device synchronization and detection of lost packets, with automatic regeneration of lost data. RTP-MIDI is compatible with real-time applications, and supports sample-accurate synchronization for each MIDI message.

AES67 is a technical standard for audio over IP and audio over Ethernet (AoE) interoperability. The standard was developed by the Audio Engineering Society and first published in September 2013. It is a layer 3 protocol suite based on existing standards and is designed to allow interoperability between various IP-based audio networking systems such as RAVENNA, Wheatnet, Livewire, Q-LAN and Dante.

Deterministic Networking (DetNet) is an effort by the IETF DetNet Working Group to study implementation of deterministic data paths for real-time applications with extremely low data loss rates, packet delay variation (jitter), and bounded latency, such as audio and video streaming, industrial automation, and vehicle control.

References

  1. 1 2 "The DAG Project". Archived from the original on 29 November 2001.
  2. ↑ "ENDACE LTD (EDA:NL): Company Description - BusinessWeek". Bloomberg Businessweek investing database. Bloomberg L.P. Archived from the original on 10 October 2012. Retrieved 9 February 2011.
  3. ↑ "Endace Spins off from Emulex in Management-led Buyout". New Zealand: Endace. 10 March 2016. Retrieved 13 March 2016.
  4. ↑ "Yoke Har Lee: Life's a bit of a DAG for hi-tech firm". The New Zealand Herald . 24 August 2009. Retrieved 11 September 2011.
  5. 1 2 Cleary, John; Donnelly, Stephen; Graham, Ian; McGregor, Anthony; Pearson, Murray. Design Principles for Accurate Passive Measurement (PDF) (Report). Waikato University. Retrieved 13 May 2017.
  6. ↑ "What is a managed service provider? Strategic outsourcing for IT services". CIO. Retrieved 9 August 2024.
  7. ↑ "Growth Business: Endace poised to take AIM". Archived from the original on 19 November 2005.
  8. ↑ Inder, Richard (5 June 2006). "Endace's performance on UK AIM listing gives fuel to critics". The New Zealand Herald . Retrieved 11 September 2011.
  9. ↑ "§A.4.8 UTC(USNO) Offset Accuracy". Global Positioning System Standard Positioning Service Performance Standard (PDF) (Report) (4th ed.). US Department of Defense. September 2008. p. A-16. Retrieved 13 May 2017.
  10. ↑ Micheel, Jörg; Donnelly, Stephen; Graham, Ian (2001). "Precision timestepping of network packets" (PDF). Proceedings of the First ACM SIGCOMM Workshop on Internet Measurement – IMW '01. Waikato University. p. 273. doi:10.1145/505202.505236. ISBN   1581134355. S2CID   14567389. Archived from the original (PDF) on 25 February 2018. Retrieved 13 May 2017.
  11. ↑ Donnelly, Stephen F. (2002). High Precision Timing in Passive Measurements of Data Networks (PhD). CiteSeerX   10.1.1.136.1730 .
  12. ↑ ""Extensible Record Format" header description". WireShark. Retrieved 13 May 2017.
  13. ↑ "The Little-Known Company That Enables Worldwide Mass Surveillance". The Intercept. 23 October 2016. Retrieved 2 November 2016.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.