NetFlow

Last updated

NetFlow is a feature that was introduced on Cisco routers around 1996 that provides the ability to collect IP network traffic as it enters or exits an interface. By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components: [1]

Contents

Protocol description

Routers and switches that support NetFlow can collect IP traffic statistics on all interfaces where NetFlow is enabled, and later export those statistics as NetFlow records toward at least one NetFlow collector—typically a server that does the actual traffic analysis.

Network flows

Cisco standard NetFlow version 5 defines a flow as a unidirectional sequence of packets that all share seven values which define a unique key for the flow: [2]

  1. Ingress interface (SNMP ifIndex)
  2. Source IP address
  3. Destination IP address
  4. IP protocol number
  5. Source port for UDP or TCP, 0 for other protocols
  6. Destination port for UDP or TCP, type and code for ICMP, or 0 for other protocols
  7. IP Type of Service

Note that the Egress interface, IP Nexthop or BGP Nexthops are not part of the key, and may not be accurate if the route changes before the expiration of the flow, or if load-balancing is done per-packet.

This definition of flows is also used for IPv6, and a similar definition is used for MPLS and Ethernet flows.

Advanced NetFlow or IPFIX implementations like Cisco Flexible NetFlow allow user-defined flow keys.

A typical output of a NetFlow command line tool (nfdump in this case) when printing the stored flows may look as follows:

 Date flow start          Duration Proto   Src IP Addr:Port      Dst IP Addr:Port     Packets    Bytes Flows  2010-09-01 00:00:00.459     0.000 UDP     127.0.0.1:24920   ->  192.168.0.1:22126        1       46     1  2010-09-01 00:00:00.363     0.000 UDP     192.168.0.1:22126 ->  127.0.0.1:24920          1       80     1

Export of records

The router will output a flow record when it determines that the flow is finished. It does this by flow aging: when the router sees new traffic for an existing flow it resets the aging counter. Also, TCP session termination in a TCP flow causes the router to expire the flow. Routers can also be configured to output a flow record at a fixed interval even if the flow is still ongoing.

Packet transport protocol

NetFlow records are traditionally exported using User Datagram Protocol (UDP) and collected using a NetFlow collector. The IP address of the NetFlow collector and the destination UDP port must be configured on the sending router. A common value is UDP port 2055, but other values like 9555 or 9995, 9025, 9026 etc. can also be used.

For efficiency reasons, the router traditionally does not keep track of flow records already exported, so if a NetFlow packet is dropped due to network congestion or packet corruption, all contained records are lost forever. The UDP protocol does not inform the router of the loss so it can send the packets again. This can be a real problem, especially with NetFlow v8 or v9 that can aggregate a lot of packets or flows into a single record. A single UDP packet loss can cause a huge impact on the statistics of some flows.

That is why some modern implementations of NetFlow use the Stream Control Transmission Protocol (SCTP) to export packets so as to provide some protection against packet loss, and make sure that NetFlow v9 templates are received before any related record is exported. Note that TCP would not be suitable for NetFlow because a strict ordering of packets would cause excessive buffering and delays.

The problem with SCTP is that it requires interaction between each NetFlow collector and each router exporting NetFlow. There may be performance limitations if a router has to deal with many NetFlow collectors, and a NetFlow collector has to deal with many routers, especially when some of them are unavailable due to failure or maintenance.

SCTP may not be efficient if NetFlow must be exported toward several independent collectors, some of which may be test servers that can go down at any moment. UDP allows simple replication of NetFlow packets using Network taps or L2 or L3 Mirroring. Simple stateless equipment can also filter or change the destination address of NetFlow UDP packets if necessary. Since NetFlow export almost only use network backbone links, packet loss will often be negligible. If it happens, it will mostly be on the link between the network and the NetFlow collectors.

Packet headers

All NetFlow packets begin with version-dependent header, that contains at least these fields:

  • Version number (v1, v5, v7, v8, v9)
  • Sequence number to detect loss and duplication
  • Timestamps at the moment of export, as system uptime or absolute time.
  • Number of records (v5 or v8) or list of templates and records (v9)

Records

A NetFlow record can contain a wide variety of information about the traffic in a given flow.

NetFlow version 5 (one of the most commonly used versions, followed by version 9) contains the following:

For ICMP flows, the Source Port is zero, and the Destination Port number field codes ICMP message Type and Code (port = ICMP-Type * 256 + ICMP-Code) [ citation needed ].

The source and destination Autonomous System (AS) number fields can report the destination AS (last AS of AS-Path) or the immediate neighbor AS (first AS of AS-Path) depending on the router configuration. But the AS number will be zero if the feature is not supported, the route is unknown or not announced by BGP, or the AS is the local AS. There is no explicit way to distinguish between these cases.

NetFlow version 9 can include all of these fields and can optionally include additional information such as Multiprotocol Label Switching (MPLS) labels and IPv6 addresses and ports,

By analyzing flow data, a picture of traffic flow and traffic volume in a network can be built. The NetFlow record format has evolved over time, hence the inclusion of version numbers. Cisco maintains details of the different version numbers and the layout of the packets for each version.

Interfaces

NetFlow is usually enabled on a per-interface basis to limit load on the router components involved in NetFlow, or to limit the amount of NetFlow records exported.

NetFlow usually captures all packets received by an ingress IP interface, but some NetFlow implementations use IP filters to decide if a packet can be observed by NetFlow.

Some NetFlow implementations also allow the observation of packets on the egress IP interface, but this must be used with care: all flows from any ingress interface with NetFlow enabled to any interface with NetFlow enabled could be counted twice.

Sampled NetFlow

Standard NetFlow was designed to process all IP packets on an interface. But in some environments, e.g. on Internet backbones, that was too costly, due to the extra processing required for each packet, and large number of simultaneous flows.

So Cisco introduced sampled NetFlow on Cisco 12000, and that is now used in all high-end routers that implement NetFlow.

Only one packet out of n is processed, where n, the sampling rate, is determined by the router configuration.

The exact selection process depends on the implementation:

Some implementations have more complex methods to sample packets, like per-flow sampling on Cisco Martinez Catalysts.

The sampling rate is often the same for all interfaces, but can be adjusted per interface for some routers. When Sampled NetFlow is used, the NetFlow records must be adjusted for the effect of sampling - traffic volumes, in particular, are now an estimate rather than the actual measured flow volume.

The sampling rate is indicated in a header field of NetFlow version 5 (same sampling rate for all interfaces) or in option records of NetFlow version 9 (sampling rate per interface)

Versions

VersionComment
v1First implementation, now obsolete, and restricted to IPv4 (without IP mask and AS Numbers).
v2Cisco internal version, never released.
v3Cisco internal version, never released.
v4Cisco internal version, never released.
v5Most common version, available (as of 2009) on many routers from different brands, but restricted to IPv4 flows.
v6No longer supported by Cisco. Encapsulation information (?).
v7Like version 5 with a source router field. Used (only?) on Cisco Catalyst switches.
v8Several aggregation form, but only for information that is already present in version 5 records
v9Template Based, available (as of 2009) on some recent routers. Mostly used to report flows like IPv6, MPLS, or even plain IPv4 with BGP nexthop.
v10Used for identifying IPFIX. Although IPFIX is heavily based on NetFlow, v10 does not have anything to do with NetFlow.

NetFlow and IPFIX

NetFlow was initially implemented by Cisco, and described in an "informational" document that was not on the standards track: RFC 3954 – Cisco Systems NetFlow Services Export Version 9. The NetFlow protocol itself has been superseded by Internet Protocol Flow Information eXport (IPFIX). Based on the NetFlow Version 9 implementation, IPFIX is on the IETF standards track with RFC 5101 (obsoleted by RFC 7011), RFC 5102 (obsoleted by RFC 7012), etc. which were published in 2008.

Equivalents

Many vendors other than Cisco provide similar network flow monitoring technology. NetFlow may be a prevalent name in the area of flow monitoring, because of Cisco dominant market share in the networking industry. NetFlow is thought to be a Cisco trademark (even though as of March 2012 it is not listed in Cisco Trademarks [3] ):

Also flow-tools collection of software [5] allows to process and manage NetFlow exports from Cisco and Juniper routers. [6]

Support

Vendor and typeModelsNetFlow VersionImplementationComments
Cisco IOS-XR routers CRS, ASR9000 old 12000 v5, v8, v9Software running on line card CPUComprehensive support for IPv6 and MPLS
Cisco IOS routers10000, 7200, old 7500v5, v8, v9Software running on Route Processorsupport for IPv6 or MPLS require recent model and IOS
Cisco Catalyst switches7600, 6500, 4500v5, v8, v9Dedicated hardware TCAM, also used for ACLs.Support for IPv6 on high-end models RSP720 and Sup720, but at most 128K or 256K flows per PCF card. [7]
Cisco Nexus switches5600, 7000, 7700v5, v9Dedicated hardware TCAM, also used for ACLs. Up to 512K flows. Support IPv4/IPv6/L2.MPLS not supported
Juniper legacy routers M-series, T-series, MX-series with DPCv5, v8Software running on Routing Engine, called software jflowIPv6 and MPLS not supported
Juniper legacy routers M-series, T-series, MX-series with DPCv5, v8, v9Software running on service PIC, called hardware jflow or sampledIPv6 or MPLS supported on MS-DPC, MultiService-PIC, AS-PIC2
Juniper routers MX-series with MPC-3D, FPC5 for T4000v5, IPFIX Hardware (trio chipset), called inline jflowIPv6 requires JUNOS 11.4R2 (back port target), MPLS support unknown, MPC3E excluded until 12.3, incorrect start time field causing incorrect data throughput result [8]
Nokia routers7750SRv5, v8, v9, v10 IPFIX Software running on Central Processor ModuleIPv6 or MPLS using IOM3 line cards or better
Huawei routersNE5000E NE40E/X NE80Ev5, v9Software running on service cardsSupport for IPv6 or MPLS is unknown
Enterasys SwitchesS-Serie [9] and N-Serie [10] v5, v9Dedicated hardwareIPv6 support is unknown
Flowmon Probes Flowmon Probe 1000, 2000, 4000, 6000, 10000, 20000, 40000, 80000, 100000v5, v9, IPFIX Software or hardware-acceleratedComprehensive support for IPv6 and MPLS, wire-speed
Nortel SwitchesEthernet Routing Switch 5500 Series (ERS5510, 5520 and 5530) and 8600 (Chassis-based)v5, v9, IPFIXSoftware running on line card CPUComprehensive support for IPv6
PC and Servers Linux FreeBSD NetBSD OpenBSD v5, v9, IPFIXSoftware like fprobe, [11] ipt-netflow, [12] pflow, [13] flowd, [14] Netgraph ng_netflow [15] or softflowdIPv6 support depend on the software used
VMware servers vSphere 5.x [16] v5, IPFIX (>5.1) [17] SoftwareIPv6 support is unknown
Mikrotik RouterOSRouterOS 3.x, 4.x, 5.x, 6.x [18] v1, v5, v9, IPFIX (>6.36RC3)Software and Routerboard hardwareIPv6 is supported using v9. Currently RouterOS does not include BGP AS numbers.

Variants

Cisco's NetFlow Security Event Logging

Introduced with the launch of the Cisco ASA 5580 products, NetFlow Security Event Logging utilizes NetFlow v9 fields and templates in order to efficiently deliver security telemetry in high performance environments. NetFlow Security Event Logging scales better than syslog while offering the same level of detail and granularity in logged events.[ citation needed ]

Monitoring based on standalone probes

NetFlow architecture using standalone probes. NewNetFlowApproach.png
NetFlow architecture using standalone probes.

NetFlow collection using standalone NetFlow probes is an alternative to flow collection from routers and switches. This approach can overcome some limitations of router-based NetFlow monitoring. The probes are transparently connected to the monitored link as a passive appliance using the TAP or SPAN port of the appliance.

Historically, NetFlow monitoring is easier to implement in a dedicated probe than in a router. However, this approach also has some drawbacks:

The easiest way to address the above drawbacks is to use a packet capture appliance inline in front of the router and capture all of the NetFlow output from the router. This method allows for storage of large amount of NetFlow data (typically many years worth of data) and does not require reconfiguration of the network.

NetFlow collection from dedicated probes is well suited for observation of critical links, whereas NetFlow on routers provides a Network-wide view of the traffic that can be used for capacity planning, accounting, performance monitoring, and security.

History

NetFlow was originally a Cisco packet switching technology for Cisco routers, implemented in IOS 11.x around 1996. It was originally a software implementation for the Cisco 7000, 7200 and 7500, [19] where it was thought as an improvement over the then current Cisco Fast Switching. Netflow was invented by Darren Kerr and Barry Bruin [20] from Cisco (U.S. patent # 6,243,667 ).

The idea was that the first packet of a flow would create a NetFlow switching record. This record would then be used for all later packets of the same flow, until the expiration of the flow. Only the first packet of a flow would require an investigation of the route table to find the most specific matching route. This is an expensive operation in software implementations, especially the old ones without Forwarding information base. The NetFlow switching record was actually some kind of route cache record, and old versions of IOS still refer to the NetFlow cache as ip route-cache.

This technology was advantageous for local networks. This was especially true if some of the traffic had to be filtered by an ACL as only the first packet of a flow had to be evaluated by the ACL. [21]

NetFlow switching soon turned out to be unsuitable for big routers, especially Internet backbone routers, where the number of simultaneous flows was much more important than those on local networks, and where some traffic causes many short-lived flows, like Domain Name System requests (whose source port is random for security reasons).

As a switching technology, NetFlow was replaced around 1995 by Cisco Express Forwarding. This first appeared on Cisco 12000 routers, and later replaced NetFlow switching on advanced IOS for the Cisco 7200 and Cisco 7500.

As of 2012, technologies similar to NetFlow switching are still in use in most firewalls and software-based IP routers. For instance the conntrack feature of the Netfilter framework used by Linux.

RFCs

See also

Related Research Articles

The Internet Control Message Protocol (ICMP) is a supporting protocol in the Internet protocol suite. It is used by network devices, including routers, to send error messages and operational information indicating success or failure when communicating with another IP address. For example, an error is indicated when a requested service is not available or that a host or router could not be reached. ICMP differs from transport protocols such as TCP and UDP in that it is not typically used to exchange data between systems, nor is it regularly employed by end-user network applications.

The Internet Protocol (IP) is the network layer communications protocol in the Internet protocol suite for relaying datagrams across network boundaries. Its routing function enables internetworking, and essentially establishes the Internet.

The Internet protocol suite, commonly known as TCP/IP, is a framework for organizing the set of communication protocols used in the Internet and similar computer networks according to functional criteria. The foundational protocols in the suite are the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and the Internet Protocol (IP). Early versions of this networking model were known as the Department of Defense (DoD) model because the research and development were funded by the United States Department of Defense through DARPA.

In computing, traceroute and tracert are diagnostic command-line interface commands for displaying possible routes (paths) and transit delays of packets across an Internet Protocol (IP) network.

In computer networking, the User Datagram Protocol (UDP) is one of the core communication protocols of the Internet protocol suite used to send messages to other hosts on an Internet Protocol (IP) network. Within an IP network, UDP does not require prior communication to set up communication channels or data paths.

The Routing Information Protocol (RIP) is one of the oldest distance-vector routing protocols which employs the hop count as a routing metric. RIP prevents routing loops by implementing a limit on the number of hops allowed in a path from source to destination. The largest number of hops allowed for RIP is 15, which limits the size of networks that RIP can support.

<span class="mw-page-title-main">Network address translation</span> Technique for making connections between IP address spaces

Network address translation (NAT) is a method of mapping an IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device. The technique was initially used to bypass the need to assign a new address to every host when a network was moved, or when the upstream Internet service provider was replaced but could not route the network's address space. It has become a popular and essential tool in conserving global address space in the face of IPv4 address exhaustion. One Internet-routable IP address of a NAT gateway can be used for an entire private network.

In computing, a stateful firewall is a network-based firewall that individually tracks sessions of network connections traversing it. Stateful packet inspection, also referred to as dynamic packet filtering, is a security feature often used in non-commercial and business networks.

In computer networking, Layer 2 Tunneling Protocol (L2TP) is a tunneling protocol used to support virtual private networks (VPNs) or as part of the delivery of services by ISPs. It uses encryption ('hiding') only for its own control messages, and does not provide any encryption or confidentiality of content by itself. Rather, it provides a tunnel for Layer 2, and the tunnel itself may be passed over a Layer 3 encryption protocol such as IPsec.

<span class="mw-page-title-main">Anycast</span> Network addressing and routing methodology

Anycast is a network addressing and routing methodology in which a single IP address is shared by devices in multiple locations. Routers direct packets addressed to this destination to the location nearest the sender, using their normal decision-making algorithms, typically the lowest number of BGP network hops. Anycast routing is widely used by content delivery networks such as web and name servers, to bring their content closer to end users.

sFlow, short for "sampled flow", is an industry standard for packet export at Layer 2 of the OSI model. sFlow was originally developed by InMon Corp. It provides a means for exporting truncated packets, together with interface counters for the purpose of network monitoring. Maintenance of the protocol is performed by the sFlow.org consortium, the authoritative source of the sFlow protocol specifications. The current version of sFlow is v5.

A network socket is a software structure within a network node of a computer network that serves as an endpoint for sending and receiving data across the network. The structure and properties of a socket are defined by an application programming interface (API) for the networking architecture. Sockets are created only during the lifetime of a process of an application running in the node.

Internet Protocol Flow Information Export (IPFIX) is an IETF protocol, as well as the name of the IETF working group defining the protocol. It was created based on the need for a common, universal standard of export for Internet Protocol flow information from routers, probes and other devices that are used by mediation systems, accounting/billing systems and network management systems to facilitate services such as measurement, accounting and billing. The IPFIX standard defines how IP flow information is to be formatted and transferred from an exporter to a collector. Previously many data network operators were relying on Cisco Systems' proprietary NetFlow technology for traffic flow information export.

In networking, a black hole refers to a place in the network where incoming or outgoing traffic is silently discarded, without informing the source that the data did not reach its intended recipient.

Path MTU Discovery (PMTUD) is a standardized technique in computer networking for determining the maximum transmission unit (MTU) size on the network path between two Internet Protocol (IP) hosts, usually with the goal of avoiding IP fragmentation. PMTUD was originally intended for routers in Internet Protocol Version 4 (IPv4). However, all modern operating systems use it on endpoints. In IPv6, this function has been explicitly delegated to the end points of a communications session. As an extension to the standard path MTU discovery, a technique called Packetization Layer Path MTU Discovery works without support from ICMP.

A routing protocol specifies how routers communicate with each other to distribute information that enables them to select paths between nodes on a computer network. Routers perform the traffic directing functions on the Internet; data packets are forwarded through the networks of the internet from router to router until they reach their destination computer. Routing algorithms determine the specific choice of route. Each router has a prior knowledge only of networks attached to it directly. A routing protocol shares this information first among immediate neighbors, and then throughout the network. This way, routers gain knowledge of the topology of the network. The ability of routing protocols to dynamically adjust to changing conditions such as disabled connections and components and route data around obstructions is what gives the Internet its fault tolerance and high availability.

Web Cache Communication Protocol (WCCP) is a Cisco-developed content-routing protocol that provides a mechanism to redirect traffic flows in real-time. It has built-in load balancing, scaling, fault tolerance, and service-assurance (failsafe) mechanisms. Cisco IOS Release 12.1 and later releases allow the use of either Version 1 (WCCPv1) or Version 2 (WCCPv2) of the protocol.

In packet switching networks, traffic flow, packet flow or network flow is a sequence of packets from a source computer to a destination, which may be another host, a multicast group, or a broadcast domain. RFC 2722 defines traffic flow as "an artificial logical equivalent to a call or connection." RFC 3697 defines traffic flow as "a sequence of packets sent from a particular source to a particular unicast, anycast, or multicast destination that the source desires to label as a flow. A flow could consist of all packets in a specific transport connection or a media stream. However, a flow is not necessarily 1:1 mapped to a transport connection." Flow is also defined in RFC 3917 as "a set of IP packets passing an observation point in the network during a certain time interval." Packet flow temporal efficiency can be affected by one-way delay (OWD) that is described as a combination of the following components:

RDMA over Converged Ethernet (RoCE) is a network protocol which allows remote direct memory access (RDMA) over an Ethernet network. There are multiple RoCE versions. RoCE v1 is an Ethernet link layer protocol and hence allows communication between any two hosts in the same Ethernet broadcast domain. RoCE v2 is an internet layer protocol which means that RoCE v2 packets can be routed. Although the RoCE protocol benefits from the characteristics of a converged Ethernet network, the protocol can also be used on a traditional or non-converged Ethernet network.

<span class="mw-page-title-main">SoftEther VPN</span> Open-source VPN client and server software

SoftEther VPN is free open-source, cross-platform, multi-protocol VPN client and VPN server software, developed as part of Daiyuu Nobori's master's thesis research at the University of Tsukuba. VPN protocols such as SSL VPN, L2TP/IPsec, OpenVPN, and Microsoft Secure Socket Tunneling Protocol are provided in a single VPN server. It was released using the GPLv2 license on January 4, 2014. The license was switched to Apache License 2.0 on January 21, 2019.

References

  1. ↑ Hofstede, Rick; Čeleda, Pavel; Trammell, Brian; Drago, Idilio; Sadre, Ramin; Sperotto, Anna; Pras, Aiko (2014). "Flow Monitoring Explained: From Packet Capture to Data Analysis with NetFlow and IPFIX". IEEE Communications Surveys & Tutorials. 16 (4): 2037–2064. doi:10.1109/COMST.2014.2321898. S2CID   14042725.
  2. ↑ "InterProjektWiki: NetFlow". Archived from the original on 2017-02-22.
  3. ↑ "Cisco Trademarks".
  4. ↑ "sFlow Products: Network Equipment". sFlow.org.
  5. ↑ "Adsr/Flow-tools". GitHub . 5 October 2021.
  6. ↑ "Adsr/Flow-tools". GitHub . 5 October 2021.
  7. ↑ "Cisco RSP720 Sup720 NetFlow characteristics". cisco.com. July 2010. Retrieved 2012-03-08.
  8. ↑ "pps and bps incorrect on Juniper j-flow". Aug 2012. Retrieved 2016-03-17.
  9. ↑ "NetFlow on Enterasys S-Serie" (PDF). enterasys.com. February 2012. Retrieved 2012-03-04.
  10. ↑ "NetFlow on Enterasys N-Serie" (PDF). enterasys.com. February 2012. Retrieved 2012-03-04.
  11. ↑ "fprobe".
  12. ↑ "ipt-netflow".
  13. ↑ Henning Brauer; Joerg Goltermann (2014-03-29). "pflow — kernel interface for pflow data export". BSD Cross Rererence. OpenBSD . Retrieved 2019-08-09.
  14. ↑ "flowd-0.9.1.20140828 – NetFlow collector". OpenBSD ports . 2019-07-17. Retrieved 2019-08-09.
  15. ↑ Gleb Smirnoff (2005). "ng_netflow — Cisco's NetFlow implementation". BSD Cross Rererence. FreeBSD . Retrieved 2019-08-09.
  16. ↑ "vSphere 5 New Networking Features - NetFlow - VMware vSphere Blog". 15 August 2011.
  17. ↑ "vSphere 51 Network Technical Whitepaper" (PDF). vmware.com. Retrieved 1 July 2023.
  18. ↑ "Manual:IP/Traffic Flow - MikroTik Wiki".
  19. ↑ "NetFlow Switching Enhancements Feature Module [Cisco IOS Software Releases 11.1] - Cisco Systems". www.cisco.com. Archived from the original on 2009-12-21.
  20. ↑ "Networking, Cloud, and Cybersecurity Solutions". Cisco. Retrieved 1 July 2023.
  21. ↑ "NetFlow, sFlow, and Flow Extensibility, Part 1". Kentik Blog. 28 March 2016. Retrieved 1 July 2023.
  22. ↑ Phaal, Peter; Lavine, Marc (July 2004). "sFlow Version 5". sFlow.org. Retrieved 2010-10-23.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.