Multi-Tap Network Packet Capturing
Overview
This section will demonstrate the use of Multi-Tap Network Packet Capture with NST. The NST WUI Network Packet Capture implementation supports simultaneous packet capture on up to 4 network interfaces (Quad Tap) per multi-tap capture session. Multi-segment network packet capture and decode analysis can be performed. NST uses the Wireshark network protocol analyzer suite for network packet capture and decode.
Theory Of Operation
Prior to commencing a multi-tap capture session, each enabled "Network Interface Tap" interface must be associated with a registered network interface on the NST probe. Up to 4 "Tap" interfaces can be enabled. A "Tap" interface is independent of any registered network interface on the NST probe. By assigning a separate network interface to each "Tap", it is possible to perform a network capture as packets flow across a networking device (e.g., router or firewall). One can also associate the same network interface to multiple "Tap" interfaces. This allows one to apply a different "Capture Filter" for the same network interface during packet capture.
A "Capture Data Directory" must also be selected for the resultant merged multi-tap capture file, multi-tap log file and if chosen, the individual tap member capture files. One should consider using a "RAM-based" file system for packet storage while capturing data packets on high traffic network segments. The Linux "RAMFS" or "TMPFS" file systems make excellent choices. The NST WUI provides convenient access for the creation of either one of these file systems during multi-tap capture setup.
The Wireshark light-weight network packet capture tool: "dumpcap" is used as the capture engine. Each enabled "Tap" interface will run a separate "dumpcap" process when the multi-tap capture session is started. Each "dumpcap" process can be configured to have a separate "Capture Filter" and other associated options.
A "Multi-Tap Merge Manager" process is also started when the multi-tap capture session is commenced. The "Multi-Tap Merge Manager" is responsible for monitoring each separate "dumpcap" process during the course of the multi-tap capture session. If any one multi-tap member "dumpcap" process terminates, the "Multi-Tap Merge Manager" will terminate all other "dumpcap" processes associated with the multi-tap capture session. Typically, a termination threshold (i.e. Duration, File Size or Packet Count) that has been satisfied is the usual reason for a "dumpcap" process to terminate. Once all "dumpcap" processes have been terminated, all "Tap" member capture files will then be merged using the Wireshark "mergecap" utility resulting with a multi-tap capture file. Each individual "Tap" member capture file will normally be deleted after the merge by the "Multi-Tap Merge Manager" unless an option was selected to disable removal of these files. A multi-tap capture log file will also be produced by merging all multi-tap member log files. The multi-tap capture log file contains relevant "Tap" member information and identity for historical multi-tap decode analysis.
The NST WUI is session state aware for each attached browser client. This means that each user or multiple users can in many areas of the NST WUI have their own separate configuration and operation. The NST WUI Multi-Tap Network Packet Capture implementation is fully session state capable. Based on this, multiple instances of a multi-tap capture session can occur simultaneously. This type of operation is typically more suited to an enterprise deployment of NST on systems configured with 8 or more network interfaces.
Capture Taps
When capturing packets at Gigabit Ethernet rates and one needs total visibility on the link, then a passive tap is required. Net Optics, a global leader in passive monitoring, makes an excellent 10/100/1000BaseT Tap (TP-CU3) for passively allowing access to monitor GigaBit traffic.
The TP-CU3 tap also allows for full-duplex traffic as if it were in-line (i.e, line rate non-blocking speeds), including Layer 1 and Layer 2 errors to be presented to the NST multi-tap capture interface. Typically a switch mirroring port may drop layer 1 and select layer 2 errors depending on what has been deemed as high priority for that particular switch.
Configuration 1: Multi-Tap Network Packet Capture Across A Firewall - NAT/PAT Traffic
The diagram depicted below shows an example Multi-Tap Capture Setup for monitoring GigaBit traffic across a firewall boundary. One may need to explore the capturing of packets as they transverse the firewall and undergo both Network and Port Address Translation (NAT/PAT).
Configuration 2: Multi-Tap Network Packet Capture - Traffic Between Gigabit Switches
The diagram displayed below shows an example Dual-Tap Capture Setup for monitoring network traffic between 2 Gigabit switches. In this case a generic notebook computer was used and configured with 3 network interfaces (A built-in Gigabit LAN adapter, a Gigabit LAN adapter PC-Card and a built-in 802.11g/n wireless adapter for secure remote access and control of NST).
Example 1: Multi-Tap Network Packet Capture Session Step-By-Step
This section will demonstrate a Multi-Tap Network Packet Capture session using 2 network adapters (eth1 and eth2) as the target capture interfaces. We will concentrate on email traffic (POP and SMTP) occurring on network: "172.28.8.0/24". It is assumed that the NST probe has sufficient network interface adapters (at least 2 in this case) to perform this capture and that these interfaces are attached to a network tap similar to what is shown in the diagrams above or some type of port mirroring (e.g., SPAN - Switched Port Analyzer) configuration.
Step: 1 NST WUI Multi-Tap Network Packet Capture Page
From the NST WUI Start page go to the NST WUI Multi-Tap Network Packet Capture page. This is accomplished by using the NST WUI Menu. Go to "Networking" then "Protocol Analyzers" and then go to the "Multi-Tap Network Packet Capture" page as shown below.
Step: 2 Multi-Tap Interfaces & Data Directory Selection
Use the navigational aid link: [Select Tap Interfaces/Directory] to go to the "Multi-Tap Interfaces & Data Directory Selection" section. This is where you select the target capture interfaces and data directory on the NST probe when you will store your multi-tap capture. We will enable Network Interface Taps: "Tap0" and "Tap1" and assign each one to a registered network interface on the NST probe.
Select interface: "eth1" for Tap0 and select interface: "eth2" for Tap1. For now use the default capture data directory: "/var/nst/wuiout/wireshark". Your configuration should look similar to the NST WUI image caption above. Next select the "Select Multi-Tap Interfaces/Capture Data Directory" button to make your configuration known to NST.
One can also click on a "NIC Adapter" icon to view and control a specific network interface adapter. The caption below shows information for NIC Adapter: "eth2" and results from running the "ethtool eth2" command. Even though the interface state is: "Down", a link has been established and detected on this interface. This is valuable information to know prior to capturing on a particular network interface.
Both interfaces are initially shown in the "Down" state indicated by the red down arrow superimposed on the "NIC Adapter" icons. The network interface will automatically be put in the "Up" state prior to starting a capture session.
Step: 3 Start Multi-Tap Capture Form
After you have selected your "Tap Interfaces" and "Capture Data Directory" next go to the "Start Multi-Tap Capture" section. For this demonstration we will use a "dumpcap" capture filter expression for "POP" and "SMTP" traffic on network: "172.28.8.0/24". Since we selected to use only 2 Taps (Tap0 and Tap1), then only 2 Taps will be dynamically available for configuration out of a possible 4 Taps in the "Start Multi-Tap Capture" form as shown below.
Fill in the form similar to what is shown above. Optionally use the action: ['dumpcap' Capture Filter Expressions] to help fill in the "dumpcap" capture filter expression field for each Tap. Each Tap can have a unique "dumpcap" capture filter expression. Use: "Monitor Start" as one of the "WUI Startup Options". Adjust your capture termination thresholds accordingly. In this case, the capture will terminate automatically when either of the following thresholds have been exceeded: a "10 seconds" capture time has elapsed, a file size of: "4MB" has been reached or "100" network packets have been collected.
Step: 4 Multi-Tap Network Packet Capture Start
Once you are satisfied with values in the "Start Multi-Tap Network Packet Capture" form we will now commence a "Multi-Tap Network Packet Capture" session. Click on the "Multi-Tap Network Packet Capture Start" button to begin the capture session. An automatic sequence of NST WUI Multi-Tap Network Packet Capture pages will now appear until the "Pending Multi-Tap Capture" page that is shown below is reached.
Near real-time dynamic updates using AJAX will occur until one of the termination thresholds is reached. Once termination has been reached, another automatic sequence of NST WUI Multi-Tap Network Packet Capture pages will now appear until the "Multi-Tap Capture Summary" section is now displayed as show below:
One can see from the Wireshark "capinfos" summary output that the packet count termination threshold of: "100 Packets" caused the capture session to end.
Step: 5 Multi-Tap Network Packet Capture Decode Analysis
At this point we are now ready to perform a decode of the packet capture. Use the [New Decode] navigation aid link to go to the "Multi-Tap Text-Based Protocol Analyzer Decode" section.
Example 2: Multi-Tap Network Packet Capture Across A Firewall Boundary - NAT/PAT Traffic
This example will explore network packet traffic as it flows across a firewall device. We will use a remote "SSH" client request and "SSH" server response for this demonstration and explore the NAT/PAT translation that occurs by decoding the capture. The following diagram shows the network environment for this multi-tap capture. The "SSH" client is located remotely on the Internet using IP Address: 24.97.150.194 and the "SSH" server is NAT/PAT translated using IP Address: 69.205.41.87:22222 its real internal IP Address is: 10.222.222.14:22.
We will be using 3 tap interfaces (Tap0 - 2) for this multi-tap capture. The Hub works in "Half-Duplex" mode and presents both the Transmit and Receive network traffic from the "Dirty" (unfiltered) side of the Firewall device to: "Tap0: eth2". Taps "1" and "2" will be connected to a Net Optics Gigabit copper tap: TP-CU3 for capturing network traffic between the Firewall device and the Gigabit Switch.
Multi-Tap Interfaces & Data Directory Selection Form: NAT/PAT SSH Traffic
Prior to starting up a capture session, each enabled "Network Interface Tap" must be associated with a registered network interface on the NST probe. In this example network interface: "eth2" is assigned to: "Tap0", network interface: "eth1" is assigned to: "Tap1" and network interface: "eth3" is assigned to: "Tap2". Leave "Tap3" Disabled.
A "Capture Data Directory" must also be selected for capture and log file storage. In this case we have chosen directory: "/tmp/pkt/wikidemo". Use the "Select Multi-Tap Interfaces/Capture Data Directory" button to register this configuration with the current Multi-Tap Network Packet Capture session. This will build out the appropriate corresponding "Multi-Tap Start Capture" form which is shown below.
Multi-Tap Network Interface Information (eth2): NAT/PAT SSH Traffic
The "Multi-Tap Interfaces & Data Directory Selection" form also includes network interface action icons allowing one to view specific information and control various properties of a network interface. Dynamic counter values and setting updates will occur periodically via AJAX.
The above view was enabled by clicking on the "eth2" NIC icon. This pop-up view can be dragged around within your browser's window. This is accomplished by first moving your mouse cursor to the top header control area (Note: the cursor icon should change to the "move" icon) and then secondly while holding down the "left" mouse control button the view can be dragged to your desired location.
A pop-up network interface view for each registered network interface can be shown simultaneously. The dynamic update period for counter values and settings can be changed using the parameter value: "Network Interface Information Update Interval" found on the NST WUI: "Global/Session Configuration Management" page section: "DOM Session Configuration".
Multi-Tap Start Capture Form: NAT/PAT SSH Traffic
After registering the enabled "Network Interface Taps" and "Capture Data Directory" above, the "Multi-Tap Start Capture" form should contain input fields for "Tap0", "Tap1" and "Tap2" as shown below. For this example we chose to use a small "Packet Count" termination threshold value of "8 Packets" to control when the multi-tap capture session will end. Once any "dumpcap" process has captured "8 Packets" or more, the entire multi-tap capture session will complete. The other 2 capture termination thresholds: "Duration" and "File Size" will be set to large values and will unlikely be the cause of capture termination. A quick way to set the capture termination thresholds shown is to select the action: [Long Capture Session] and then adjust the "Packet Count" termination threshold to: "8 Packets".
On the public external side a "dumpcap" capture filter expression: -f 'host 69.205.41.87 and (tcp port 22222)' is applied to network interface tap: "tap0:eth2". This filter states that only packets to or from IP Address: "69.205.41.87" And having a source or destination TCP/IP port: "22222" will be captured.
On the private internal side the same "dumpcap" capture filter expression: -f 'host 10.222.222.14 and (tcp port ssh)' is applied to network interface tap: "tap1:eth1" and tap: "tap2:eth3". This filter states that only packets to or from IP Address: "10.222.222.14" And having a source or destination TCP/IP port: "ssh (22)" will be captured.
Hint: A convenient action: ['dumpcap' Capture Filter Expressions] can be used to automate the process of filling in a desired capture filter expression. This action allows one to create, edit, restore, save or load your favorite set of capture filter expressions. The action: [Replicate] can then be used to populated the selected capture filter expression to all other multi-tap member capture filter expression fields.
There are no additional "dumpcap" options used for this example.
For historical purposes, it is important that one annotates the capture appropriately. Each network tap interface section provides an annotation entry field. There is also a multi-tap capture session annotation entry field. Annotations are very useful and will appear in various tooltip status views throughout the NST WUI Network Packet Capture implementation and can be later searched on using the "Network Packet Capture Management & Status" page.
We have chosen not to remove the individual capture files produced from each "dumpcap" process after the multi-tap capture merge has taken place. This is indicated by the checkbox selection next to the "Disable Tap Capture/Log Removal" text. One may want to retain the individual capture file components as well as the multi-tap capture to perform a rigorous comprehensive decode analysis requiring original data.
The "Monitor Start" option with an update period of "5" seconds has also been selected. This allows for dynamic capture status updates to occur every "5" seconds using AJAX until the multi-tap capture session terminates.
Multi-Tap Pending Capture: NAT/PAT SSH Traffic
The multi-tap network packet capture session is now configured to start. The "Multi-Tap Network Packet Capture Start" button shown above is used to commence the capture. Since the "Monitor Start" option was selected, a series of pages will automatically sequence until the Pending Multi-Tap Capture page appears below. This page will remain active until the packet capture count threshold of "8" packets is reached. Periodic updates occurring every "5" seconds via an AJAX transaction will refresh this page with current capture status information.
Each individual tap "dumpcap" command is also shown in its entire form. One may find it convenient to "copy" and "paste" a complete "dumpcap" command for command line usage or to check for proper options passed to a particular "dumpcap" command.
Hint 1: Click on a NIC icon to view network interface information for that capture interface during a real-time multi-tap network packet capture session. One may observe if traffic is being received or not on a network interface even though the capture filter expression in effect is discarding the capturing of these packets.
Hint 2: Click on a Tap interface button (e.g., "Tap2: eth3") to perform decode analysis on the associated "dumpcap" capture file using the NST WUI "Single-Tap Network Packet Capture" implementation. This procedure can occur during a live multi-tap network packet capture session. Use this technique to ascertain if sufficient data has already been captured. If this is the case, one may then terminate the capture session manually.
The "Terminate Multi-Tap Capture" button allows one to manually stop the current multi-tap network packet capture session in progress.
Multi-Tap Capture NST WUI And Capinfos Summary Section: NAT/PAT SSH Traffic
Once a multi-tap network packet capture session has completed, decode analysis can be performed on the multi-tap capture file. The caption below shows the output from the Wireshark "capinfos" command and NST WUI summary information for the multi-tap capture file: "/tmp/pkt/wikidemo/capture_mtap.cap". This section is displayed initially right after the multi-tap capture session has terminated.
Multi-Tap Capture ToolTip Summary : NAT/PAT SSH Traffic
The caption below also shows a summary view of the current multi-tap capture file. One can enable this view by hovering your mouse pointer over the Multi-Tap Capture Status: "Capture Available" header information at the top of the Multi-Tap Network Packet Capture page.
Note 1: The total number of packets captured: "22" was the sum of "8" packets captured on network interface: "Tap0: eth2", "8" packets captured on "Tap1: eth1" and "6" packets captured on "Tap2: eth3".
Note 2: The multi-tap capture termination threshold of "8" packets was reached on both network interface "Tap0" and "Tap1".
We will now demonstrate various types of decodes and analysis using the multi-tap capture file. Use the [New Decode] navigation link to move to the "Multi-Tap Text-Based Protocol Analyzer Decode" section.
Multi-Tap Capture Decode Form: NAT/PAT SSH Traffic
Multi-Tap Capture Decode Summary: NAT/PAT SSH Traffic
Multi-Tap Capture PDML TCP/IP Decode: NAT/PAT SSH Traffic
Multi-Tap Capture PDML Network Packet Details Decode: NAT/PAT SSH Traffic
Multi-Tap Capture PDF Decode: NAT/PAT SSH Traffic
