HowTo Geolocate ntop Data

Overview

This HowTo explains the procedure for setting up an ntop session and producing on demand host geolocations rendered on either a Mercator World Map projection or on a KML Earth Browser such as Google Earth, Google Maps or Marble.

One of the goals of the NST WUI is to provide a web-based frontend to numerous open source network security applications. Trying to build out a web-based interface that has a common look-and-feel across the vast spectrum of applications is a daunting task. Once an NST WUI interface is mastered, it will become a routine task for the network security administrator to use it across different NST systems and network infrastructure environments.

Before diving into producing ntop Hosts Geolocations, one needs to understand best practices on how to setup an ntop session as a Host data source collector. This first involves getting ntop up and running using its NST WUI management interface and then secondly controlling how much data ntop is configured to collect using ntop administrative settings.

Note: The NST WUI only supports setting up and managing one instance of an ntop session.

ntop Setup Management

This section describes how to setup an ntop session using the NST WUI. The screen shot below shows one how to locate the ntop Management page using the NST menu bar.

NST WUI ntop Management Menu Selection

Network Interface(s)

The input and selection fields provided by the NST WUI management interface will be explained so that one can quickly start up ntop. One or more network interfaces can be selected (i.e., click on an associated check box) to be monitored by the ntop application. One can click on a NIC adapter icon to examine detailed counter data and interface controls associated with a network adapter. This feature can be particularly useful if one wants to know if traffic is currently occurring on a network interface prior to bring up the ntop session. For best results when geolocating Hosts using ntop data, select a network interface to monitor that has a public IP Address presence (e.g., Network Tap or SPAN port associated with the dirty side of a corporate firewall or a web server farm located in a DMZ).

A network interface does not have to have a binding IP Address (i.e., stealth interface) and can also be in the Down state. The NST WUI will bring all selected network interfaces from the Down state to the Up state prior to starting the ntop session.

Note: When starting ntop as a service at boot time, all configured ntop network interfaces must be in the Up state. See the section: Configuring A Stealth Network Interface for more information.

HTTP / HTTPS Access

Access to the ntop user interface can be configured to use either the HTTP and/or the HTTPS protocol. This is done by setting a non-zero Access Port value for the associated protocol. An access port value of Zero (0) will disable the use of the protocol. Typically, an access port value of 3000 is used for HTTP and an access port value of 3001 is used for HTTPS.

Note: Access to the ntop user interface for the configuration shown in the screen shot below would be: https://10.222.222.124:3001

Setup ntop Options

The script: "setup_ntop" is used to support the NST WUI by creating a runtime execution environment for ntop and starting up the ntop daemon. The Setup ntop Options field allows one to add setup and runtime options associated with the script: "setup_ntop". Normally this field does not need to be altered. If you do alter this field, make sure that you specify a runtime directory ("--rdir /var/nst" shown below). For detailed additional setup ntop options, expand the Setup ntop Options section on this NST WUI page.

NST WUI ntop Setup Management

ntop Options

The ntop Options field is used to specific additional ntop parameters that are not provided automatically by the NST WUI ntop Setup Management interface (e.g., Add an option so that Idle hosts are not purged from memory and set the Refresh page time in seconds to 1 min: "--sticky-hosts --refresh-time 60"). For detailed information about these ntop options, expand the ntop Command Line Options section on this NST WUI page.

ntop Annotation

Enter a Map Title or a short (i.e., 22 characters or less) Annotation to describe this ntop session (e.g., Wiki Web Site Traffic). This text will appear on rendered geolocation map projections or within a geolocation KML document description balloon.

ntop GeoIP Options

In order for both the ntop application and NST to perform Host geolocations, a database must exist to look up the earth coordinate (i.e., latitude, longitude) for a given IPv4 Address. The GeoIP database from MaxMind is used by ntop. There are two approaches for downloading the GeoIP database. One can simply choose the Download radio button or follow the method for downloading the GeoIP database found on the Geolocation Tools & Management page. Use the Standard radio button to disable downloading the GeoIP database. Use the Update radio button to refresh an existing GeoIP database. Lastly, use the Remove radio button to delete the entire GeoIP database from the NST system.

View / Edit ntop Configuration File

One can also use the NST File Editor to view or make permanent changes to the ntop configuration file: "/etc/ntop.conf". Click on the 'Edit ntop Conf' button to perform this task.

Starting Up An ntop Session

Once you are satisfied with your ntop configuration options, commence an ntop session by clicking on the 'Start ntop' button. An intermediate page displaying the results of starting up the ntop session will be presented next as shown below. Click on the 'Return' button to continue with the setup sequence.

NST WUI: ntop Session Setup

ntop Setup In Progress

The ntop session setup time will take longer if the GeoIP database needs to downloaded or updated. You can monitor the progression of the setup using the 'Monitor ntop Setup Log' button or periodically click on the 'Refresh' button to see if the session setup has completed.

NST WUI ntop Setup Progress

ntop Runtime Management

Once an ntop session is up and running, one can enter the ntop user interface, manage the ntop daemon or produce on demand host geolocations.

NST WUI ntop Runtime Management

Geolocate ntop Hosts

This section will show how to produce on demand host geolocations rendered on either a Mercator World Map projection or on a KML Earth Browser such as Google Earth, Google Maps or Marble. The NST script: nstgeolocate is used to support the NST WUI when generating host geolocations.

Mercator World Map

Use the 'ntop Hosts - World Map' button to generate on demand Geolocated ntop Hosts rendered on a Mercator World Map projection. One can customize the host geolocation marker type and color. One can also override the Map Annotation if necessary. The following Progress Generation page is displayed while the ntop Hosts are in the process of being geolocated and rendered on the Mercator World Map projection.

Note: One may need to allow pop-up window generation on your browser for the NST host so that the Progress Generation page can be displayed correctly.

You have some control while waiting for the map generation on the progress page. If you do nothing, the process will automatically leave the progress page and display the map. You can use the "Cancel" button to cancel the map generation and close the progress page. You can use the "Hold" button to prevent the automatic transition to the map display. This is useful if you would like to use the "View Generate Log" button to see all commands and log output for geolocating ntop Hosts on the Mercator World Map projection.

ntop Progress Generation Page

Once the process of geolocating and rendering hosts on the Mercator World Map has completed, the resultant map will be displayed. The ntop World Map Hosts image shown below was produced using marker type: "star" with marker color: "blue".

ntop Mercator World Map Hosts

Use the Image Control Button Grid located in the upper left-hand corner of the map to both position and resize the map projection. Hover your mouse pointer over the information icon to reveal a detailed summary report associated with the Geolocated ntop Hosts. The map below was zoomed in and shows the information tooltip.

ntop Mercator World Map Hosts - Zoomed In with Information Tooltip Balloon

World Map: Image Control Button Grid

The caption below describes each button associated with the World Map Image Control Grid. For supported browsers, try creating a full screen map view by using a combination of the 'F11' function key and the 'Fit To Browser Window Viewport' button.

World Map Image Control Button Grid Description

World Map: Drag & Resize

Try resizing and positioning the Mercator World Map using a combination of your Left Mouse button and the Shift key while dragging the mouse. See the description in the 'ntop Mercator World Map Hosts - Zoomed In with Information Tooltip Balloon' screen shot above. The Drag & Resize functionality uses a JavaScript library created by Walter Zorn. Documentation for the library can be view here: DHTML API, Drag & Drop for Images and Layers

KML Document (Google Earth)

Use the 'ntop Hosts - KML' button to generate on demand Geolocated ntop Hosts rendered on a KML Earth Browser such as Google Earth. One can override the Map Annotation if necessary. Generating a KML document can take some time depending on the number of hosts to geolocate. We have observed that it takes roughly 50 seconds to geolocate 1000 hosts. Once the document has completed, you can view it in Google Earth (See the HowTo document: HowTo Setup Your Client System To View Geolocation Data).

Each host that was geolocated appears as a host marker and contains a 'Host Description' balloon depicting selective ntop network traffic statistics. Click on a host marker to reveal the 'Host Description' balloon. Hyperlinks are also provided to the ntop (Host Collector) user interface and the NST WUI 'IP Tools' page for additional network processing using the host's IP Address.

ntop Hosts KML Document

ntop Hosts KML Document - with Description Balloon

ntop Hosts KML Document - with Host Balloon

When using Google Earth, you can view the 'Document Description' balloon by clicking on the generated KML ntop Hosts place found under Temporary Places within the sidebar on the left-hand side. You can also expand the ntop Hosts place to explore all geolocated hosts and associated network statistics.

IP Geolocation Adjustments

Use the ' IP Geolocate Configure' button to manage the global geolocation policy for this NST system. This allows one to make latitude and longitude coordinate adjustments, configure Private IPv4 Address & Network coordinate locations and select a geolocation database source. In addition, one can also download and manage the MaxMind "GeoIP Country Edition", the enhanced "GeoIP Lite City Edition" and the "GeoIP AS Number Edition" data sets.

Auto-Generated ntop Hosts Geolocations

The NST WUI provides the "nstgeolocate Session Manager" page to manage a variety of geolocation types on an NST system. One can setup and auto-generate ntop Hosts Geolocations from one or more (i.e., local or remote) ntop sessions on this page. See section HowTo Automate & Manage NST Geolocation Results for details.

Setting The ntop Host Collection Window Size

There are preference settings in ntop which allows for how long ntop maintains information about a particular host prior to purging it from its internal buffers. These settings control how many hosts will be collected when the nstgeolocate script interrogates the ntop session. This in turn will determine how many hosts get geolocated and rendered on a map. Generating a KML Document takes considerably more time to produce than a Mercator World Map. One needs to tweak these parameters and tailor them to the type of network traffic being monitored so that excessive processing does not occur when geolocating too many hosts at one time. As a rule of thumb, for low volume traffic, open up the collection window size and for high volume traffic, be more conservative on the time interval for the collection window. When auto-generating a Mercator World Map, there are other controls that allow for host geolocation rendering to occur for a long duration (i.e., 1 day, 1 week, 1 month or a year). This allows one to maintain a short ntop collection time window size and still generate a map using a long time period for host geolocations. See section Auto-Generated ntop Hosts Geolocations for details.

Editing ntop preference settings can be found using the menu shown below. You will need to log in as user: 'admin' and then use the password for the NST 'root' user. The ntop 'admin' password is set when the NST script: nstpasswd is run.

Access ntop Edit Preferences from Menu

There are 2 ntop preference settings that control host collection window size. One is: 'purge_host.seconds_idle_with_sessions' and the other: 'purge_host.seconds_idle_with_no_sessions'. These settings take a value in 'seconds'. Typically, one sets these values identical. The image below shows a collection time window size configured for 1 day (60 x 60 x 24 = 86400 - Number of seconds in 1 day).

ntop Edit Preferences - Collection Window Size