View source for HowTo Geolocate ntop Data

__TOC__
== '''Overview''' ==

This '''HowTo''' explains the procedure for setting up an '''[http://www.ntop.org ntop]''' session and producing on demand '''host''' geolocations rendered on either a '''Mercator World Map''' projection or on a '''[http://en.wikipedia.org/wiki/Keyhole_Markup_Language KML]''' '''Earth Browser''' such as '''[http://earth.google.com Google Earth]''', '''[http://maps.google.com Google Maps]''' or '''[http://edu.kde.org/marble Marble]'''.

One of the goals of the NST WUI is to provide a web-based frontend to numerous open source network security applications. Trying to build out a web-based interface that has a common look-and-feel across the vast spectrum of applications is a daunting task. Once an NST WUI interface is mastered, it will become a routine task for the network security administrator to use it across different NST systems and network infrastructure environments.

Before diving into producing '''[http://www.ntop.org ntop]''' '''Hosts Geolocations''', one needs to understand best practices on how to setup an '''[http://www.ntop.org ntop]''' session as a '''Host''' data source collector. This first involves getting '''[http://www.ntop.org ntop]''' up and running using its NST WUI management interface and then secondly controlling how much data '''[http://www.ntop.org ntop]''' is configured to collect using '''[http://www.ntop.org ntop]''' administrative settings.

<div class="centerBlock"><div class="noteMessage">'''Note:''' The NST WUI only supports setting up and managing <u>one</u> instance of an '''[http://www.ntop.org ntop]''' session.</div></div>

== '''ntop Setup Management''' ==

This section describes how to setup an '''[http://www.ntop.org ntop]''' session using the '''NST WUI'''. The screen shot below shows one how to locate the '''[http://www.ntop.org ntop] Management''' page using the NST menu bar.

[[Image:Ntop-menu.png|center|frame|NST WUI ntop Management Menu Selection]]

=== '''Network Interface(s)''' ===

The input and selection fields provided by the NST WUI management interface will be explained so that one can quickly start up '''[http://www.ntop.org ntop]'''. One or more network interfaces can be selected (i.e., click on an associated check box) to be monitored by the '''[http://www.ntop.org ntop]''' application. One can click on a '''NIC''' adapter icon to examine detailed counter data and interface controls associated with a network adapter. This feature can be particularly useful if one wants to know if traffic is currently occurring on a network interface prior to bring up the '''[http://www.ntop.org ntop]''' session. For best results when geolocating '''Hosts''' using '''[http://www.ntop.org ntop]''' data, select a network interface to monitor that has a public '''IP Address''' presence (e.g., [[Multi-Tap_Network_Packet_Capturing#Network_Tap | Network Tap or SPAN]] port associated with the dirty side of a corporate firewall or a web server farm located in a DMZ).

A network interface does not have to have a binding IP Address (i.e., [[Network_Setup_Tips#Configuring_A_Stealth_Network_Interface | stealth interface]]) and can also be in the '''Down''' state. The NST WUI will bring all ''selected'' network interfaces from the '''Down''' state to the '''Up''' state prior to starting the '''[http://www.ntop.org ntop]''' session.

<div class="centerBlock"><div class="noteMessage">'''Note:''' When starting '''[http://www.ntop.org ntop]''' as a service at boot time, all configured '''[http://www.ntop.org ntop]''' network interfaces must be in the '''Up''' state at a minimal. See section: [[Network_Setup_Tips#Configuring_A_Stealth_Network_Interface | Configuring A Stealth Network Interface]] with NST for more information.</div></div>

=== '''HTTP / HTTPS Access''' ===
Access to the '''[http://www.ntop.org ntop]''' user interface can be configured to use either the '''HTTP''' and/or the '''HTTPS''' protocol. This is done by setting a non-zero '''Access Port''' value for the associated protocol. An access port value of '''Zero (0)''' will disable the use of the protocol. Typically, an access port value of '''3000''' is used for '''HTTP''' and an access port value of '''3001''' is used for '''HTTPS'''. 

<div class="centerBlock"><div class="noteMessage">'''Note:''' Access to the '''[http://www.ntop.org ntop]''' user interface for the configuration shown in the screen shot below would be: '''''<nowiki>https://10.222.222.124:3001</nowiki>'''''</div></div>

=== Setup ntop Options ===
The script: "'''setup_ntop'''" is used to support the NST WUI by creating a runtime execution environment for '''[http://www.ntop.org ntop]''' and starting up the '''[http://www.ntop.org ntop]''' daemon. The '''Setup ntop Options''' field allows one to add setup and runtime options associated with the script: "'''setup_ntop'''". Normally, other than specifying the ''runtime'' directory: "'''--rdir /var/nst'''", this field does not need to be altered. For detailed additional setup '''[http://www.ntop.org ntop]''' options, expand the '''Setup ntop Options''' section on this NST WUI page. 

[[Image:Ntop-setup.png|center|frame|NST WUI ntop Setup Management]]

=== ntop Options ===
The '''ntop Options''' field is used to specific additional '''[http://www.ntop.org ntop]''' parameters that are not provided automatically by the NST WUI '''ntop Setup Management''' interface (e.g., Add an option so that ''Idle'' hosts are not purged from memory and set the ''Refresh'' page time in seconds to 1 min: "'''--sticky-hosts --refresh-time 60'''"). For detailed additional '''[http://www.ntop.org ntop]''' options, expand the '''ntop Command Line Options''' section on this NST WUI page.

=== ntop Annotation ===
Enter a '''Map Title''' or a short (i.e., 22 characters or less) '''Annotation''' to describe this '''[http://www.ntop.org ntop]''' session (e.g., '''Wiki Web Site Traffic'''). This text will appear on rendered geolocation map projections or within a geolocation '''[http://en.wikipedia.org/wiki/Keyhole_Markup_Language KML]''' document description balloon.

=== ntop GeoIP Options ===
In order for both the '''[http://www.ntop.org ntop]''' application and NST to perform '''Host''' geolocations, a database must exist to look up the earth coordinate (i.e., latitude, longitude) for a given IPv4 Address. The '''GeoIP''' database from '''[http://www.maxmind.com/ MaxMind]''' is used by '''[http://www.ntop.org ntop]'''. There are two approaches for downloading the '''GeoIP''' database. One can simply choose the '''Download''' radio button or follow the method for ''downloading'' the '''GeoIP''' database found on the '''[[HowTo_Setup_The_NST_System_To_Geolocate_Data#Download_Geolocation_Databases | Geolocation Tools & Management]]''' page. Use the '''Standard''' radio button to ''disable'' downloading the '''GeoIP''' database. Use the '''Update''' radio button to ''refresh'' an existing '''GeoIP''' database. Lastly, use the '''Remove''' radio button to delete the entire '''GeoIP''' database from the NST system.

=== View / Edit ntop Configuration File ===
One can also use the '''NST File Editor''' to view or make permanent changes to the '''[http://www.ntop.org ntop]''' configuration file: "'''/etc/ntop.conf'''". Click on the ''''Edit ntop Conf'''' button to perform this task.

=== Starting Up An ntop Session ===
Once you are satisfied with your '''[http://www.ntop.org ntop]''' configuration options, commence an '''[http://www.ntop.org ntop]''' session by clicking on the ''''Start ntop'''' button. An intermediate page displaying the results of starting up the '''[http://www.ntop.org ntop]''' session will be presented next as shown below. Click on the ''''Return'''' button to continue with the setup sequence.

[[Image:Ntop-setup-return.png|center|frame|NST WUI: ntop Session Setup]]

=== '''ntop Setup In Progress''' ===

The '''[http://www.ntop.org ntop]''' session setup time will take longer if the '''GeoIP''' ''database needs to downloaded'' or ''updated''. You can monitor the progression of the setup using the ''''Monitor ntop Setup Log'''' button or periodically click on the ''''Refresh'''' button to see if the session setup has completed. 

[[Image:Ntop-setup-progress.png|center|frame|NST WUI ntop Setup Progress]]

== '''ntop Runtime Management''' ==

Once an '''[http://www.ntop.org ntop]''' session is up and running, one can enter the '''[http://www.ntop.org ntop]''' user interface, manage the '''[http://www.ntop.org ntop]''' daemon or produce on demand '''host''' geolocations.

[[Image:Ntop-runtime.png|center|frame|NST WUI ntop Runtime Management]]


== '''Geolocate ntop Hosts''' ==
This section will show how to produce on demand '''host''' geolocations rendered on either a '''Mercator World Map''' projection or on a '''[http://en.wikipedia.org/wiki/Keyhole_Markup_Language KML]''' '''Earth Browser''' such as '''[http://earth.google.com Google Earth]''', '''[http://maps.google.com Google Maps]''' or '''[http://edu.kde.org/marble Marble]'''. The NST script: '''[http://nst.sourceforge.net/nst/docs/scripts/nstgeolocate.html nstgeolocate]''' is used to support the NST WUI when generating host geolocations.

=== '''Mercator World Map''' ===
Use the ''''ntop Hosts - World Map'''' button to generate on demand '''Geolocated ntop Hosts''' rendered on a '''Mercator World Map''' projection. You can customize the host geolocation marker type and color. One can also override the '''Map Annotation''' if necessary. The following '''Progress Generation''' page is displayed while the '''ntop Hosts''' are in the process of being geolocated and rendered on the '''Mercator World Map''' projection.

[[Image:Ntop-progress-page.png|center|frame|ntop Progress Generation Page]]

Once the process of geolocating and rendering hosts on the '''Mercator World Map''' has completed, the resultant map will be shown in a new tab or browser window. The '''ntop World Map Hosts''' image shown below was produced using marker type: "'''star'''" with marker color: "'''blue'''". 

[[Image:Ntop-wm-hosts.png|center|frame|ntop Mercator World Map Hosts]]
Use the '''Image Control Button Grid''' located in the upper left-hand corner of the map to both position and resize the map projection. Hover your mouse pointer over the information icon to reveal a detailed description associated with the '''Geolocated ntop Hosts'''. The map below was zoomed in and also is showing the information tooltip balloon.  

[[Image:Ntop-wm-hosts-tt.png|center|frame|ntop Mercator World Map Hosts - Zoomed In with Information Tooltip Balloon]]

==== '''World Map: Image Control Button Grid''' ====
The caption below describes each button associated with the '''World Map Image Control Grid'''. For supported browsers, try creating a full screen map view by using a combination of the ''''F11'''' function key and the ''''Fit To Browser Window Viewport'''' button.
[[Image:Button-grid.png|center|frame|World Map Image Control Button Grid Description]]

==== '''World Map: Drag & Resize''' ====
Try resizing and positioning the '''Mercator World Map''' using a combination of your '''Left Mouse''' button and the '''Shift''' key while ''dragging'' the mouse. See the description in the ''''ntop Mercator World Map Hosts - Zoomed In with Information Tooltip Balloon'''' screen shot above. The '''Drag & Resize''' functionality uses a JavaScript library created by '''Walter Zorn'''. Documentation for the library can be view here: '''[http://gualtierozorni.altervista.org/dragdrop/dragdrop_e.htm DHTML API, Drag & Drop for Images and Layers]'''

=== '''KML Document''' ===
Use the ''''ntop Hosts - KML'''' button to generate on demand '''Geolocated ntop Hosts''' rendered on a '''[http://en.wikipedia.org/wiki/Keyhole_Markup_Language KML]''' '''Earth Browser''' such as '''[http://earth.google.com Google Earth]'''. One can override the '''Map Annotation''' if necessary. Generating a '''[http://en.wikipedia.org/wiki/Keyhole_Markup_Language KML]''' document can take some time depending on the number of hosts to geolocate. We have observed that to geolocate 1000 hosts, it takes roughly between 40 and 50 seconds to complete.

Each host marker contains a ''''Host Description'''' balloon depicting selective '''[http://www.ntop.org ntop]''' network traffic statistics information. Click on a host marker to reveal the ''''Host Description'''' balloon. Hyperlinks are also provided to the '''[http://www.ntop.org ntop]''' (Host Collector) user interface and to one or more NST WUI ''''IP Tools'''' pages for additional network processing using the host's IP Address.

{|cellpadding="5"
|[[Image:Ntop-ge.png|thumb|300px|center|ntop Hosts KML Document]]
|[[Image:Ntop-ge-description.png|thumb|300px|center|ntop Hosts KML Document - with Document Description]]
|[[Image:Ntop-ge-host.png|thumb|290px|center|ntop Hosts KML Document - with Host Description]]
|}

When using '''[http://earth.google.com Google Earth]''', one can also view the ''''Document Description'''' balloon by clicking on the generated '''[http://en.wikipedia.org/wiki/Keyhole_Markup_Language KML]''' '''ntop Hosts''' place found under '''Temporary Places''' within the sidebar on the left-hand side. You can also expand the '''ntop Hosts''' place to ''explore'' all geolocated hosts and associated network statistics.

=== '''IP Geolocation Adjustments''' ===
Use the ''''[[HowTo_Setup_The_NST_System_To_Geolocate_Data | IP Geolocate Configure]]'''' button to manage the global geolocation policy for this NST system. This allows one to make latitude and longitude coordinate adjustments, configure private IPv4 Address/Network coordinate locations and select a Geolocation database source. In addition, one can also download and manage the '''[http://www.maxmind.com/ MaxMind]''' "'''GeoIP Country Edition'''", the enhanced "'''GeoIP Lite City Edition'''" and the "'''GeoIP AS Number Edition'''" data sets.

=== '''Auto-Generated ntop Hosts Geolocations''' ===
The NST WUI provides a page to manage a variety of geolocation types on an NST system. One can setup and auto-generate '''ntop Hosts Geolocations''' on this page. See section '''[[HowTo_Manage_NST_Geolocation_Results | HowTo Manage NST Geolocation Results]]''' for details.

== '''Setting The ntop Host Collection Window Size''' ==
There are preference settings in '''[http://www.ntop.org ntop]''' which allows for how long '''[http://www.ntop.org ntop]''' maintains information about a particular host prior to purging it from its internal buffers. These settings control how many hosts will be collected when the '''[http://nst.sourceforge.net/nst/docs/scripts/nstgeolocate.html nstgeolocate]''' script interrogates the '''[http://www.ntop.org ntop]''' session. This in turn will determine how many hosts get geolocated and rendered on a map. Generating a '''[[HowTo_Geolocate_ntop_Data#KML_Document | KML Document]]''' takes considerable amount of more time to produce than generating a '''[[HowTo_Geolocate_ntop_Data#Mercator_World_Map | Mercator World Map]]'''. One needs to tweak these parameters and tailor them to the type of network traffic being monitored so that excessive processing  does not occur when geolocating to many hosts at one time. As a rule of thumb, for low volume traffic, open up the collection window size and for high volume traffic, be more conservative on the time interval for the collection window. When auto-generating a '''[[HowTo_Geolocate_ntop_Data#Mercator_World_Map | Mercator World Map]]''', there are other controls that allow for host geolocation rendering to occur for a long time period (i.e., 1 day, 1 week, 1 month or a year). This allows one to maintain a short '''[http://www.ntop.org ntop]''' collection time window size and still generate map using a long time period for host geolocations. See section '''[[HowTo_Geolocate_ntop_Data#Auto-Generated_ntop_Hosts_Geolocations | Auto-Generated ntop Hosts Geolocations]]''' for details. 

Editing '''[http://www.ntop.org ntop]''' preference settings can be found using the menu shown below. You will need to log in as user: ''''admin'''' and then use the password for the NST ''''root'''' user. The '''[http://www.ntop.org ntop]''' ''''admin'''' password is set when the NST script: '''[http://nst.sourceforge.net/nst/docs/scripts/nstpasswd.html nstpasswd]''' is run.

[[Image:Ntop-user-menu.png|center|frame|Access ntop Edit Preferences from Menu]]

There are 2 '''[http://www.ntop.org ntop]''' preference settings that control host collection window size. One is: ''''purge_host.seconds_idle_with_sessions'''' and the other: ''''purge_host.seconds_idle_with_no_sessions''''. These settings take a value in ''''seconds''''. Typically, one sets these values identical. The image below shows a collection time window size configured for '''1 day''' (60 x 60 x 24 = 86400 - Number of seconds in 1 day). 

[[Image:Ntop-preferences.png|center|frame|ntop Edit Preferences - Collection Window Size]]