HowTo Automate & Manage NST Geolocation Results

Overview - nstgeolocate Session Manager

The nstgeolocate Session Manager page was designed to help manage and automate the generation of certain geolocation types. The NST script: nstgeolocate is mostly used for results presented on the page. Currently, auto-generated "ntop Host" geolocation sessions can be created, monitored and managed. Also, all previously generated "IPv4 Address Conversation" geolocation sessions can be viewed or managed. Using the NST menu, one can navigate to the nstgeolocate Session Manager page as shown in the screen shot below.

NST WUI nstgeolocate Session Manager Menu Selection

Collapse / Expand Session Sections

The amount of information presented on the nstgeolocate Session Manager page can be large depending on the number of ntop Host sessions configured and/or the number of IPv4 Address Conversations archived. The "Collapse All Sessions" button and the "Expand All Sessions" button can be used in combination with the "Hide / Show" session section folder icons to display just the information relevant to you. This will not only reduce the amount of information you need to visually process, but will also speed up the rendering of the page.

Manage & Monitor ntop Host Sessions

If your NST system if running ntop, it can be set up to periodically produce geographic representations of the information collected by ntop. This is known as setting up a ntop host session.

There are 2 sections that display configured ntop Host sessions. One for auto-generating ntop Host Geolocations on one or more Mercator World Map projections and the other for auto-generating one or more ntop Host Geolocations KML documents that can be rendered on Google Earth. ntop Host sessions are created and configured in the section: Create / Update / Import nstgeolocate Host Session. A running ntop server can be local (i.e., Running on your NST probe) or remote (i.e., Running on a system other than your local NST probe).

ntop Host Session - Table Column Descriptions

Each configured ntop Host session type is presented in a table layout format. The following describes each column header associated with the tables.

World Map nstgeolocate Host Session Directory:

Click on a link in this column to use the NST Directory/File Browser to view supporting files associated with the generation of producing the ntop World Map Host bit image. Depending on how long you have set the "Map Pruning Interval" value for each session, you can also view historical generated maps in the associated directory.

KML Document nstgeolocate Host Session Directory:

Click on a link in this column to use the NST Directory/File Browser to view supporting files associated with the generation of producing the ntop KML Document. Depending on how long you have set the "Map Pruning Interval" value for each session, you can also view historical generated KML Documents in the associated directory.

C (Cron Control Status):

This column indicates if the configured ntop Host session is under cron control. A green circle icon indicates that it is and a red stop sign icon indicates that it is not. Use an associated "D" action button to disable cron control for the session.

Active 'ntop' Session :

This column is used for linkage back to either the configured NST WUI probe ntop management interface or the ntop session user interface.

ntop Mgt (HTTP):

Use this button to enter the NST WUI ntop web-based management interface (HTTP access) for the ntop session.

ntop Mgt (HTTPS):

Use this button to enter the NST WUI ntop web-based management interface (HTTPS access) for the ntop session.

HTTP Access:

Use this link to enter the ntop User web-based management interface (HTTP access) for the ntop session (Default port: 3000).

HTTPS Access:

Use this link to enter the ntop User web-based management interface (HTTPS access) for the ntop session (Default port: 3001).

Description:

The description for each ntop session includes the "Session Annotation", participating "Network Capture Interfaces" and the current "ntop Session Uptime".

Action (Buttons):

U (Update ntop Session Settings):

This action button will pre-fill all values associated with the ntop session into the "Create / Update / Import nstgeolocate Host Session" configuration form. Use this action when making changes to an existing ntop session.

R (Remove ntop Host Session):

This action button will remove an entire ntop session including supporting files and all previously generated maps or KML documents. Make backups accordingly prior to using this action.

Note: Use either the "Remove All nstgeolocate Host World Map Sessions" button or the "Remove All nstgeolocate Host KML Doc Sessions" button to completely remove All host sessions for the associated geolocation rendered type. Remember to make any backups prior to performing the removal of all sessions.

D (Disable Cron Control):

This action button is used to disable cron control for the session. The ntop session will still exist but auto-generation of maps or KML documents will not occur. To re-enable cron control, use a combination of the "U" action button and the "Create / Update nstgeolocate Session" button located in the "Create / Update / Import nstgeolocate Host Session" section.

M (Monitor Auto-Generated Maps or KML Documents):

This action is used to monitor each auto-generated ntop Hosts session. See section: "Monitor ntop Hosts Mercator World Map" or section: "Monitor ntop Hosts KML Document (Google Earth)" for further details associated with each geolocation rendered type.

nstgeolocate Session Manager: Configured ntop Host Sessions

Monitor ntop Mercator World Map Hosts

The "ntop World Map Hosts" page allows one to monitor geolocated hosts from an ntop session in pseudo-realtime within your web browser using AJAX and image caching disabled. By enabling a full screen map view (Kiosk mode) , you could display the "ntop World Map Hosts" presentation in a NOC setting providing continual geolocated host updates.

The photo below demonstrates the Full Screen Map View of an "ntop World Map Host" session displayed on a 27" iMac using the Google Chrome web browser.

ntop World Map Host: Displayed on a 27" iMac using Google Chrome in a Full Screen Map View

Note: An NST menu shortcut can be used to view the default "ntop World Map Hosts" page by using: "Network" => "Monitors" => "ntop World Map Hosts".

Every 10 seconds a request is made from your web browser back to the NST probe to check for any new map updates and status information changes. The "Map Update Interval" for generating a new "ntop World Map Hosts" image can be configured in section: Create / Update / Import nstgeolocate Host Session.

One can hover the mouse pointer over the "Information" icon to reveal a tool tip providing the Map Description, ntop World Map Hosts Geolocation Update Information and Image Control Button Grid Usage. There are two status "Circle" icons depicting the state of the ntop session and the combination of running the NST script: nstgeolocate under cron control. Normally the status "Circle" icons will appear in the color green. A "Warning" or "Stopped" condition may occur and will appear in the color orange or red. Hover the mouse pointer over each status "Circle" icon to show the current state information and the associated color status definition. The ntop "Circle" icon tool tip will also show "ntop System" and "ntop Session information similar to what is displayed in the "ntop Hosts World Map - with ntop Tool Tip" screen shot above.

Monitor ntop Hosts KML Document (Google Earth)

Auto-generated ntop Hosts KML documents can be monitored and analyzed on a KML Earth Browser such as Google Earth (See: HowTo Setup Your Client System To View Geolocation Data). These documents will be updated each "Map Update Interval" which can be configured in section: Create / Update / Import nstgeolocate Host Session.

Note: Generating an ntop Hosts KML document can use significant CPU resources and time depending on the number of hosts to geolocate from an ntop session. Be cautious of this fact especially on a slower CPU configured NST probe. One may need to choose a less frequent "Map Update Interval" or limit the number of hosts to geolocate to mitigate system resources necessary in the production these types of KML documents.

Each host that was geolocated appears as a host marker and contains a 'Host Description' balloon depicting selective ntop network traffic statistics information. Click on a host marker to reveal the 'Host Description' balloon. Hyperlinks are also provided to the ntop (Host Collector) user interface and to the NST WUI 'IP Tools' page for additional network processing using the host's IP Address.

When using Google Earth, one can also view the 'Document Description' balloon by clicking on the generated KML ntop Hosts place found under Temporary Places within the sidebar on the left-hand side. You can also expand the ntop Hosts place to explore all geolocated hosts and associated network statistics.

Manage & View IPv4 Address Conversation Sessions

Each time IPv4 Address Conversations are geolocated using the decode section from either the Single or the Multi-Tap Network Packet Capture page, they will be cataloged here in this section. Existing IPv4 Address Conversations are listed in two different sections based on the rendered output (i.e., Mercator World Map or KML Document). Each section is Web Browser session aware. The sections are grouped by which browser generated the IPv4 Address Conversation geolocations (i.e., Your browser and all other browsers), sorted by date/time in descending order (i.e., More recent geolocations are listed first) and displayed in a table layout format.

The geolocate file naming convention used includes the geolocation render type (i.e., "wm" - Mercator World Map or "kml" - KML Document), network entity to geolocate (i.e, "conv" - IPv4 Address Conversation) and the time the output was generated (i.e., "YYMMDD-hhmmss" - YY - Year, MM - Month, DD - Day, hh - Hour, mm - Minute and ss - Second).

IPv4 Address Conversation - Table Column Descriptions

The archived IPv4 Address Conversation sessions are presented in a table layout format. Each table is grouped by which browser generated the IPv4 Address Conversation geolocations. The "Conversation Directory" for this and/or other browsers can be viewed with the NST Directory/File Browser (i.e., Click on the browser Session ID link that starts with "sid_") . The following describes each column header associated with the tables.

World Map IPv4 Address Conversation:

Each link in this column represents an archived Mercator World Map IPv4 Address Conversation geolocation bit image using the file naming convention described above. Click on an image link to allow your browser to render it.

Conversation XML:

Each link in this column represents a supporting XML file to an associated archived Mercator World Map IPv4 Address Conversation session. Click on the link to allow your browser to present the XML data in its native format.

Capture / Description:

This column includes the "Capture File Name" and the "Session Annotation" text.

Action (Buttons):

Conversation Session Directory Browser Headers

R (Remove Conversation Session Directory):

This action button will remove an entire IPv4 Address Conversation session directory including supporting files and all previously generated maps or KML documents for an associated browser. Make backups accordingly prior to using this action.

Note: Use either the "Remove All Mercator World Map Conversation Sessions" button or the "Remove All KML Document Conversation Sessions" button to completely remove All generated IPv4 Address Conversation sessions for each browser and for the associated geolocation rendered type. Remember to make any backups prior to performing the removal of all sessions.

B (Browse Conversation Session Directory)

This action button uses the NST Directory/File Browser to view an IPv4 Address Conversation session directory for an associated browser.

Individual Conversations

R (Remove Individual Conversation):

This action button will remove an individual IPv4 Address Conversation session including supporting files and the generated map or KML document for an associated browser. Make backups accordingly prior to using this action.

V (View Individual Conversations):

This action is used to view each archived IPv4 Address Conversation session for an associated browser. See section: "HowTo Geolocate Network Packet Capture Data: Mercator Wold Map" or section: "HowTo Geolocate Network Packet Capture Data: KML Document (Google Earth)" for further details associated with each geolocation rendered type.

nstgeolocate Session Manager: Archived IPv4 Address Conversation Geolocations

Create / Update / Import nstgeolocate Host Sessions

Note: An ntop server must already be running in order to create an nstgeolocate auto-generated "Host" session using the form in this section. If this is not the case, then see the NST Wiki page: "HowTo Geolocate ntop Data" for the steps on how to start up an ntop server.

This section explains the procedure for auto-generating selective network entity geolocations using the NST script: nstgeolocate. Currently only the geolocation of "Hosts" from an ntop session is supported. Both Mercator World Map bitmap images and KML documents can be produced. The Linux System cron facility is used for scheduling, updates and pruning expired maps and documents.

Create / Update / Import nstgeolocate Host Sessions Configuration Form

The following sections will describe each related selection and text input field configuration that comprise the "Create / Update / Import nstgeolocate Host Sessions Configuration Form".

nstgeolocate Session Location

Use these selection and text input fields to clearly identify a new or existing nstgeolocate Session configuration. The nstgeolocate script uses a hierarchical directory structure based on these selection and text input fields to determine the directory location for network entity geolocation Mercator World Maps, KML documents and associated supporting files. One can also import an existing nstgeolocate Session configuration by correctly filling in each configuration selection and text input field and then click on the "Import nstgeolocation Session" button. The location of an existing nstgeolocate Session configuration could be on a shared file system or on a removable flash drive.

Geolocation Type:

Generate

Use this selection to generate either a Mercator World Map bitmap image or a KML document.

Map Type (Network Entity):

The "Map Type" is the name of the Network Entity that is to be geolocated. Currently only the geolocation of "Hosts" from an ntop session is supported.

Session Name:

The "Session Name" is a unique directory base name to describe the location for this nstgeolocate session configuration. If the directory base name does not already exist, it will be created. One can be creative with custom names. Lets say your configuration rotates the Mercator World Map every week. One may choose a Session Name like: "wm_host_weekly" for this configuration.

The associated "Select..." button to the right of the text input field can be used to automatically populate default or custom "Session Name" directory base names. The default "Session Name" directory base name for generating a Mercator World Map Host session is: "wm_host". The default "Session Name" directory base name for generating a KML Document Host session is: "kml_host". A newly created session or an imported session configuration will automatically populate the selection list if it is unique. Use the "Edit nstgeolocate Session Name List" button to manually alter the "Session Name" selection list.

Base Session Directory:

This is the base session directory used for storing the generated network entity geolocation Mercator World Maps, KML documents and associated supporting files. Use the "Browse" button to bring up the NST Directory/File Browser for locating and automatically populating the base directory input text field. The default location is: "/var/nst/nstgeolocate". If the base session directory name does not already exist, it will be created.

Computed Session Directory Example:

These configuration settings will result in producing the following computed session directory:

Configuration Settings:

Generate: Mercator, Map Type: Host, Session Name: wm_host and the Base Session Directory: /var/nst/nstgeolocate

Computed Session Directory:

"/var/nst/nstgeolocate/mercator/host/wm_host"

ntop URL Data Source:

Enter a URL to a running ntop server that can be local (i.e., Running on your NST probe) or remote (i.e., Running on a system other than your local NST probe).

Note: The remote ntop server does not necessarily have to be running on an "NST Probe". If this is the case, then some data and status information degradation may result in various related tool tips.

Map Annotation

ntop Map Title

Use this text input field to override the session ntop Annotation. Enter a short descriptive phrase (i.e., 22 Characters or less) to identify the traffic that the ntop server is monitoring (e.g., "Corporate Web Site"). Leave this field blank to use the ntop Annotation entered when the ntop server was setup.

ntop Interface Name(s)

Use this text input field to override the ntop session Interface Name(s). Leave this field blank to use the Interface Name(s) entered when the ntop server was setup.

Mercator Map Attributes (Mercator World Map Only)

Geolocate Marker Option

Mark Type

Select a marker symbol type for each network entity geolocated on the Mercator World Map projection.

Mark Color

Select a marker symbol color for each network entity geolocated on the Mercator World Map projection.

Map Automation

Map Update Interval

Use one of these predefined selection list Time Interval values to specify how often the cron facility should update or create a new geolocation session.

Mercator World Map Update Note: A brand-new Mercator World Map will only be created if the "Map Rotate Interval" period has not been exceeded. Therefore, the "Map Update Interval" allows one to accumulate history or build a collage with your network entity geolocations. If the "Map Rotate Interval" period has not been exceeded, the previously generated network entity geolocation Mercator World Map will be used as the base map thus continuing the accumulated geolocation history. The nstgeolocate script is optimized so that only newly collected network entities each Map Update Interval will be geolocated thus saving system resources. If the "Map Rotate Interval" period has been exceeded, a blank Mercator World Map will be used as the base map thus starting a brand-new map.

KML Document Update Note: A brand-new KML document will always be created with each update interval.

Minute

Select a Map Update Interval value between "1" and "59" minutes.

Hour

Select a Map Update Interval value between "1" and "23" Hours.

Other

Select a predefined Map Update Interval using one of the following values: "hourly", "daily", "midnight", "weekly", "monthly", "yearly", or "annually".

Map Rotate Interval (Mercator World Map Only)

Select a Map Rotate Interval value that will determine how long to geolocate network entities on the same Mercator Map Projection. For each "Map Update Interval", if the "Map Rotate Interval" period has not been exceeded, the previously generated network entity geolocation Mercator World Map will serve as the base map thus continuing the accumulated geolocation history until the complete "Map Rotate Interval" has expired.

Minute

Select a Map Rotate Interval value between "1" and "59" minutes.

Hour

Select a Map Rotate Interval value between "1" and "23" hours.

Day

Select a predefined Map Rotate Interval value between "1" and "365" days.

HowTo Generate A Long Duration ntop Host Mercator World Map

When using ntop to monitor a relatively high network traffic environment, one could set a short "ntop Collection Window Size Interval" (e.g., 300 Seconds - 5 Minutes) and use a frequent "Map Update Interval" (e.g., 1 Minute) but with an extended "Map Rotate Interval" (e.g., 1 Day) to generate a long duration ntop Host geolocation map.

Map Pruning (Delete Older Maps & Documents)

Use this section to purge older Mercator World Maps and KML documents. Essentially use the controls in this section to keep a window back in time of how many auto-generated geolocation maps and documents you would like to maintain.

Map Pruning Check

Select a time interval for checking when to prune expired maps and documents.

Minute

Select a Map Pruning Check interval value between "1" and "59" minutes.

Hour

Select a Map Pruning Check interval value between "1" and "23" hours.

Other

Select a predefined Map Pruning Check interval using one of the following values: "hourly", "daily", "midnight", "weekly", "monthly", "yearly", or "annually".

Prune Map Back

Select a window duration back in time of how many auto-generated geolocation maps and documents you would like to maintain.

Minutes

Select a predefined Prune Map Back window value between "0" and "1440 (i.e., 1 Day)" minutes.

Days

Select a predefined Prune Map Back window value between "0" and "99999 (i.e., Disable Purging)" days.

For example, in the image below the Map Pruning Check field is set to 12 hours and the Prune Map Back field is set to 4 days. This would result in the system checking for old map data every 12 hours. During each check, any map data found to be more than 4 days old would be removed from the system.

nstgeolocate Session Manager: Create, Update and Import ntop Host Sessions

Cron Service Information

Use this section to maintain the crond service and other related services providing local network entity data sources for geolocation.

Current Crontab Configuration

Use this section to view the root user's crontab configuration in particular the entries related to a nstgeolocate session.