HowTo Geolocate NST API Reference

From NST Wiki
Revision as of 07:29, 4 May 2011 by Paul Blankenbaker (talk | contribs) (Added ASCII API section using nstipgeolocate command)
Jump to navigationJump to search

NST Geolocation API Overview

While the NST system provides many different and specialized geolocation renderings, it also provides a generic set of APIs to allow you to plot IPv4 addresses on a map. In general, this is accomplished as follows:

  • You set up your NST system such that is enabled to geolocate IPv4 addresses.
  • You produce a file (or stream) containing a list of IPv4 addresses.
  • You feed your list of IPv4 addresses into one of the NST rendering engines.
  • You get a rendering of your geolocated IPv4 addresses.

NST Geolocation APIs

ASCII Output API (nstipgeolocate)

The nstipgeolocate command can take a list of IPv4 addresses as input and produce a ASCII output file containing geolocated results in various formats.

The following shows the help output when nstipgeolocate is invoked with the --help option:

[root@dhcp143 ~]# nstipgeolocate --help
Usage: nstipgeolocate [options]

This program reads a list of IPv4 addresses from stdin or the command line and
attempts to look up the associated latitude and longitude coordinate values to
geolocate the IPv4 addresses passed.  The results are then echoed to standard
out in the format specified on the command line indicating the locations (if
possible) for each IPv4 address processed.  An associated resource file
(Custom: "$HOME/.etc/nst/" or Global:
"/etc/nst/") can be configured to provide latitude and
longitude coordinate adjustments, private IPv4 Address/Network coordinate
locations and the selection of the Geolocation database source. Alternatively,
on can set the python environment variable: 'PYTHONPATH' to a directory
containing a "" resource file.  This program exits with the
following return codes (useful if calling from a script): 0 - No issues
encountered. 1 - A unknown issue, typically a Python error in the
configuration file. 2 - The GeoIP implementation method was configured, but
the GeoIP database did not exist. To test, check the exit status of:
(nstipgeolocate </dev/null &>/dev/null).

  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -i 'IP0[ IP1[...]]', --ip-list='IP0[ IP1[...]]'
                        Specify a list of IPv4 space separated addresses on
                        command line instead of reading stdin
  -f plain|csv|xml|xmlext|wui|json, --format=plain|csv|xml|xmlext|wui|json
                        The output format to generate: plain, csv (Comma
                        Separated Values), xml (short XML format), xmlext
                        (extended XML format), json (JavaScript Object
                        Notation), or wui (for NST WUI page) [default: plain]
  -n, --disallow-host-names
                        Only IPv4 addresses are processed, ignore host names
                        or Fully Qualified Domain Names (FQDN)
  -r, --resolve-host-names
                        Try to resolve Fully Qualified Domain Names (FQDN) for
                        each IPv4 address
[root@dhcp143 ~]#  

While there are more than three different output formats available, we will demonstrate the csv, xmlext and json outputs as they are well suited for further processing by your own custom code.

Example IP File

The following IPv4 address will be saved to the file /tmp/ip-list.txt and used as the input to the example invocations of the nstipgeolocate command.

Comma Separated Values (CSV) Output

The following demonstrates how to produce a ASCII table of geolocated information for the IPv4 addresses in our file. The ASCII output will be in a comma separated value (CSV) format suitable for importing into a spreadsheet or processing with your own custom code:

[pkb@rice ~]$ nstipgeolocate -r -f csv < /tmp/ip-list.txt 
HostName,IpAddress,Latitude,Longitude,Alias,CountryName,CountryCode,CountryCode3,RegionName,Region,City,PostalCode,AreaCode,DmaCode,TimeZone,,33.9393997192,-84.2077026367,Norcross,United States,US,USA,Georgia,GA,Norcross,,404,524,America/New_York,,-36.8666992188,174.766693115,Auckland,New Zealand,NZ,NZL,Auckland,E7,Auckland,,0,0,Pacific/Auckland,,61.1692008972,-149.844299316,Anchorage,United States,US,USA,Alaska,AK,Anchorage,,907,743,America/Anchorage
[pkb@rice ~]$ 

XML Output

The following demonstrates how to produce a ASCII XML file of geolocated information for a set of IPv4 addresses. This XML file is suitable for processing in a variety of methods and is what we use when building KML and KMZ output for many of our custom Google Earth renderings.

[pkb@rice ~]$ nstipgeolocate -r -f xmlext < /tmp/ip-list.txt 
<?xml version="1.0" encoding="UTF-8"?>
<!-- NST Tools Generated: Wed May 04 07:14:30 EDT 2011
'nstipgeolocate' Host Geolocate Document -->
    <comment>nstipgeolocate Host Geolocate Document</comment>
    <entry key="document_source">NST tools generated: nstipgeolocate</entry>
    <entry key="document_version">2130.000</entry>
    <entry key="create_time_formatted">2011-05-04 07:14:30.447865000-0400</entry>
    <entry key="create_time_secs">1304507670.447865</entry>
      <entry key="city">Norcross</entry>
      <entry key="region_name">Georgia</entry>
      <entry key="alias">Norcross</entry>
      <entry key="region">GA</entry>
      <entry key="area_code">404</entry>
      <entry key="time_zone">America/New_York</entry>
      <entry key="metro_code">524</entry>
      <entry key="country_code3">USA</entry>
      <entry key="dma_code">524</entry>
      <entry key="country_code">US</entry>
      <entry key="country_name">United States</entry>
      <entry key="quality">6</entry>
      <entry key="city">Auckland</entry>
      <entry key="region_name">Auckland</entry>
      <entry key="alias">Auckland</entry>
      <entry key="region">E7</entry>
      <entry key="time_zone">Pacific/Auckland</entry>
      <entry key="country_code3">NZL</entry>
      <entry key="country_code">NZ</entry>
      <entry key="country_name">New Zealand</entry>
      <entry key="quality">6</entry>
      <entry key="city">Anchorage</entry>
      <entry key="region_name">Alaska</entry>
      <entry key="alias">Anchorage</entry>
      <entry key="region">AK</entry>
      <entry key="area_code">907</entry>
      <entry key="time_zone">America/Anchorage</entry>
      <entry key="metro_code">743</entry>
      <entry key="country_code3">USA</entry>
      <entry key="dma_code">743</entry>
      <entry key="country_code">US</entry>
      <entry key="country_name">United States</entry>
      <entry key="quality">6</entry>
[pkb@rice ~]$ 

JavaScript Object Notation Output

The following demonstrates how to produce a ASCII file in JavaScript Object Notation (JSON) format for a set of IPv4 addresses. This JSON file is suitable for processing in a variety of scripting languages or can be used directly by JavaScript code running within a web browser.

[pkb@rice ~]$ nstipgeolocate -r -f json < /tmp/ip-list.txt 
[["", "", [33.939399719238281, -84.20770263671875, {"city": "Norcross", "region_name": "Georgia", "alias": "Norcross", "region": "GA", "area_code": 404, "time_zone": "America/New_York", "metro_code": 524, "country_code3": "USA", "postal_code": null, "dma_code": 524, "country_code": "US", "country_name": "United States", "quality": "6"}]], ["", "", [-36.86669921875, 174.76669311523438, {"city": "Auckland", "region_name": "Auckland", "alias": "Auckland", "region": "E7", "area_code": 0, "time_zone": "Pacific/Auckland", "metro_code": 0, "country_code3": "NZL", "postal_code": null, "dma_code": 0, "country_code": "NZ", "country_name": "New Zealand", "quality": "6"}]], ["", "", [61.169200897216797, -149.84429931640625, {"city": "Anchorage", "region_name": "Alaska", "alias": "Anchorage", "region": "AK", "area_code": 907, "time_zone": "America/Anchorage", "metro_code": 743, "country_code3": "USA", "postal_code": null, "dma_code": 743, "country_code": "US", "country_name": "United States", "quality": "6"}]]]
[pkb@rice ~]$ 

Unfortunately, this format is not human readable. However, if you are familiar with JSON files, you will find it quite useful. We use this output format frequently via AJAX calls from our NST WUI when presenting information on a Google Map rendering within a web browser.

Name Resolution Warning

In the above examples, we included the -r option which instructs the nstipgeolocate to try and resolve host names. Since there were only 3 IPv4 addresses to process this was not a big deal. However, if you have a lot of IPv4 addresses to process you may want to omit this option as it adds a significant amount of time to perform the lookups.

The following shows that the time to process 564 IPv4 processes jumped from 1.073 seconds to 92.536 seconds once we enabled the resolve host names option:

[pkb@rice ~]$ wc /tmp/ip-list-full.txt 
 564  564 7923 /tmp/ip-list-full.txt
[pkb@rice ~]$ time nstipgeolocate -f csv < /tmp/ip-list-full.txt >/dev/null

real	0m1.073s
user	0m0.230s
sys	0m0.086s
[pkb@rice ~]$ time nstipgeolocate -r -f csv < /tmp/ip-list-full.txt >/dev/null

real	1m32.536s
user	0m0.456s
sys	0m0.153s
[pkb@rice ~]$ 

URL (Google Maps)


  • Extremely simple to implement.
  • Highly interactive.
  • Requires web browser and access to a NST system configured for geolocation.
  • Does not scale well. Both the length of a URL string and the number of locations you can plot on Google Maps can be a limiting factor (not a good solution if you have thousands of IPv4 addresses to plot).
  • Optional host name resolution is fairly quick.

URL API Reference

This is a extremely simple API, you start with a base URL of (change the to the IP address of your NST system):

The above URL will simply pull up a empty Google Map. However, the web page which renders the map also accepts the following parameters (all are optional):

This parameter allows you to tell the page what IPv4 address you would like to have geolocated. Each IPv4 address in the list is separated by a + symbol (which corresponds to the escaped space character).
This parameter can be used to turn on (or off) name resolution. If name resolution is enabled, the names will be shown when you click on the markers on the map and pull up the information balloon.
If this parameter is set to true, then the map will be opened in Full Screen mode (you will only see the map in the browser window - there will be no scroll bar).
If this parameter set to true, then the map will be opened in Fit Screen mode (you will the map and the control area in the browser window - there will be a scroll bar).
This parameter controls which marker symbol is used for the IPv4 addresses that are plotted on the map. It is essentially a index (starting from 0) corresponding to the available markers shown under the Options panel.
This parameter controls which marker symbol is used for the IPv4 addresses that have been moved (dragged by the user) the map. It is essentially a index (starting from 0) corresponding to the available markers shown under the Options panel.
This controls whether or not marker shadows are shown.
This parameter controls whether or not the map should try to center itself on one of the markers. This is typically only useful when you know you are plotting exactly one marker.
This parameter can be used to force the initial type of map shown to the end user.
This parameter can be used to force the initial zoom level in the range of 0 to 15 where 0 is zoomed all the way out and 15 is zoomed all the way in.
This parameter can be used to force the initial latitude (in the range of -90.0 to 90.0) for the location that the map will center on.
This parameter can be used to force the initial longitude (in the range of -180.0 to 180.0) for the location that the map will center on.

NOTE: You will typically omit all but the NstMapIpList parameter and let everything else default to the user's preferences.

Example URL

The following builds a URL string which uses the NST system to render three IPv4 addresses (,, and on a web page (we also included the NstMapResolve parameter to force name resolution lookup for the IPv4 addresses passed):

If your NST system is set up for geolocation, you should be able to:

  • Copy/paste the above URL into your browser.
  • Change the to the IP address of your NST system.
  • Press the enter key.
  • See the three IPv4 addresses plotted on a Google Map.

You should be able change the three IPv4 addresses assigned to the NstMapIpList parameter in the URL shown above to your own custom set of IPv4 addresses.

Example Commands

Typing in IPv4 addresses by hand gets old quick. So let's take a look at a real world example where we:

  • Extract a list of IPv4 addresses from a log file.
  • Use a few commands to build our URL.
  • Open the URL in the Google Chrome browser.

For this example, we will extract some IPv4 addresses from a Apache access file. Here is a snippet of the log file we will be extracting the IPv4 addresses from: - - [03/May/2011:09:16:53 -0600] "GET /nst/images/nstwiki_110x32.gif HTTP/1.1" 200 2443 "" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24" - - [03/May/2011:09:16:53 -0600] "GET /nst/images/nstpro_110x32.gif HTTP/1.1" 200 2375 "" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24" - - [03/May/2011:09:16:53 -0600] "GET /nstpro/images/npc_conv_ge_thumb.png HTTP/1.1" 200 58331 "" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24"

This is a easy log file to extract the IPv4 addresses from. We'll simply combine the use of the awk command (to print the first word of each line) with the use of the sort command to filter out duplicates:

[pkb@rice ~]$ gzip -dc < /tmp/access_log.gz | awk -- '{ print $1; }' | sort -u >| /tmp/ip-list.txt
[pkb@rice ~]$ wc /tmp/ip-list.txt
 564  564 7923 /tmp/ip-list.txt
[pkb@rice ~]$ 

The following takes a list of 564 IPv4 addresses found above and builds a very long URL. It then opens the URL using Google Chrome:

[pkb@rice ~]$ MAP_URL="";
[pkb@rice ~]$ SEP="${MAP_URL}"; URL="";
[pkb@rice ~]$ for ip in $(cat /tmp/ip-list.txt); do URL="${URL}${SEP}${ip}"; SEP="+"; done
[pkb@rice ~]$ google-chrome "${URL}"
Created new window in existing browser session.
[pkb@rice ~]$ 

Here is what shows up in the browser from running the commands shown above:

Geolocating IPv4 Using A URL

KML Output (Google Earth)

***Note: Section Under Construction***

Mercator (Bitmap Image)

***Note: Section Under Construction***