HowTo Setup The NST System To Geolocate Data
Geolocation
We use the term "geolocate", "geolocating" and "geolocation" throughout this page when referring to the process of determining latitude and longitude coordinates associated with an IP address.
Quick Setup
There are many geolocation features available starting with the 2.13.0 release of the NST distribution. Unfortunately, a bit of setup is required in order to make use of these features. The NST WUI provides the "GeoLocation Tools & Management" page in order to simply this setup process and allow one to start geolocating network entities.
Locate: "GeoLocation Tools & Management" Page
Once you NST system is booted and running, use a web browser to pull up the NST WUI using the URL of "https://IP_ADDRESS/" where IP_ADDRESS is the IP address of your NST system. Alternatively, if you are on the NST system itself, you can run Firefox and it should automatically pull up the NST WUI for you. When prompted to login, use root and the master password which you previously set using the nstpasswd command.
From the menu bar on the main NST WUI page, select the "GeoLocation Tools & Management" page as shown below:
Download Geolocation Databases
Once you've reached the "GeoLocation Tools & Management", scroll down to the "GeoIP Management Options" area, select the Lite City Edition, AS Number Edition checkboxes, the Download radio button and press the Process GeoIP Management Option button as highlighted below:
This will take a few moments to complete as the NST will download roughly 40MB of information from the public versions of the databases maintained at http://www.maxmind.com/.
Verify Geolocation is Working
Once the database has been downloaded, you can verify that the NST system is capable of geolocating IP addresses using the same "GeoLocation Tools & Management" page in the NST WUI. To do this:
- Locate the GeoLocation IPv4 Addresses / Hosts section near the top of the web page.
- Enter a well known host name or IP address (for example: www.google.com).
- Press the IPv4 GeoLocate button.
The image below demonstrates provides an example of what should be entered:
The image below shows the results displayed after the IPv4 GeoLocate button is pressed:
If you look closely in the results output above, you notice that the NST system was able to determine the latitude and longitude coordinates of (37.4192008972, -122.057403564) for www.google.com. This tells us that the NST system has been successfully setup to geolocate public IP addresses.
Advanced Geolocation Setup Notes
After mastering the quick setup and experimenting with some of the geolocation enabled tools on the NST, you may start noticing some of the following issues:
- Private addresses and networks are never geolocated.
- Some public IP addresses are geolocated incorrectly.
- Some public IP addresses are not geolocated very well (always showing up at the center of a country or region).
This section is dedicated to pointing out to some of the issues you may encounter and how you can work around them.
MaxMind Download Restrictions
Unfortunately, we are not permitted to bundle the public database from MaxMind in the NST distribution. Because of this, you will need to download this database yourself from [1] (we did try to make the download process as simple and painless as possible).
MaxMind has a policy that limits the number of times you can download the database over a certain time period. This is not typically an issue if you are working with a hard disk installation of the NST. However, it can become an issue in the following circumstances:
- You are working with a Live NST and find yourself rebooting the Live NST system multiple times in a day. Each time you reboot a Live NST system, you will need to download the database.
- You are working with multiple NST systems behind the same firewall and all of these NST systems present the same public IP address to the [2] web servers.
Private/Missing/Incorrect IP Addresses
By default, private IP addresses (like: 192.168.1.25) can never be geolocated. In order to geolocate private addresses, you must enter the the private addresses you would like to geolocate. This is done by editing the file: /etc/nst/nstipgeolocate.py. Look for the ADDR_MAP definition (it's a Python dictionary), and copy/paste from the examples provided in the configuration file.
As an example, the following sets the location for a fictional Linksys router having a IP address of 192.168.4.1 and a cable modem having a IP address of 192.168.5.1:
# DO NOT MODIFY THE FOLLOWING LINE (start of ADDR_MAP dictionary)
ADDR_MAP = {
"192.168.4.1" : (39.979489, -86.121299, {
"alias" : "Linksys Router",
"city" : "Carmel",
"region" : "IN",
"region_name" : "Indiana",
"country_code" : "US",
"quality": "9",
}),
"192.168.5.1" : (39.979495, -86.121290, {
"alias" : "Cable Modem",
"city" : "Carmel",
"region" : "IN",
"region_name" : "Indiana",
"country_code" : "US",
"quality": "9",
}),
# DO NOT MODIFY THE FOLLOWING LINE (end of ADDR_MAP dictionary)
}
You can use the same technique to provide public IP addresses which are missing or you'd like to correct as well.
Private/Missing/Incorrect Networks
By default, private IP networks (like: 192.168.1.0/24) can never be geolocated. In order to geolocate private networks, you must enter the the private networks you would like to geolocate. This is done by editing the file: /etc/nst/nstipgeolocate.py. Look for the NET_MAP definition (it's a Python array of objects) and copy/paste from the examples provided in the configuration file.
As an example, the following sets the location for several networks
# DO NOT MODIFY THE FOLLOWING LINE (start of NET_MAP list)
NET_MAP = [
# NOTE: IPRange("START_IP", "END_IP") allows you to explicitly set what IPs
# to include, whereas IPNetwork("IP/BITS") includes the end points (like .0)
# Note: This will inherit a default quality of 5
(IPRange("192.168.222.1", "192.168.222.254"), (50, -110, {})),
# Example of a private address assigned to Team 868 (TechHounds)
# during competition, notice how many of the attributes
# can be filled in as key/value pairs.
(IPNetwork("10.8.68.0/24"), (39.979489, -86.121299, {
"alias" : "Team 868",
"city" : "Carmel",
"region" : "IN",
"region_name" : "Indiana",
"country_code" : "US",
"drive" : "Mechanum",
"quality" : "8",
})),
# Default location for all private networks, remove leading "##"
# comment characters if you want to enable and make sure you put
# any custom networks ABOVE these entries.
(IPNetwork("10.0.0.0/8"), (40.0, -73.0, {
"alias" : "10.0.0.0/8",
"quality" : "0",
})),
(IPNetwork("172.16.0.0/12"), (40.0, -72.0, {
"alias" : "172.16.0.0/12",
"quality" : "0",
})),
(IPNetwork("192.168.0.0/16"), (39.0, -73.0, {
"alias" : "192.168.0.0/16",
"quality" : "0",
})),
# DO NOT MODIFY THE FOLLOWING LINE (end of NET_MAP list)
]
You'll notice that in the above example, a IP address of 192.168.222.14 would match both the IPRange("192.168.222.1", "192.168.222.254") and the IPNetwork("192.168.0.0/16"). Since, IPRange("192.168.222.1", "192.168.222.254") appears first in the list, its location would be used.
You can use the same technique to provide entries public IP networks which are missing or you'd like to correct as well.
Configuration File
The configuration file for the nstipgeolocate command is at: /etc/nst/nstipgeolocate.py. There are many things which can be tweaked in this file.
You must be careful when editing this file as it contains Python code. If you introduce any errors when editing this file it will cause the nstipgeolocate command to fail and your system will no longer be able to geolocate IP addresses. However, you shouldn't be afraid to edit this file as it is heavily commented and provides many template configurations which you should be able to copy/paste/adjust to your liking.
Geolocation Methods
The METHODS parameter in the configuration file lists the preferred methods to use and the order to use them in when attempting to geolocate a IP address. The METHODS parameter is specified as a Python string containing one or more of the following available implementations separated by commas:
- ADDR_MAP
- Use the Python hash table defined in the configuration file (look for ADDR_MAP in the configuration file). This is a very quick lookup and allows you to add or correct locations for specific IP addresses.
- NET_MAP
- Use the Python array of networks defined in the configuration file. Each IP address is checked to see if it falls in any of the networks defined in the NET_MAP array (also defined in this configuration file). If found, the location information associated with the network is used.
- GeoIP
- Lookup IP addresses using a local copy of the IP database downloaded from MaxMind. This is a quick operation and is pretty good at locating IP addresses. In addition to the free public database, MaxMind also offers a non-free version of its database. We have not tried this, but we are guessing that if you have access, you should be able to use this database by adjusting the value of the GEOIP_DAT_FILE in the configuration file.
- hostip.info
- Use free web service based geolocation service provided at: http://www.hostip.info/. This is a relatively slow look up done via HTTP requests. It's not a good choice if you need to geolocate thousands of addresses. We typically see a maximum throughput rate around 15 geolocation lookups per second when using this method.
- CATCH_ALL
- Returns a fixed location regardless of IP address passed. If included, it should always be the last entry in the METHODS setting. You can modify this default location by modifying the CATCH_ALL_LOCATION Python tuple that is specified in the configuration file.
For example, the default setting of the METHODS parameter is: "ADDR_MAP,NET_MAP,GeoIP". This means that a NST system will:
- First try to geolocate using the ADDR_MAP implementation.
- If the IP address isn't found, it will next try using the NET_MAP implementation
- If the IP address isn't found, it will next try using the GeoIP implementation.
- If the IP address still isn't found, it will give up (no location found).
