HowTo Geolocate NST API Reference - Revision history

Paul Blankenbaker: /* Getting a list of IPv4 Addresses */

2011-05-05T00:20:03Z

Getting a list of IPv4 Addresses

← Older revision		Revision as of 20:20, 4 May 2011
Line 399:		Line 399:
	<pre><nowiki>		<pre><nowiki>
	[pkb@rice ~]$ gzip -dc /tmp/access_log.gz \| head -1		[pkb@rice ~]$ gzip -dc /tmp/access_log.gz \| head -1
	66.92.170.40 - - [03/May/2011:09:16:53 -0600] "GET /nst/images/sf-icon2.gif HTTP/1.1" 200 2934 "http://www.~~networksecuritytoolkit~~.~~org/nst/" "Mozilla/5~~.~~0 (Windows NT 6~~.~~1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24"~~		66.92.170.40 - - [03/May/2011:09:16:53 -0600] "GET /nst/images/sf-icon2.gif HTTP/1.1" 200 2934 "http://www.networksecuritytool...
	[pkb@rice ~]$ gzip -dc /tmp/access_log.gz \| awk -- '{ print $1; }' \| sort -u >\| /tmp/ip-list.txt		[pkb@rice ~]$ gzip -dc /tmp/access_log.gz \| awk -- '{ print $1; }' \| sort -u >\| /tmp/ip-list.txt
	[pkb@rice ~]$ head -3 /tmp/ip-list.txt		[pkb@rice ~]$ head -3 /tmp/ip-list.txt

Paul Blankenbaker: Trimmed lines in apache log snippet

2011-05-05T00:19:33Z

Trimmed lines in apache log snippet

← Older revision		Revision as of 20:19, 4 May 2011
Line 358:		Line 358:
	For this example, we will extract some IPv4 addresses from a Apache access file. Here is a snippet of the log file we will be extracting the IPv4 addresses from:		For this example, we will extract some IPv4 addresses from a Apache access file. Here is a snippet of the log file we will be extracting the IPv4 addresses from:

	66.92.170.40 - - <nowiki>[03/May/2011:09:16:53 -0600] "GET /nst/images/nstwiki_110x32.gif HTTP/1.1" 200 2443 "http://www.~~networksecuritytoolkit~~.~~org/nst/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0~~.~~696.60 Safari/534~~.~~24"~~</nowiki>		66.92.170.40 - - <nowiki>[03/May/2011:09:16:53 -0600] "GET /nst/images/nstwiki_110x32.gif HTTP/1.1" 200 2443 "http://www.networksecuritytool...</nowiki>
	66.92.170.40 - - <nowiki>[03/May/2011:09:16:53 -0600] "GET /nst/images/nstpro_110x32.gif HTTP/1.1" 200 2375 "http://www.~~networksecuritytoolkit.org/nst/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0~~.~~696~~.~~60 Safari/534~~.~~24"~~</nowiki>		66.92.170.40 - - <nowiki>[03/May/2011:09:16:53 -0600] "GET /nst/images/nstpro_110x32.gif HTTP/1.1" 200 2375 "http://www.networksecuritytool...</nowiki>
	66.92.170.40 - - <nowiki>[03/May/2011:09:16:53 -0600] "GET /nstpro/images/npc_conv_ge_thumb.png HTTP/1.1" 200 58331 "http://www.~~networksecuritytoolkit.org/nst/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0~~.~~696~~.~~60 Safari/534~~.~~24"~~</nowiki>		66.92.170.40 - - <nowiki>[03/May/2011:09:16:53 -0600] "GET /nstpro/images/npc_conv_ge_thumb.png HTTP/1.1" 200 58331 "http://www.networksecuritytool...</nowiki>

	This is a ''easy'' log file to extract the IPv4 addresses from. We'll simply combine the use of the '''awk''' command (to print the first ''word'' of each line) with the use of the '''sort''' command to filter out duplicates:		This is a ''easy'' log file to extract the IPv4 addresses from. We'll simply combine the use of the '''awk''' command (to print the first ''word'' of each line) with the use of the '''sort''' command to filter out duplicates:

Paul Blankenbaker: /* JavaScript Object Notation Output */

2011-05-05T00:18:06Z

JavaScript Object Notation Output

← Older revision		Revision as of 20:18, 4 May 2011
Line 160:		Line 160:
	<pre><nowiki>		<pre><nowiki>
	[pkb@rice ~]$ nstipgeolocate -r -f json < /tmp/ip-list.txt		[pkb@rice ~]$ nstipgeolocate -r -f json < /tmp/ip-list.txt
	[["108.107.15.248", "108-107-15-248.pools.spcsdns.net", [33.939399719238281, -84.20770263671875, {"city": "Norcross", "region_name": "Georgia", "alias": "Norcross", "region": "GA", "area_code": 404, "time_zone": "America/New_York", "metro_code": 524, "country_code3": "USA", "postal_code": null, "dma_code": 524, "country_code": "US", "country_name": "United States", "quality": "6"}]], ["118.90.104.224", "ip-118-90-104-224.xdsl.xnet.co.nz", [-36.86669921875, 174.76669311523438, {"city": "Auckland", "region_name": "Auckland", "alias": "Auckland", "region": "E7", "area_code": 0, "time_zone": "Pacific/Auckland", "metro_code": 0, "country_code3": "NZL", "postal_code": null, "dma_code": 0, "country_code": "NZ", "country_name": "New Zealand", "quality": "6"}]], ["12.12.28.31", "12.12.28.31", [61.169200897216797, -149.84429931640625, {"city": "Anchorage", "region_name": "Alaska", "alias": "Anchorage", "region": "AK", "area_code": 907, "time_zone": "America/Anchorage", "metro_code": 743, "country_code3": "USA", "postal_code": null, "dma_code": 743, "country_code": "US", "country_name": "United States", "quality": "6"}]]]		[
			["108.107.15.248", "108-107-15-248.pools.spcsdns.net", [33.939399719238281, -84.20770263671875, {
			"city": "Norcross", "region_name": "Georgia", "alias": "Norcross", "region": "GA", "area_code": 404,
			"time_zone": "America/New_York", "metro_code": 524, "country_code3": "USA", "postal_code": null,
			"dma_code": 524, "country_code": "US", "country_name": "United States", "quality": "6"
			}]],
			["118.90.104.224", "ip-118-90-104-224.xdsl.xnet.co.nz", [-36.86669921875, 174.76669311523438, {
			"city": "Auckland", "region_name": "Auckland", "alias": "Auckland", "region": "E7", "area_code": 0,
			"time_zone": "Pacific/Auckland", "metro_code": 0, "country_code3": "NZL", "postal_code": null,
			"dma_code": 0, "country_code": "NZ", "country_name": "New Zealand", "quality": "6"
			}]],
			["12.12.28.31", "12.12.28.31", [61.169200897216797, -149.84429931640625, {
			"city": "Anchorage", "region_name": "Alaska", "alias": "Anchorage", "region": "AK", "area_code": 907,
			"time_zone": "America/Anchorage", "metro_code": 743, "country_code3": "USA", "postal_code": null,
			"dma_code": 743, "country_code": "US", "country_name": "United States", "quality": "6"
			}]]
			]
	[pkb@rice ~]$		[pkb@rice ~]$
	</nowiki></pre>		</nowiki></pre>

	Unfortunately, this format is not ''human readable''. However, if you are familiar with JSON files, you will find it quite useful. We use this output format frequently via AJAX calls from our NST WUI when presenting information on a Google Map rendering within a web browser.		NOTE: New lines and white space characters were manually added to the above output to make it slightly easier to read.

			Unfortunately, this format is not ''human readable'' (especially if you are not familiar with JSON). However, if you are familiar with JSON files, you will find it quite useful. We use this output format frequently via AJAX calls from our NST WUI when presenting information on a Google Map rendering within a web browser.

	=== Name Resolution Warning ===		=== Name Resolution Warning ===

Paul Blankenbaker: Cleaned up example output

2011-05-05T00:14:09Z

Cleaned up example output

← Older revision		Revision as of 20:14, 4 May 2011
Line 240:		Line 240:
	[pkb@rice ~]$ SEP="op=nstipgeolocate&format=xmlext&iplist=";		[pkb@rice ~]$ SEP="op=nstipgeolocate&format=xmlext&iplist=";
	[pkb@rice ~]$ for ip in $(cat /tmp/ip-list.txt); do echo -n "${SEP}${ip}" >> /tmp/ip-list.post; SEP="+"; done		[pkb@rice ~]$ for ip in $(cat /tmp/ip-list.txt); do echo -n "${SEP}${ip}" >> /tmp/ip-list.post; SEP="+"; done
	[pkb@rice ~]$ wget --no-check-certificate -O /tmp/ip-list.xml --user root --ask-password --post-file /tmp/ip-list.post --quiet ~~'https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php'~~		[pkb@rice ~]$ URL='https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php';
			[pkb@rice ~]$ wget --no-check-certificate -O /tmp/ip-list.xml --user root --ask-password --post-file /tmp/ip-list.post --quiet "${URL}"
	Password for user “root”:		Password for user “root”:
	[pkb@rice ~]$ wc /tmp/ip-list.xml 10748 17690 377318 /tmp/ip-list.xml		[pkb@rice ~]$ wc /tmp/ip-list.xml
			10748 17690 377318 /tmp/ip-list.xml
	[pkb@rice ~]$ tail -21 /tmp/ip-list.xml		[pkb@rice ~]$ tail -21 /tmp/ip-list.xml
	<location>		<location>

Paul Blankenbaker: Fixed output (had been missing a new line) also used URL parameter to keep commands shorter

2011-05-05T00:12:17Z

Fixed output (had been missing a new line) also used URL parameter to keep commands shorter

← Older revision		Revision as of 20:12, 4 May 2011
Line 215:		Line 215:

	<pre><nowiki>		<pre><nowiki>
	[pkb@rice ~]$ ~~wget --quiet --no-check-certificate -O /tmp/ip-list.csv --user root --ask-password~~ 'https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php?op=nstipgeolocate&resolve=false&format=csv&iplist=108.102.15.248+118.90.104.224'		[pkb@rice ~]$ URL='https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php?op=nstipgeolocate&resolve=false&format=csv&iplist=108.102.15.248+118.90.104.224';
			[pkb@rice ~]$ wget --quiet --no-check-certificate -O /tmp/ip-list.csv --user root --ask-password "${URL}"
	Password for user “root”:		Password for user “root”:
	[pkb@rice ~]$ wc /tmp/ip-list.csv 3 5 399 /tmp/ip-list.csv		[pkb@rice ~]$ wc /tmp/ip-list.csv 3 5 399 /tmp/ip-list.csv
	[pkb@rice ~]$ cat /tmp/ip-list.csv HostName,IpAddress,Latitude,Longitude,Alias,CountryName,CountryCode,CountryCode3,RegionName,Region,City,PostalCode,AreaCode,DmaCode,TimeZone		[pkb@rice ~]$ cat /tmp/ip-list.csv
			HostName,IpAddress,Latitude,Longitude,Alias,CountryName,CountryCode,CountryCode3,RegionName,Region,City,PostalCode,AreaCode,DmaCode,TimeZone
	108.102.15.248,108.102.15.248,32.8097991943,-96.7993011475,Dallas,United States,US,USA,Texas,TX,Dallas,,214,623,America/Chicago		108.102.15.248,108.102.15.248,32.8097991943,-96.7993011475,Dallas,United States,US,USA,Texas,TX,Dallas,,214,623,America/Chicago
	118.90.104.224,118.90.104.224,-36.8666992188,174.766693115,Auckland,New Zealand,NZ,NZL,Auckland,E7,Auckland,,0,0,Pacific/Auckland		118.90.104.224,118.90.104.224,-36.8666992188,174.766693115,Auckland,New Zealand,NZ,NZL,Auckland,E7,Auckland,,0,0,Pacific/Auckland
	[pkb@rice ~]$		[pkb@rice ~]$
	</nowiki></pre>		</nowiki></pre>

	=== A wget Example (using POST) ===		=== A wget Example (using POST) ===

Paul Blankenbaker: Fixed show output of interpage links

2011-05-05T00:06:48Z

Fixed show output of interpage links

← Older revision		Revision as of 20:06, 4 May 2011
Line 365:		Line 365:

	* Get a list of IPv4 addresses you would like to geolocate.		* Get a list of IPv4 addresses you would like to geolocate.
	* Use the '''nstipgeolocate''' command or '''ipgeolocate-ajax.php''' web service to convert to XML (the ''xmlext'' format as detailed in the "[[#ASCII_Output_API_.28nstipgeolocate.29]]" and "[[#ASCII Output As A Service]]" sections above).		* Use the '''nstipgeolocate''' command or '''ipgeolocate-ajax.php''' web service to convert to XML (the ''xmlext'' format as detailed in the "[[#ASCII_Output_API_.28nstipgeolocate.29\|ASCII Output API (nstipgeolocate)]]" and "[[#ASCII Output As A Service\|ASCII Output As A Service]]" sections above).
	* Apply a XSL style sheet to convert the XML file into a KML file.		* Apply a XSL style sheet to convert the XML file into a KML file.
	* Open the KML file in Google Earth.		* Open the KML file in Google Earth.

Paul Blankenbaker: Working on interpage links

2011-05-05T00:04:47Z

Working on interpage links

← Older revision		Revision as of 20:04, 4 May 2011
Line 365:		Line 365:

	* Get a list of IPv4 addresses you would like to geolocate.		* Get a list of IPv4 addresses you would like to geolocate.
	* Use the '''nstipgeolocate''' command or '''ipgeolocate-ajax.php''' web service to convert to XML (the ''xmlext'' format as detailed above).		* Use the '''nstipgeolocate''' command or '''ipgeolocate-ajax.php''' web service to convert to XML (the ''xmlext'' format as detailed in the "[[#ASCII_Output_API_.28nstipgeolocate.29]]" and "[[#ASCII Output As A Service]]" sections above).
	* Apply a XSL style sheet to convert the XML file into a KML file.		* Apply a XSL style sheet to convert the XML file into a KML file.
	* Open the KML file in Google Earth.		* Open the KML file in Google Earth.

Paul Blankenbaker: /* Convert To KML */

2011-05-04T23:50:36Z

Convert To KML

← Older revision		Revision as of 19:50, 4 May 2011
Line 420:		Line 420:

	NOTE: You will likely want to customize the ''/tmp/geoip2kml.xsl'' file. It's just meant to serve as a starting point.		NOTE: You will likely want to customize the ''/tmp/geoip2kml.xsl'' file. It's just meant to serve as a starting point.

			=== Open In Google Earth ===

			You should be able to open the KML file produced using a tool like Google Earth:

			[pkb@rice ~]$ google-earth /tmp/ip-list.kml
			[pkb@rice ~]$

			Here is what the KML rendering resembles:

			[[File:geoip-kml-api.png\|frame\|center\|Geolocating IPv4 Using KML API]]

	== Mercator (Bitmap Image) ==		== Mercator (Bitmap Image) ==

	<center>'''<span style='color: red; font-size: 200%;'>*Note: Section Under Construction*</span>'''</center>		<center>'''<span style='color: red; font-size: 200%;'>*Note: Section Under Construction*</span>'''</center>

Paul Blankenbaker: /* KML Output (Google Earth) */

2011-05-04T21:31:35Z

KML Output (Google Earth)

@@ Line 362: / Line 362: @@
 == KML Output (Google Earth) ==
-<center>'''<span style='color: red; font-size: 200%;'>***Note: Section Under Construction***</span>'''</center>
+Producing KML output involves the following steps:
 == Mercator (Bitmap Image) ==
 <center>'''<span style='color: red; font-size: 200%;'>***Note: Section Under Construction***</span>'''</center>

Paul Blankenbaker: Working on geolocate API as web service section

2011-05-04T14:56:28Z

Working on geolocate API as web service section

← Older revision		Revision as of 10:56, 4 May 2011
Line 186:		Line 186:
	[pkb@rice ~]$		[pkb@rice ~]$


			== ASCII Output As A Service ==

			While the '''nstipgeolocate''' command is quite useful if you are on the NST system itself, it can't be run if you are on a different system. You can however, use the NST WUI web service page to run the equivalent '''nstipgeolocate''' commands remotely. The following outlines how to do this.

			=== The Base URL ===

			The following URL is used to request that a set of IPv4 address be geolocated (change ''192.168.1.143'' to the address of your NST system):

			<nowiki>https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php</nowiki>

			In order to use the above URL, you must provide a set of parameters. These parameters can either be appended directly on the end of the URL (a GET request), or submitted separately via a POST request. Examples of both methods will be described later. The GET approach is simpler, but the POST method is typically better suited when you need to geolocate a lot of IPv4 addresses.

			Here are the parameters which can be included:

			;op=nstipgeolocate
			: This parameter is required and tells the service that you want to geolocate a set of IPv4 addresses.
			;iplist=IP0[+IP1[+IP2...]]
			: This parameter is required and allows you to tell the service what IPv4 address you would like to have geolocated. Each IPv4 address in the list is separated by a ''+'' symbol (which corresponds to the ''escaped'' space character).
			;format=csv\|xmlext\|json\|...
			: This parameter is required and allows you to tell the service what type of output you would like back. If omitted, it defaults to ''xmlext'' (extended XML output). The common output formats are ''csv'', ''xmlext'', and ''json''. For a full list of available output formats, look at the help output of the '''nstipgeolocate''' command.
			;resolve=true\|false
			: This parameter can be used to turn on (or off) name resolution. Name resolution tends to be very slow when processing large sets of data. For example, it may only take a couple of seconds to process a thousand IPv4 addresses without name resolution and several minutes if name resolution is enabled. It is recommended you only set this option to ''true'' when processing a small set of IPv4 addresses.

			=== A wget Example (using GET) ===

			The following demonstrates using the NST web service to geolocate two IPv4 addresses using a simple '''wget''' invocation:

			<pre><nowiki>
			[pkb@rice ~]$ wget --quiet --no-check-certificate -O /tmp/ip-list.csv --user root --ask-password 'https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php?op=nstipgeolocate&resolve=false&format=csv&iplist=108.102.15.248+118.90.104.224'
			Password for user “root”:
			[pkb@rice ~]$ wc /tmp/ip-list.csv 3 5 399 /tmp/ip-list.csv
			[pkb@rice ~]$ cat /tmp/ip-list.csv HostName,IpAddress,Latitude,Longitude,Alias,CountryName,CountryCode,CountryCode3,RegionName,Region,City,PostalCode,AreaCode,DmaCode,TimeZone
			108.102.15.248,108.102.15.248,32.8097991943,-96.7993011475,Dallas,United States,US,USA,Texas,TX,Dallas,,214,623,America/Chicago
			118.90.104.224,118.90.104.224,-36.8666992188,174.766693115,Auckland,New Zealand,NZ,NZL,Auckland,E7,Auckland,,0,0,Pacific/Auckland
			[pkb@rice ~]$
			</nowiki></pre>

			=== A wget Example (using POST) ===

			The following takes a list of ''564'' IPv4 addresses in the file ''/tmp/ip-list.txt'' found above and builds a files containing the appropriate parmaters to use the POST method. The '''wget''' command is then used to POST this information to the web service and store the results as the XML file ''/tmp/ip-list.xml''.

			<pre><nowiki>
			[pkb@rice ~]$ wc /tmp/ip-list.txt
			564 564 7923 /tmp/ip-list.txt
			[pkb@rice ~]$ head -3 /tmp/ip-list.txt
			108.107.15.248
			109.154.109.210
			109.230.220.96
			[pkb@rice ~]$ rm -f /tmp/ip-list.post
			[pkb@rice ~]$ SEP="op=nstipgeolocate&format=xmlext&iplist=";
			[pkb@rice ~]$ for ip in $(cat /tmp/ip-list.txt); do echo -n "${SEP}${ip}" >> /tmp/ip-list.post; SEP="+"; done
			[pkb@rice ~]$ wget --no-check-certificate -O /tmp/ip-list.xml --user root --ask-password --post-file /tmp/ip-list.post --quiet 'https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php'
			Password for user “root”:
			[pkb@rice ~]$ wc /tmp/ip-list.xml 10748 17690 377318 /tmp/ip-list.xml
			[pkb@rice ~]$ tail -21 /tmp/ip-list.xml
			<location>
			<hostname>99.59.87.159</hostname>
			<addr>99.59.87.159</addr>
			<lat>41.867500</lat>
			<lon>-87.674400</lon>
			<properties>
			<entry key="city">Chicago</entry>
			<entry key="region_name">Illinois</entry>
			<entry key="alias">Chicago</entry>
			<entry key="region">IL</entry>
			<entry key="area_code">312</entry>
			<entry key="time_zone">America/Chicago</entry>
			<entry key="metro_code">602</entry>
			<entry key="country_code3">USA</entry>
			<entry key="dma_code">602</entry>
			<entry key="country_code">US</entry>
			<entry key="country_name">United States</entry>
			<entry key="quality">6</entry>
			</properties>
			</location>
			</geoip>
			[pkb@rice ~]$
			</nowiki></pre>

	== URL (Google Maps) ==		== URL (Google Maps) ==