View source for HowTo Build NST 24

= Building NST 24 =

The following page provides details on building the NST 24 dnf repository and ISO images directly from source code. The procedure boils down to the following steps:

* Setting up a development machine.
* Checking out copy of the source code.
* Installing the necessary packages to build.
* Building the dnf repository and ISO image.

<div class="noteMessage">[[File:Warning.png|left]]  There appears to be issue with building the NST ISO image if you are using a GNOME 3 desktop and logged in as the ''root'' user. You are strongly discouraged from attempting to build the NST ISO image when logged into a GNOME 3 desktop.</div>

<div class="noteMessage">[[File:Warning.png|left]]  Do not use '''sudo''' when invoking commands to build the NST. If you are logged in as a non-root user, the build process will automatically insert the necessary '''sudo''' invocations as needed.</div>

= Setting Up A Development Machine =

In order to set up a development machine, you will need to download 64 bit ''Live Installable'' version of Fedora 24 (http://fedoraproject.org/get-fedora). It is also recommended to choose the MATE spin of Fedora (https://spins.fedoraproject.org/mate-compiz/) to avoid GNOME 3 issues.

You can use a virtual environment as your development machines. If you do this, I'd recommend allocated at least 2GB of RAM and 50 GB of disk space and two CPUs. Even though it takes longer to build within a virtual environment, I find that the convenience is typically worth it. I typically use VirtualBox (https://www.virtualbox.org/) for my personal builds.

Be aware that there are parts of the build process which require ''root'' access. Because of this, it is recommended that you set up a dedicated machine for NST development.

Regardless of whether you are using real hardware or virtual hardware, after you install Fedora 24, make sure you do the following:

* Update the ''/etc/sudo'' file such that your account (or ''wheel'' group) can run the '''sudo''' command without being prompted for a password (NOTE: you can skip this step if you will be using the ''root'' account).
 sudo visudo
* Update all of the packages on the system:
 sudo dnf upgrade
* Reboot the system to load the newest kernel:
 sudo reboot
* Install/update any device drivers for you hardware or virtual environment and then reboot again (you can skip this step if you don't need to install any special drivers).
 sudo reboot
* Make sure that the ''subversion'' package is installed.
 sudo dnf install subversion 

Check your ''PATH'' setting and make certain that ''/usr/bin'' appears before the ''/bin'' directory (or omit the ''/bin'' directory entirely). Here is the ''PATH'' setting we use on our NST nightly build machines:

 [root@dev24-64 ~]# echo $PATH
 /usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
 [root@dev24-64 ~]# 

<div class="noteMessage">[[File:Warning.png|left]]  We are uncertain as to why this is required, but do not skip this step as we have systems where the RPM builds had dependencies on files like ''/bin/perl'' and ''/bin/python''. The RPMs would build cleanly with these dependencies, but the ''livecd-creator'' tool would then fail to build the ISO image when it tried to resolve the ''/bin'' dependencies.</div>

== Release Building Notes ==

If your build machine will be used to produce a signed release of the NST ISO image, it is recommended to do the following:

* Review the ''make -C yum help'' output and make sure your signing keys are configured and installed.
* Add the canonical host name you want to appear as the ''Build Host'' in the RPM package info as the first entry in your ''/etc/hosts'' file for your machine's IP address.

Example ''/etc/hosts'' file for setting ''Build Host'' to ''dev24-64.networksecuritytoolkit.org'':

 127.0.0.1	localhost.localdomain localhost
 ::1		localhost6.localdomain6 localhost6
 10.8.68.11	dev24-64.networksecuritytoolkit.org dev24-64

Once signed, you can use the ''rpm'' command to verify that the ''Build Host'' comes through:

 [root@dev24-64 yum]# rpm -qip repo/noarch/nstwui-24-518.nst24.noarch.rpm 
 Name        : nstwui
 Version     : 24
 Release     : 518.nst24
 Architecture: noarch
 Install Date: (not installed)
 Group       : Applications/Internet
 Size        : 31076270
 License     : GPLv2
 Signature   : RSA/SHA256, Thu 01 Oct 2015 10:27:38 PM EDT, Key ID acfce86d4eebe9e6
 Source RPM  : nstwui-24-518.nst24.src.rpm
 Build Date  : Thu 01 Oct 2015 09:40:59 PM EDT
 Build Host  : dev24-64.networksecuritytoolkit.org
 Relocations : (not relocatable)
 Packager    : http://www.networksecuritytoolkit.org/
 Vendor      : NST Project
 URL         : http://www.networksecuritytoolkit.org/
 Summary     : The Network Security Toolkit (NST) Web User Interface (WUI).
 Description :
 This package installs the Network Security Toolkit (NST) Web User
 Interface (WUI) components.
 [root@dev24-64 yum]#

= Check Out The Source Code =

Once you have your development machine set up, you can check out a copy of the NST source code. It is recommended that you check out the ''repo'' area from subversion as this is the active area that is kept up to date.

The following commands will check out a copy of the NST 24 ''repo'' area to the ''$HOME/nst'' directory on your system:

 cd $HOME
 svn checkout http://svn.code.sf.net/p/nst/code/repo/24 nst

The ''repo'' area is the more stable area of the NST source code and the development area we use when we build ISO releases or RPM packages that are suitable for pushing out to other machines. If you prefer to live a little closer on the edge, you can use the ''dev'' area which heavy (think daily) changes are occurring. To check out the ''dev'' area instead of the ''repo'' area, use the following command:

 cd $HOME
 svn checkout http://svn.code.sf.net/p/nst/code/dev/24 nst

It may take a bit, but the above should get a current copy of what we are currently using to build the current version of the NST distribution.

= Initial Configuration =

When you run the top level ''./configure'' script, the configuration script may automatically be able to determine what packages are missing and add them to your development system. In order for the script to do this, you must either be logged in as the ''root'' user or using a account which has ''sudo'' privileges. To check whether or not your account has ''sudo'' privileges, try running the following:

 sudo tail /var/log/messages

If the above command fails, it means you skipped a step during the development machine setup section. You will need to fix this before continuing.

If the above command passes (or you are logged in as ''root''), you should be able to run the following commands:

 cd $HOME/nst
 ./configure

The first invocation will take a long time as many packages will need to be installed onto your development machine in order to make it capable of building the NST.

If the configuration script does not complete successfully, it indicates one of the following issues:

* A core package needs to be installed by hand. In this case, a error message should be displayed showing you what ''dnf'' command to run in order to install the core package.
* There was a networking issue in downloading and installing one of the required packages.
* One of the packages that needed to be built and installed locally failed. This typically indicates that the remote server where the source is downloaded from was temporarily down (or that we need to update the source URL for the package in the NST source code).

= Initial Build =

Once you have successfully completed the configuration process, you should be able to build the NST dnf repository and NST ISO image using the following commands:

 cd $HOME/nst
 make

If things work correctly, this should take a long time (many hours) and produce:

;livecd/nst-24.x86_64.iso
: The 64 bit ISO image which can be burned to DVD or installed on a USB memory stick (using the '''live-iso-to-disk''' command).
;yum/repo
: The directory containing the NST repository. You can push this out to a web server if you want to maintain your own repository for your deployed NST systems to get updates from.

= Subsequent Builds =

Subsequent builds will run much faster (less than two hours) as less things should need to be downloaded, installed and built. Use the following commands to update your subversion area to the latest code and rebuild the NST dnf repository and ISO image:

 cd $HOME/nst
 svn update && ./configure && make

= Troubleshooting Notes =

A build may fail for one of the following reasons:

* One of the web servers (there are many) that the build downloads source code is unreachable at the time of your build. For example, if the http://www.thc.org/ web site is down, you won't be able to download the source code for the ''hydra'' package. In this case the build will fail if you don't already have a copy of the ''hydra'' source code downloaded to your ''yum/cache/downloads'' directory.
* The download URL for a source package is no longer valid. For example if there is a new release of the ''wireshark'' source code and current release the NST is trying to download a older release which is no longer available from the ''wireshark'' web site. In this case the build will fail if you don't already have a copy of the ''wireshark'' source code downloaded to your ''yum/cache/download area we actively maintain).
* Your source code is not up to date (''svn update && ./configure'').
* We made a mistake when merging updates into the ''repo'' area. If you could notify us via the NST forum (http://sourceforge.net/p/nst/discussion/361524/) we would appreciate it.

If the build fails because it was unable to download the source code for a particular package it will probably tell you the name of the file it was trying to download. If you can find this file elsewhere on the Internet (or in the ''yum/cache/downloads'' directory of another of your NST build machines), you can copy it into the ''yum/cache/downloads'' directory on your build machine to work through the process.

Unfortunately, the easiest method to get a successful build often requires some patience. If a build fails one day, it may just be easier to wait another day, update, configure and try again. Once you get your first successful build  things typically go much more smoothly (and quickly) as files are cached.

= Customizing Your Build =

There are many command line options you can specify to the top level ''./configure'' script. To see the various options that can be specified, run:

 ./configure --help | less

Instead of specifying a long set of parameters each time when invoking the ''./configure'' command, it is recommended that you use standard configuration files to configure your build. The remaining sections will describe the files which can be used to customize your build.

== Create the Configuration Directory ($HOME/.nst) ==

By default, when your run the ''./configure'' script, it will assume the configuration directory is: ''$HOME/.nst''. You can create this directory with the following command:

 mkdir $HOME/.nst

== $HOME/.nst/configure.sh ==

This file will be sourced by the ''./configure'' script and allow you to override variables which change the build. The following shows a sample template of the most common variables you might want to adjust:

 # Set the default password
 NSTPASSWD="myNewPasswd";
 
 #
 # These are only used if your run "make test" or "make probe-check"
 # from the top level directory
 #
 
 # Set the TEST_EXTENDED variable to "true" to enable test that take a long time to complete
 TEST_EXTENDED="false";
 
 # Following are related to tests requiring SMTP server access (tests that generate email)
 TEST_SMTP_SERVER="smtp.linux.bogus";
 TEST_SMTP_DOMAIN="linux.bogus";
 TEST_SENDMAIL_FROM="first.last@linux.bogus";
 TEST_SENDMAIL_TO="first.last@destination.bogus";

There used to be a lot more variables that could be "tweaked" in this file, but most of them have become deprecated since version 2.11.0 of the NST. You can review the file ''include/data/configure.in'' for a list of some of the other variables that can be overridden.

== $HOME/.nst/custom.ks ==

This file (if present) is "tacked" onto the kickstart file used to create the NST live ISO image. If you are familiar with kickstart files, you can use this to add other packages to your NST distribution.

For example, lets assume that you wanted the following things to be done in your custom NST build:

* Add a package repository for the Google Chrome browser and add Google Chrome to the ISO.
* Include both the ''gcc'' and ''kernel-devel'' packages (as well as all the packages they depend on).
* Transfer the contents of your ''$HOME/.ssh'' configuration from the development system to the ISO image being built.
* Set the default proxy configuration for your local area network.

To accomplish this, you would create the file: ''$HOME/.nst/custom.ks'' with the following contents:

# Make google repo available for ISO build
 repo --name=google-chrome --baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64
 
 # This %packages block is used to indicate what extra packages we
 # would like to include in the Live ISO image. The packages listed
 # here along with any required dependencies will be added to the ISO.
 
 %packages
 # Include google chrome in custom build
 google-chrome-beta
 
 # Pre-install gcc and kernel-devel to simplify installing VirtualBox
 # guest additions
 gcc
 kernel-devel
 
 # You can add additional packages here as long as they are part of the
 # Fedora or NST repositories, if they are not there, you will
 # need to add "repo" line similar to what is shown above for Google Chrome
 %end
 
 
 # This code runs non-chrooted. This allows us to transfer files from
 # the host development system into the build image under $INSTALL_ROOT
 #
 # Useful variables:
 #
 # LIVE_ROOT - Top level directory of all the file that will appear in the
 # ISO image.
 #
 # INSTALL_ROOT - The directory that is used to make the compressed file system
 # image (/) that you see after booting (you'll find: /etc, /var, /usr, ... under here).
 
 %post --nochroot
 
 # Install our ssh keys
 if [ -d /root/.ssh ]; then
   [ -d ${INSTALL_ROOT}/root/.ssh ] && /bin/rm -fr ${INSTALL_ROOT}/root/.ssh;
   cp -rp /root/.ssh ${INSTALL_ROOT}/root/.ssh;
 fi
 
 %end
 
 
 # This %post block does not include the --nochroot option. It
 # runs in a chroot environment under file system being created
 # for the live boot (its as if you are running inside the live boot
 # and unable to see the full development filesystem).
 
 %post
 
 # Set default proxy configuration
 nstsetproxy --http-host 192.168.1.2 --http-port 3128
 
 %end

This simple ''$HOME/.nst/custom.ks'' file provides the necessary hooks to allow you to extend, customize and tweak your NST ISO build as much as you want.

For more information about kickstart files, see: http://fedoraproject.org/wiki/Anaconda/Kickstart.

= Customizing The NST WUI CSS =

If you would like to customize the look of the HTML produced by the ''nstweb'' and ''nstwui'' packages, you can do so by creating the file ''$HOME/.nst/site.css''. The CSS commands which you specify within this file will override the default CSS provided by the packages.

For example, if you prefer to have less color in your tooltips, create the ''$HOME/.nst/site.css'' file and add the following:

 /* Reduce the number of colors used in the tooltips */
 
 div.niceTitle .ttCommand,
 div.niceTitle .ttDocument,
 div.niceTitle .ttSite,
 div.niceTitle .ttNav,
 div.niceTitle .ttValue,
 div.niceTitle .ttAction,
 div.niceTitle .ttAcronym,
 div.niceTitle .ttEmphasis,
 div.niceTitle .ttEmphasisPFont,
 div.niceTitle .ttNote,
 div.niceTitle .ttNotePFont {
   color: white;
 }

When the ''nstweb'' and ''nstwui'' packages are built, the contents of ''$HOME/.nst/site.css'' will be appended to ''/usr/share/nstweb/css/domtt.css'' and ''/usr/share/nstwui/domtt.css'' as these normally the last CSS files are included on most of the HTML pages.