Building From Source: Difference between revisions
| Line 265: | Line 265: | ||
* Transfer the '''ISO''' image to a different machine (we typically use '''scp''', if you are a Windows user, you may want to consider [http://winscp.sourceforge.net/ WinSCP]). | * Transfer the '''ISO''' image to a different machine (we typically use '''scp''', if you are a Windows user, you may want to consider [http://winscp.sourceforge.net/ WinSCP]). | ||
* Boot directly from the '''ISO''' image using a '''VMware''' product or '''Virtual PC'''. | * Boot directly from the '''ISO''' image using a '''VMware''' product or '''Virtual PC'''. NOTE: At the time of this writing, there is an issue with '''NST''' and the recommended virtual LSI SCSI controller in '''VMware Server''' (beta testers should opt for the '''BusLogic''' controller when creating new virtual machines for the time being). | ||
= Tweaking The Configure/Build Process = | = Tweaking The Configure/Build Process = | ||
Revision as of 10:52, 22 August 2007
Preparing The Build Machine
You will need to install Fedora 7 (the i386 variant) onto your development system.
If you want to build a 64 bit variant (x86_64) of the NST, you will need to download the x86_64 variant of Fedora 7. While the i386 variant will run on both 32 bit and 64 bit machines, the x86_64 variant will only run on 64 bit machines. Most, if not all, 64 bit AMD processors and the Intel Core 2 series should be compatible with the x86_64 variant.
If you don't want to use physical hardware, you can use a Virtual Machine (even running under Windows) to do your NST development under. We have found that either VMWare Server (our preference) or VMWare Player from VMware can be used (even if you want to build a x86_64 version of the NST). We have also found that Virtual PC can be used.
See the Fedora Installation Notes for additional information on installing the base OS.
Getting the NST Source Code
There are several ways to acquire the source code for NST developement (one can find an entire section related to this in the NST FAQ).
For our purposes we will choose the anonymous CVS approach. This permits one to build a current snapshot of the NST distribution as it is being developed.
The following demonstrates how one can create a: nst sub-directory on a Fedora Core 6 based system, and then populate it with the current NST source code (NOTE: Just press the Enter key when prompted for the password as there is no password required for anonymous access):
Logging in to :pserver:anonymous@nst.cvs.sourceforge.net:2401/cvsroot/nst CVS password:
... You should see file names streaming by in your
console window - it may take several moments to
download all of the source files ...
As time goes on, you may want to update your source files (the NST developers check in new code quite frequently). Use the following commands:
... You should see directory names streaming by in your
console window as CVS searches for updated files ...
As setting the CVSROOT variable can be quite tedious, it is recommended that you include its definition in your: "~/.bashrc" file. Add the following lines to the end of: "~/.bashrc".
CVSROOT=":pserver:anonymous@nst.cvs.sourceforge.net:/cvsroot/nst"; export CVSROOT;
Configuring The System
After downloading the source code, you will need to run the: configure command from the top level directory. Before doing so, here are a couple of things to note:
- Before configuring the system, it is recommend that one run: "yum update" to make sure the latest kernel and package updates are installed.
- Sun's Java Run Time Environment (JRE) is used when building the NST. The configure script will attempt to download and install it automatically (if it isn't found on your development system). If the script is unable to automatically install it, you will need to download and install the JRE from: http://java.com/ (get the appropriate RPM version).
- You will probably need to run the configure multiple times the first time you setup your development system (as you will most likely be missing some packages).
- If configure determines that there are missing packages, it will indicate the yum command which you need to run in order to add the necessary packages to your system (or at least indicate which packages need to be added).
Here's an example of what one might go through when trying to initially configure a development system (NOTE: In this example, yum found that the current packages installed on the system were up to date):
Loading "installonlyn" plugin Setting up Update Process Setting up repositories core 100% |=========================| 1.1 kB 00:00 updates 100% |=========================| 1.2 kB 00:00 extras 100% |=========================| 1.1 kB 00:00 Reading repository metadata in from local files primary.xml.gz 100% |=========================| 1.6 MB 00:11 extras : ################################################## 5097/5097 No Packages marked for Update/Obsoletion
***ERROR*** unable to find executable program 'docbook2html' on system
***ERROR*** unable to find executable program 'docbook2pdf' on system
***ERROR*** unable to find executable program 'make' on system
***ERROR*** unable to find executable program 'mkzftree' on system
***ERROR*** unable to find executable program 'ncftpput' on system
***ERROR*** unable to find executable program 'pear' on system
***ERROR*** unable to find executable program 'rpmbuild' on system
***ERROR*** unable to find executable program 'svn' on system
Try the following to add the missing packages:
yum install docbook-utils docbook-utils-pdf make zisofs-tools ncftp php-pear rpm-build subversion
... Lot's of yum output as it figures out package dependencies,
then you will be prompted. Enter 'y' when you see:
Checking for DocBook XSL [ OK ]
Verifying settings [ OK ]
Creating: "./config/config.sh" [ OK ]
... Omitting Output.
This time configure found the system up to date and completed normally ...
Use "make" to build src/nst-1.7.0.iso.gz, or "make help" to see
additional targets.
NOTE: You may want to run "make package-update" to update MANY of the
packages needed for the NST. You can then run "make package-check" to
see what is left to do by hand (there are many helper scripts under
"src/bin" to aid in the package install/update process).
Installing Packages
At this point you might be tempted to run: "make" and produce a bootable ISO image. However, you would be missing MANY of the extra packages which are typically included in a NST distribution.
To get "most" of the extra packages installed onto your development system, you can use the: "make package-update" command as shown below (NOTE: This may take hours to complete):
... First a check will be made for the RPM packages which
can be installed or updated via yum (you will need to
answer yes if any packages are found ...
... After the yum installation of RPM packages completes,
most of the packages defined in the include/data/packages.tsv
(or include/data/packages.x86_64.tsv) file will be
installed using custom scripts (some will be binary installs
and some will build from source) ...
-------------------------------------------------------------------------------
***ERROR*** Following package updates failed:
snort unicornscan wireshark autohide colorful_tabs dictionarysearch
image_zoom LinkChecker long_titles quickrestart reloadevery tidy webdeveloper
Check the log files under: /root/nst/tmp/updates for details
-------------------------------------------------------------------------------
make: *** [tsv-update] Error 1
The above output shows that several of the "automatic" package installations failed. This typically occurs for one of the following reasons:
- The package required that a X desktop be running. All of the firefox add-ons fall into this category (autohide, colorful_tabs, ...).
- The package is no longer available. This often occurs when a new release of a package comes out and indicates that the file: "include/data/packages.tsv" needs to be updated. This is what happened with snort (they were at release 2.7.0 where as the "include/data/packages.tsv" file was configured for 2.6.1.5).
- A library was not found because the ldconfig command needs to be run (this is what caused the issue with wireshark shown in the above output).
- The package has dissappeared, moved, or no longer builds cleanly (this is what occurred with unicornscan in the output shown above).
When you encounter packages which fail to install cleanly, you should be able to review the entire log related to the installation by looking in the: "tmp/updates" directory (for example, the file: "tmp/updates/unicornscan.log" contained many error messages).
While the "make package-update" will automate the installation of 98% of the packages needed to build a full NST system, there are a handful of packages which require user interaction to complete the installation. For each of these packages, you should find an appropriately named script under the: "src/bin" directory.
To see what packages still need to be installed, the "make package-check" command can be used. The following demonstrates how to identify which packages still need to be installed, and then installs the ieee80211 package:
autohide (version 1.1.5) [failed] colorful_tabs (version 2.0.1) [failed] dictionarysearch (version 2.0.1) [failed] ieee80211 (version 1.2.16) [failed] image_zoom (version 0.2.7) [failed] ipw2200 (version 1.2.1) [failed] LinkChecker (version 0.6.1) [failed] long_titles (version 1.2.4) [failed] madwifi (version 0.9.3) [failed] metasploit (version 2.7) [failed] nessus (version 2.2.9) [failed] netw (version 5.34.0) [failed] ntop (version 3.3) [failed] quickrestart (version 1.1.0) [failed] reloadevery (version 2.0) [failed] snort (version 2.6.1.5) [failed] snort_inline (version 2.6.1.5) [failed] snort-mysql (version 2.6.1.5) [failed] snorter (version 2.1) [failed] tidy (version 0.8.3.9) [failed] vmware-tools (version 1.0.3) [failed] webdeveloper (version 1.1.4) [failed]
... Lots of output followed by a series of questions you must
answer, followed by the build/install of the package ...
-------------------------------------------------------------------------------
SUCCESS: Successfully installed ieee80211
-------------------------------------------------------------------------------
NOTE: You may need to re-build modules which depended upon
this one. Look for failures when you run "make package-check".
ieee80211 (version 1.2.16) [ok]
Building a ISO Image
To build the NST ISO image, one runs the following command:
... You will see a lot of output and it will take awhile to build ...
-rw-r--r-- 1 root root 509790208 2007-08-25 17:27 src/nst-1.7.0.iso
You will always see a [warn]] indicator when the java package is installed. At the time of this writing, you may also expect to see a [warn] indicator for the cups and clamav packages (this should be resolved prior to the final release). Other packages flagged by a [warn] and/or [failed]] indicator, would indicate that the layout of the package has changed since its installation script was last updated (the NST developers will get to it before the next release).
Now that a ISO image has been produced, one can do any of the following:
- Burn the image to a CD or DVD (we've had good luck with CDRW media during our development). You might be able to use the: "make cd", "make cdrw" and/or "make dvd+rw" (see the output of: "make help" for more information). NOTE: If you are developing inside a virtual machine, you probably won't be able to burn a disk immediately. Instead, you'll need to transfer the image to a different system which has a burner.
- Transfer the ISO image to a different machine (we typically use scp, if you are a Windows user, you may want to consider WinSCP).
- Boot directly from the ISO image using a VMware product or Virtual PC. NOTE: At the time of this writing, there is an issue with NST and the recommended virtual LSI SCSI controller in VMware Server (beta testers should opt for the BusLogic controller when creating new virtual machines for the time being).
Tweaking The Configure/Build Process
During the configuration and build process, several files in the: "${HOME}/.nst" directory are examined. These files are optional (you don't have to create any of them). However, creating these files, allows one to customize the results of configuring and building the NST ISO image without having to specify a lot of command line options to the configure script.
The top level: "README" file contains a lot of information about the various configuration options available.
The Magic "${HOME}/.nst" Directory
By default, the configuration and build process will look under the: "${HOME}/.nst" directory for build customization files. You can append the: "--config-dir DIRECTORY" option to the ./configure command if you would like to use a different directory.
"${HOME}/.nst/configure.sh"
The first file you will likely want to create is the: "${HOME}/.nst/configure.sh" file. This file can be used to set MANY of the available options supported by the top level ./configure script.
# Set a custom password and disable prompt to set the initial password
NSTPASSWD="MY_NST_PASSWORD";
NSTPASSWD_PROMPT="false";
# Uncomment if you want the extra development packages added (src/packages/devextra)
#PKGCATS="${PKGCATS} devextra";
# Uncomment if you want the "extra" disk packages added (src/packages/diskextra)
#PKGCATS="${PKGCATS} diskextra";
# Where source code is downloaded to and built under
BUILDROOT="/usr/local/src";
# Default boot mode
NSTBOOT="desktop";
# Set to true if you want to build the NST documents, false for ISO
ON_LINE_DOCS="false";
# Compress files so ISO will likely be small enough for a CD?
NSTCOMPRESSED_ISO="true";
# Take time to strip executables to produce a smaller ISO image?
NSTSTRIPPED="true";
There are many other settings you can add to this file, but those shown above will cover 95% of what you'll want to tweak. After you run the configure script, you can check the resulting configuration by looking at the file: "config/config.sh" (the top portion of this file can also be used as a reference for other variables you might wish to set).
NOTE: You will need to re-run the ./configure script after making changes to this file. Also, make sure you set the permissions of this file to: 0600 - especially if you decide to put a clear text password in it.
"${HOME}/.nst/disable.txt"
As time has passed, many packages have been added and removed from the NST distribution. As packages are removed from the NST distribution, they are often added to the: "include/data/disable.txt" file to indicate that they should not be included in the build.
During the build process, the file: "${HOME}/.nst/disable.txt" will be searched for before falling back to: "include/data/disable.txt". This allows one to customize what packages are omitted from the build.
If you are interested in omitting packages (or categories), copy the file: "include/data/disable.txt" to: "${HOME}/.nst/disable.txt" and then edit it's contents. You should find comments at the top of the file indicating how one disables additional categories and/or packages.
"${HOME}/.nst/post_configure.sh"
This script is sourced (if it exists) after each invocation of the ./configure command. It allows one to perform "post configuration" updates to the build environment. Very few people will have need to implement this script. But if you do, your script will have access to all of the variables found in: config/config.sh (useful for locating files making up the system).
"${HOME}/.nst/post_install.sh"
This script is run (if it exists) after all of the packages have been installed during the build process (when running: make). This script can be VERY useful to customizing your resulting NST ISO.
The following example, does the following:
- Defines some function which will adjust files in the build area prior to forming the ISO image.
- Invokes a subset of the defined functions for the features we want installed.
#
# Customized files PRIOR to building ISO
#
# install_custom_isolinux_cfg
#
#
install_custom_isolinux_cfg() {
if [ -d "${NSTHOMEDIR}/base/isolinux" -a "${CFGDIR}/isolinux.cfg" ]; then
${CP} -f "${CFGDIR}/isolinux.cfg" "${NSTHOMEDIR}/base/isolinux";
fi
}
# add_setup_home
#
# Adds helper script "/root/bin/setup_home", to configure probe for home network
add_setup_home() {
create_dirs "${NSTUTILSDIR}/root/bin"
cat >| "${NSTUTILSDIR}/root/bin/setup_home" <<EOF
#!/bin/bash
#
# Detect/add swap space
#
auto_add_swap
#
# Enable NTP
#
/etc/rc.d/init.d/ntpd start >/dev/null 2>&1 &
#
# Enable home proxy
#
nstsetproxy --http-host 192.168.12.1 --http-port 3128
#
# Mount NFS drives
#
service portmap start
mount -t nfs 192.168.12.1:/home /home > /dev/null 2>&1 &
if [ ! -d /lan ]; then
mkdir /lan
fi
mount -t nfs 192.168.12.1:/opt /lan > /dev/null 2>&1 &
EOF
chmod +x "${NSTUTILSDIR}/root/bin/setup_home"
}
# config_ssh
#
# Add custom .ssh configuration (authorized_keys in particular)
config_ssh() {
if [ -d "${NSTINITRDDIR}" ]; then
${RM} -fr "${NSTINITRDDIR}/root/.ssh"
fi
create_dirs "${NSTUTILSDIR}/root/.ssh"
${CHMOD} 700 ${NSTUTILSDIR}/root/.ssh
for f in authorized_keys config id_rsa id_dsa; do
if [ -e "${HOME}/.ssh/${f}" ]; then
${CP} -p "${HOME}/.ssh/${f}" "${NSTUTILSDIR}/root/.ssh"
fi
done
}
#
# Enable CD eject on shutdown
#
enable_cd_eject() {
create_dirs "${NSTUTILSDIR}/sbin"
${LN} -s /usr/bin/eject ${NSTUTILSDIR}/sbin/halt.local
}
#
# Call features we want
#
install_custom_isolinux_cfg;
# enable_cd_eject;
config_ssh;
add_setup_home;
You can use this to customize pretty much anything in the build (add other packages, configure your network setttings, install wireless keys, etc).
As this file is sourced, you will have access to all of the variables defined in: "config/config.sh". Those that will be of most interest include:
- ${NSTLOCALDIR}
- This is the root directory which will appear as: /usr/local once the system has booted off of the Live CD. All files placed under here will be READ ONLY (don't put a configuration file which is likely to be editted here). This is where most executables and libraries are placed.
- ${NSTUTILSDIR}
- This directory contains a collection of files that will be copied to RAM once the Live CD boots and mounts the CD. Configuration files, small scripts, frequently used tools are often placed here. Be aware that any file placed here will consume RAM as the system runs.
- ${NSTINITRDDIR}
- This is the root directory which contains the files and directories which will end up in the initial RAM disk when booting from the Live CD. It typically contains small scripts, configuration files and enough executables and modules to get the CDROM mounted. You should avoid adding large files to this area.
