Revision as of 08:49, 28 January 2013

Caution

This page is a historical document of the decision making process when we made the major transition in the NST build framework from the 1.8.1 release to the 2.11.0 release. Do not treat this information as current.

Overview

While looking at the effort involved in moving NST to Fedora 10, we decided that we wanted to spend less time and effort on package maintenance and focus more on WUI development. To accomplish this, we will migrate away from our custom ISO script building techniques (that we started back in 2003) and move to the livecd-creator mechanism that has become a part of the standard Fedora distribution.

Good Things

Users will have access to yum package manager and the ability to enhance (or trash) their NST systems to their hearts content.

Using livecd-creator will keep us Fedora compatible.

Hard drive installation will be through the standard Fedora tool.

Migration to the next release of Fedora should be much less painful.

Much less effort required on our part for installing the numerous Fedora packages we depend on.

Ability for creating our own yum repository (allowing one to update their NST system using a standard yum update invocation).

Bad Things

ISO will not fit on CD (will require DVD).

Build times may be slower and it may be difficult to test individual package installations without doing a full build.

General Build Guidelines

The building of the NST ISO image will then involve the following steps:

Verifying that selinux is in Permissive mode. After days of hair pulling, Ron finally discovered that the installation from a live boot to hard disk would fail if the ISO was originally created on a system with selinux in Disabled mode.

The building of a local yum repository for packages which we want to continue building ourselves (or are not found in the Fedora repositories). This will be located under the top level yum directory.

The building of the ISO using the standard livecd-creator package. This will involve many custom scripts for "tweaking" the default ISO image created. This will be located under the top level livecd directory.

Local Yum Repository

Initial Build

To build the NST Yum repository on a development machine (from scratch):

Checkout the NST source tree.

Run the top level ./configure script.

Change to the yum directory.

Run: make help | less to review what you can build.

Run make all to build the repository (the files will appear under the repo sub directory).

The command sequence looks something like the following:

 export SVNROOT=https://nst.svn.sourceforge.net/svnroot/nst
 svn co ${SVNROOT}/trunk nst
 cd nst
 ./configure
 make all

NOTE: If the configure stage determines that your system is missing some packages, it should report what packages you need to install.

Subsequent Builds

After the initial build, you can use the following to update:

 ./configure
 make all

IMPORTANT: The make dependencies rules won't necessarily detect changes to all of the source files related to a package. This means if you modify source code for a package, a make all under the yum area may NOT detect the change and may not rebuild the package. To force the building of a package, include NAME-remove on the command line. For example, to force the building of the nstwui and nst-icons packages, use the following:

 make nstwui-remove nst-icons-remove all

Rebuilding Repository

If you want to rebuild your entire repository, remove the repo sub directory and then build:

 rm -fr repo
 make all

Rebuilding Everything

If you want to re-download all of the source and rebuild your entire repository:

 make nuke
 make all

Single Build

Once the yum area is configured, you can force the building of a single package. For example, if you want to force the build of the nst-config package, run:

make nst-config-remove nst-config

Test Build

If you have a running instance of a NST system, you can force the building of a package and install it on the remote NST system. For example, to force the building of the nst-config and do a test installation on the NST at 192.168.100.10, run:

make HOST=192.168.100.10 nst-config-remove nst-config-probe-install

ISO

Normal Build

You should be in the top level directory in order to build the ISO image.

To build the ISO image:

./configure
make iso

For more possibilities use the help target:

make help | less

There are ISO specific targets under the livecd directory:

cd livecd
make help | less

Clean Build

For a clean build:

make clean
./configure
make iso

Live USB Build

You can use the "liveusb USBDEV=USB_DEV" target to install the NST ISO image onto a USB flash drive (these drives are also known as: memory sticks and thumb drives). This will take a considerable amount of space on your thumb drive (you should have at least 1.5 GB of free space).

Before attempting this, you should review the How to create and use Live USB page at the Fedora Wiki. This page offers some useful troubleshooting tips as well.

The following demonstrates the make invocation to install the NST onto a USB flash drive. In this example invocation, I plugged in my flash drive and determined (through fdisk -l) that my flash drive partition was /dev/sdb1. Do not assume that your system will have the same device mapping if you specify a disk partition other than your USB flash drive, YOU WILL LIKELY DESTROY THE DATA ON THAT PARTITION!

make liveusb USBDEV=/dev/sdb1

NOTE: This command can only be used after the NST ISO image has already been created.

NOTE: The flash drive should not be mounted (or at least it does not need to be mounted) when invoking this make target.

You may specify a persistent overlay value using: OVERLAY=SIZE_IN_MB. This persistent overlay is used between reboots to persist changes you may make to your system. If omitted, this value defaults to 512. You can specify a value up to 2048. The following example demonstrates a invocation setting up a 1024 MB overlay:

make liveusb USBDEV=/dev/sdb1 OVERLAY=1024

liveusb-creator Notes

If you have booted the NST system from a "live" DVD image, you can use the Live USB Creator interface from the fluxbox menu system to install the "live" version of NST directly onto a USB memory stick. To do so:

Bring up the Fluxbox desktop (either on native X over via VNC).

Right click on the desktop to pull up the menu.

Select System Utilities|Create Live USB Disk.

On a GNOME desktop, you can find the Fedora LiveUSB Creator tool on the Applications|System Tools menu.

Disk Preparation

The "How to create and use Live USB" page has a lot of useful information on preparing and testing USB memory sticks. Here are a couple of items we've noticed:

If you use a FAT 16 filesystem, the maximum overlay image size will be limited to 2048MB

If you have a large USB memory stick (8GB for example), you may be better off using FAT 32.

You can format a partition with a FAT 32 file system via (change /dev/sdu1 as appropriate):

mkdosfs -F 32 -n usbdrive /dev/sdu1

If you use gparted to setup your partitions, make sure that the boot flag is set on the partition you install to (otherwise Live USB Creator will complain about the partition).

We have seen Live USB Creator fail to update the MBR on the USB device. Refer to: "How to create and use Live USB" for trouble shooting this situation.

ISO Selection

It is not obvious how to use the graphical interface of Live USB Creator to install from the booted ISO image. To tell Live USB Creator to use the ISO image which is currently booted, you need to:

Press the Browse button inside the Use existing Live CD group.

Ignore the normal browse controls! Instead, type in the device assigned to your CD/DVD drive (it should be something similar to: /dev/sr0).

Press the Enter key after typing the device name.

A message should appear in the log window. Something along the lines of:

sr0 selected

The Create Live USB button should activate (if a target device is selected).

Verify (or change) the Target Device

Specify the amount of Persistent Storage you'd like to allocate.

Press the Create Live USB button to begin the installation.

Unfortunately, this /dev/sr0 trick does not always seem to work. If it doesn't work from the GUI tool, try reading the man page for the liveusb-creator command line tool. You might be able to get it to work from the command line by including the --noverify option.

USB Installation

The NST distribution can be installed onto and run directly from a memory stick. There are two ways in which one can perform the installation.

As a NST Live installation.

As a full NST hard disk installation.

Each method has its advantages and disadvantages.

Yum Tips

Yum Reference Documentation

12 YUM Tips and Tricks - Yum Info

RPM Build Tips

RPM Reference Documentation

Maximum RPM - Taking the RPM Package Manager to the Limit
Fedora RPM Guide - Fedora RPM Reference Guide
RPM, %config, and (noreplace) - Configuration file update and replacement

Tips on RPM "%macros"

To find the secret "%macros", try commands like:

rpmbuild --showrc | grep _ln
rpmbuild --showrc | grep share

To find RPM query format macros:

rpm --querytags | less

Values Passed to the %pre, %post, %preun, and %postun Scripts

When the first version of a package is installed, its %pre and %post scripts will be passed an argument equal to 1.
When the last version of a package is erased, its %preun and %postun scripts will be passed an argument equal to 0.
When upgrading the value passed will be greater than 1 (1 for version already installed, plus 1 for version being installed).

Example: %pre, %post, %preun, and %postun Scripts for the 'ntop' RPM Package

SysV and LSB Init Service Scripts

%pre
#
# ntop package first install:
#
# Add an 'ntop' group and user...
if [ $1 -eq 1 ]; the
  /usr/bin/getent group %{name} >/dev/null || /usr/sbin/groupadd -r %{name};
  /usr/bin/getent passwd %{name} >/dev/null || \
    /usr/sbin/useradd -r -g %{name} -d %{_localstatedir}/lib/ntop -s /sbin/nologin -c "ntop" %{name} >/dev/null || :;
fi

%post
#
# ntop package first install:
#
# Add an 'ntop' service and set the boot state to: 'disabled'
if [ $1 -eq 1 ]; then
  /sbin/chkconfig --add %{name} &> /dev/null || :;
  /bin/systemctl disable %{name}.service &> /dev/null || :;
fi

%preun
#
# ntop package removal:
#
# Stop a running 'ntop' service and remove start/stop links...
if [ $1 -eq 0 ]; then
  /bin/systemctl stop %{name}.service &> /dev/null || :;
  /sbin/chkconfig --del %{name} &> /dev/null || :;
fi

%postun
#
# Package removal:
#
# Remove group and user 'ntop' from system...
if [ $1 -eq 0 ]; then
  /usr/sbin/usergroup %{name} 2>/dev/null || :;
  /usr/sbin/userdel -f %{name} 2>/dev/null || :;
fi
#
# ntop package upgrade:
#
# If 'ntop' was running, stop it and restart it with the upgraded version...
if [ $1 -ge 1 ]; then
  /bin/systemctl try-restart %{name}.service &> /dev/null || :;
fi

Systemd Service Control

%pre
#
# ntop package first install:
#
# Add an 'ntop' group and user...
if [ $1 -eq 1 ]; the
  /usr/bin/getent group %{name} >/dev/null || /usr/sbin/groupadd -r %{name};
  /usr/bin/getent passwd %{name} >/dev/null || \
    /usr/sbin/useradd -r -g %{name} -d %{_localstatedir}/lib/ntop -s /sbin/nologin -c "ntop" %{name} >/dev/null || :;
fi

%post
#
# ntop package first install:
#
# Add an 'ntop' service and set the boot state to: 'disabled'
if [ $1 -eq 1 ]; then 
  /bin/systemctl --no-reload disable %{name}.service &> /dev/null || :;
fi
/bin/systemctl --system daemon-reload &> /dev/null || :

%preun
#
# ntop package removal:
#
# Stop a running 'ntop' service and remove start/stop links...
if [ $1 -eq 0 ]; then
  /bin/systemctl stop %{name}.service &> /dev/null || :;
  /bin/systemctl --no-reload disable %{name}.service &> /dev/null || :;
fi

%postun
#
# Package removal:
#
# Remove group and user 'ntop' from system...
if [ $1 -eq 0 ]; then
  /usr/sbin/usergroup %{name} 2>/dev/null || :;
  /usr/sbin/userdel -f %{name} 2>/dev/null || :;
fi
#
# ntop package upgrade:
#
# If 'ntop' was running, stop it and restart it with the upgraded version...
if [ $1 -ge 1 ]; then
  /bin/systemctl try-restart %{name}.service &> /dev/null || :;
fi
/bin/systemctl --system daemon-reload &> /dev/null || :

HowTo: Show Spec File Scripts Within An RPM Package

Example: Show all scripts within the RPM package: "php-pear-Mail"

rpm -q php-pear-Mail --scripts | less

Example: To show post (%post) install scripts for RPM package: "nstwui"

rpm -q nstwui --queryformat "%{POSTIN}" | less

The Available RPM "Groups"

The Group: line in a spec file should be set to one of the group names allowed by Fedora. Refer to the file: "/usr/share/doc/rpm-*/GROUPS" on your development system to see what group names are available.

Downloading/Installing Source For A Fedora Package

You can use the yumdownloader and rpm commands to download (and install) the source code for most packages which can be yum installed. For example, the following demonstrates how to get the source code for the vnc-server package:

yumdownloader --source vnc-server
rpm -ivh vnc-4.1.3-1.fc10.src.rpm
ls ~/rpmbuild/SPECS ~/rpmbuild/SOURCES

How To Extract Files from an RPM Package Without Installing

First use the "yumdownloader" command to get the RPM package of interest. Then use the "rpm2cpio" and "cpio" utilities for conversion to "cpio" format and extraction. The example below extracts the "curl" package to directory: "/tmp/curl":

[root@shopper2 tmp]# mkdir -p /tmp/curl;

[root@shopper2 tmp]# yumdownloader curl;
curl-7.27.0-5.fc18.x86_64.rpm

[root@shopper2 tmp]# cd /tmp/curl/;

[root@shopper2 curl]# rpm2cpio /tmp/curl-7.27.0-5.fc18.x86_64.rpm | cpio -idmv;
./usr/bin/curl
./usr/share/doc/curl-7.27.0
./usr/share/doc/curl-7.27.0/BUGS
./usr/share/doc/curl-7.27.0/CHANGES
./usr/share/doc/curl-7.27.0/COPYING
./usr/share/doc/curl-7.27.0/FAQ
./usr/share/doc/curl-7.27.0/FEATURES
./usr/share/doc/curl-7.27.0/MANUAL
./usr/share/doc/curl-7.27.0/README
./usr/share/doc/curl-7.27.0/RESOURCES
./usr/share/doc/curl-7.27.0/TODO
./usr/share/doc/curl-7.27.0/TheArtOfHttpScripting
./usr/share/man/man1/curl.1.gz
1016 blocks

[root@shopper2 curl]# ls -alR /tmp/curl;
/tmp/curl:
total 0
drwxr-xr-x  3 root root  60 Jan 28 03:20 .
drwxrwxrwt 16 root root 460 Jan 28 03:30 ..
drwxr-xr-x  4 root root  80 Jan 28 03:20 usr

/tmp/curl/usr:
total 0
drwxr-xr-x 4 root root 80 Jan 28 03:20 .
drwxr-xr-x 3 root root 60 Jan 28 03:20 ..
drwxr-xr-x 2 root root 60 Jan 28 03:20 bin
drwxr-xr-x 4 root root 80 Jan 28 03:20 share

/tmp/curl/usr/bin:
total 152
drwxr-xr-x 2 root root     60 Jan 28 03:20 .
drwxr-xr-x 4 root root     80 Jan 28 03:20 ..
-rwxr-xr-x 1 root root 152488 Jan 15 08:43 curl

/tmp/curl/usr/share:
total 0
drwxr-xr-x 4 root root 80 Jan 28 03:20 .
drwxr-xr-x 4 root root 80 Jan 28 03:20 ..
drwxr-xr-x 3 root root 60 Jan 28 03:20 doc
drwxr-xr-x 3 root root 60 Jan 28 03:20 man

/tmp/curl/usr/share/doc:
total 0
drwxr-xr-x 3 root root  60 Jan 28 03:20 .
drwxr-xr-x 4 root root  80 Jan 28 03:20 ..
drwxr-xr-x 2 root root 240 Jan 28 03:20 curl-7.27.0

/tmp/curl/usr/share/doc/curl-7.27.0:
total 348
drwxr-xr-x 2 root root    240 Jan 28 03:20 .
drwxr-xr-x 3 root root     60 Jan 28 03:20 ..
-rw-r--r-- 1 root root   6044 Jul 20  2012 BUGS
-rw-r--r-- 1 root root 181982 Jan 15 08:41 CHANGES
-rw-r--r-- 1 root root   1044 Apr 25  2012 COPYING
-rw-r--r-- 1 root root  59952 Jul 20  2012 FAQ
-rw-r--r-- 1 root root   3904 Jul 20  2012 FEATURES
-rw-r--r-- 1 root root  36700 Jul 20  2012 MANUAL
-rw-r--r-- 1 root root   1608 Jan 15 08:41 README
-rw-r--r-- 1 root root   2287 Mar 19  2011 RESOURCES
-rw-r--r-- 1 root root  21751 Mar 19  2011 TheArtOfHttpScripting
-rw-r--r-- 1 root root  21874 Jun  3  2012 TODO

/tmp/curl/usr/share/man:
total 0
drwxr-xr-x 3 root root 60 Jan 28 03:20 .
drwxr-xr-x 4 root root 80 Jan 28 03:20 ..
drwxr-xr-x 2 root root 60 Jan 28 03:20 man1

/tmp/curl/usr/share/man/man1:
total 28
drwxr-xr-x 2 root root    60 Jan 28 03:20 .
drwxr-xr-x 3 root root    60 Jan 28 03:20 ..
-rw-r--r-- 1 root root 28391 Jan 15 08:41 curl.1.gz
[root@shopper2 curl]#

NST v2.x Guidelines: Difference between revisions