Easy deployment of software from source on multiple hosts

This page describes the technique we developed to make installing multiple software packages on many hosts as easy as possible, while retaining all the power and flexibility of compiling from source. Provided below are instructions and script templates to get you started.

Installation blues

As part of our work at Psychogenic, we perform the setup for a good number of Unix systems. Although recent Linux distributions come packed with software and RPM package managers/apt-get make fetching the rest easy, we still need to perform lots of installations from source.

In many cases, this involves little more than:

        $ ./configure
        $ make
        # make install

but this isn't always true. That means that for every install you must go over the README/INSTALL files and, if the process is midly complex, get re-acquainted with the quirks of each... Specific ./configure switches and options, environment variables, librarie paths, special user account creation, patches to apply, additions to /etc/ld.so.conf and more can make the whole process inefficient and tiresome.

Scripting a solution

Around the time of this writing, I had to setup 4 servers with various combinations of the following software from source:

cyrus-imapd-2.2.8.tar.gz
cyrus-sasl-2.1.19.tar.gz
db-4.2.52.tar.gz
fcgi-2.4.0.tar.gz
gd-2.0.28.tar.gz
gif2png-2.5.1.tar.gz
glib-2.4.0.tar.bz2
httpd-2.0.50.tar.gz
jpeg-6b.tar.gz
krb5-1.3.4.tar.gz
libpng-1.2.5.tar.gz
libxml2-2.6.11.tar.gz
mod_fastcgi-2.4.2.tar.gz
mydns-0.11.0.tar.gz
mysql-4.0.20.tar.gz
openssl-0.9.7d.tar.gz
pam_mysql-0.5.tar.gz
perl-5.8.5.tar.gz
php-5.0.0.tar.bz2
postfix-2.1.4.tar.gz
proftpd-1.2.10.tar.bz2
shared-mime-info-0.14.tar.gz
web-cyradm-0.5.4.tar.gz

Needless to say, attempting to memorize all the details of so many different installs was a bad idea and I didn't want to go through the "more README" process for every package on every server. The solution we developed is a simple yet flexible system comprised of:

• An automated installation processor;


• A series of install scripts, tailored to our needs, one for each package.

The idea is to download a given source distribution and perform the installation manually once, taking note of the entire process. Take notes, use the shell history command or the script utility to keep records of all steps. When that is done, you record this
valuable knowledge in a script based on this installation script template, by entering the list of commands after the comment. For instance, a very simple installation script might contain:

  # Enter the sequence of commands to perform the
  # installation HERE:   ./configure
   make
   make install  # install complete...
  cd $RETDIR

Have a look at two slightly more involved scripts: Cyrus SASL installation script and
Berkely DB installation script.

Making it work for you

To use our system to deploy software packages from source on any number of hosts, follow these instructions.

1. Create a source repository directory and store all the tar.gz/tar.bz2 source tar balls
   within (e.g. mkdir /home/user/sourcepackages);


2. Retrieve and save our automated installation processor script to the source repository
   directory (name it process.sh and do a chmod a+x ./process.sh);
3. Create a scripts/ directory below the source reporsitory
   (e.g. mkdir /home/user/sourcepackages/scripts)
4. Use the installation script template to create your install scripts as described previously. Save these in the
   scripts/ directory with the "versionless" name of the software source package (e.g. the script
   for db-4.2.52.tar.gz is scripts/db.sh and that for shared-mime-info-0.14.tar.gz is
   scripts/shared-mime-info.sh);



Have a look at two slightly more involved scripts: Cyrus SASL installation script and
Berkely DB installation script.

Making it work for you




To use our system to deploy software packages from source on any number of hosts, follow these instructions.





1. Create a source repository directory and store all the tar.gz/tar.bz2 source tar balls
   within (e.g. mkdir /home/user/sourcepackages);


2. Retrieve and save our automated installation processor script to the source repository
   directory (name it process.sh and do a chmod a+x ./process.sh);
3. Create a scripts/ directory below the source reporsitory
   (e.g. mkdir /home/user/sourcepackages/scripts)
4. Use the installation script template to create your install scripts as described previously. Save these in the
   scripts/ directory with the "versionless" name of the software source package (e.g. the script
   for db-4.2.52.tar.gz is scripts/db.sh and that for shared-mime-info-0.14.tar.gz is
   scripts/shared-mime-info.sh);



5. Edit the process.sh script and enter the name of the packages to install (in the appropriate order);


6. Run "./process.sh -e" to verify the list of commands to be executed and then run
   ./process.sh (with enough priveleges to perform the installations) to compile and install the software.
   

Having performed these steps, installing to another system involves little more than tar-ing up your /home/user/sourcepackages directory, transfering it over to the new host and running ./process.sh from there.

Hopefully this system will allow you extra time for more interesting challenges by automating the process of system setup. If you make additions to the system you feel are worth sharing with others, feel free to drop us a line. Enjoy!