New user manual, based upon the draft TUTORIAL

[gnucomo.git] / doc / manual.xml

<?xml version="1.0"?>
<!DOCTYPE doc SYSTEM "/usr/local/xslt/doc.dtd">
<?xml-stylesheet type="text/xsl" href="/usr/local/xslt/html.xsl"?>
<doc style="main.css">

<!--
      Gnucomo - Gnu Computer Monitoring Tutorial
      Original author :  Peter Roozemaal
      Version         : $Revision: 1.1 $

      This document is prepared for XMLDoc. Transform to HTML,
      LaTeX, Postscript or plain text with XMLDoc utilities and
      XSLT sheets from http://www.andromeda.nl/projects/xmldoc/
-->

<book>
<titlepage>
   <title>Gnucomo - User's Manual</title>
   <para><picture src='logo.png' eps='logo' scale='0.7'/></para>
   <author>Peter Roozemaal<code>&lt;mathfox@xs4all.nl&gt;</code></author>
   <author>Arjen Baart <code>&lt;arjen@andromeda.nl&gt;</code></author>
   <date>October 23, 2003</date>
   <docinfo>
      <infoitem label="Version">0.1</infoitem>
      <infoitem label="Organization">Andromeda Technology &amp; Automation</infoitem>
      <infoitem label="Organization">De Winter Information Solutions</infoitem>
   </docinfo>
   <abstract>
   </abstract>
</titlepage>

<toc/>

<chapter>
<heading>What is GnuCoMo ?</heading>

<para>
The aim of the GnuCoMo (GNU COmputer MOnitoring) project is to build a
set of applications that will help administrators to monitor networks as
a whole for errors, attacks and security-breaches in a very user-friendly
way. It is free (GPL) software, sources are available and you are free
to modify it to suit your needs.
</para>
<para>
GnuCoMo collects system and application logfiles from the monitored
systems (this can be computers, routers and other devices) and stores them
in a database. This collected data is further interpreted and analyzed
until reports and alerts are generated. Because GnuCoMo can combine data
from multiple sources and longer timespans than common IDSs it is able
to recognize attacks that previously remained undetected.
</para>
<para>
Remeber, this manual is work in progress.
Although it contains some usefull pointers to help you get started
with Gnucomo, it is still quite incomplete.
You can help us improve this manual and, in fact, Gnucomo itself,
by sending your comments and suggestions to the mailinglist:
<code>gnucomo@dewinter.com</code>.
</para>

<section>
<heading>About this Document</heading>

<para>
This document aims to guide you through the installation of GnuCoMo on
a Unix (Linux) workstation. When we have completed the installation this
document will give a quick guided tour through the functionality GnuCoMo
offers. The procedures described here aim at getting the GnuCoMo demo
up and running as quickly as possible and may not be suited for your
production environment. I assume you have some experience with installing
software on your system.
</para>

<para>
This document is part of the Gnucomo package. Gnucomo is released under
the Gnu General Public License, see the file COPYING.
</para>

</section>

<section>
<heading>Reporting Bugs</heading>

<para>
If you think you have found a bug in GnuCoMo, this tutorial or the
other GnuCoMo documentation, please report it by sending an e-mail to
the mailinglist: <code>gnucomo@dewinter.com</code>.
Please include in your bug report:

<itemize>
<item>the GnuCoMo version you've found the bug in</item>
<item>what you did to provoke the error</item>
<item>the output (or error message) you got</item>
<item>(if relevant) which output you expected</item>
<item>processor type and kernel version you use</item>
<item>the versions of the packages that gnucomo depends on</item>
</itemize>

</para>

</section>
</chapter>

<chapter>
<heading>Installation</heading>

<para>
Most of the GnuCoMo installation can be done from an ordinary Unix
account; when you need root rights you will be told so explicitly. It is
however a good idea to create a gnucomo account.
</para>

<section>
<heading>Getting the software</heading>

<para>
GnuCoMo can be downloaded from the
<reference href='http://www.gnucomo.org/'>GnuCoMo website</reference>.
If you read this file you are likely to have a
copy of GnuCoMo on your system. If you are serious about using GnuCoMo
it would be a good idea to periodicly check the website for updates
and bugfixes.
</para>
<para>
To be able to install and run GnuCoMo you'll need several other packages:

<description>
<item tag='PostgreSQL'>
   PostgreSQL is the database we use for GnuCoMo. Most
   linux distributions provide ready to install packages for
   PostgreSQL. We need at least the postgresql, postgresql-server,
   postgresql-libs and postgresql-devel packages.
   If you want to compile PostgreSQL from source: go to the
   <reference href='http://www.postgresql.org/'>PostgreSQL homepage</reference>
   and download the sources via one of the ftp sites. We need libpq++
   support for GnuCoMo.
   Though we appreciate the performance improvements that the
   PostgreSQL 7.3 server provides, we recommend sticking to the
   PostgreSQL 7.2 client versions for now; at least until we've
   solved the problems with the libpq++ libraries and PostgreSQL 7.3.
</item>

<item tag='libpqxx'>
   The C++ client interface for the PostgreSQL database server
   is a separate project, not distributed with PostgreSQL.
   This library replaces the old libpq++ library.
   The place to find libpqxx is
   <reference href='http://gborg.postgresql.org/project/libpqxx/projdisplay.php'>
   the Gborg website</reference>.
   Make sure you install libpqxx in a default library path or
   set your LB_LIBRARY_PATH environment variable to include
   the directory in which libpqxx is installed. The default
   for libpqxx is /usr/local/libpqxx/lib.
</item>

<item tag='PHP'>
   PHP is used as programming language for gcm_deamon. If you
   have packages you would want to install at least the php and
   php-pgsql packages.
   You can get PHP sources and documentation from the
   <reference href='http://www.php.net/'>PHP website</reference>.
   If you're compiling yourself, don't forget
   to include PostgreSQL support.
</item>

<item tag='libxml2'>
   We use XML for configuration that we can't (or don't want
   to) store in the database and for documentation in XMLDOC
   format. The libxml2 library usually comes with your linux
   system and you would want to install both the libxml2 and
   libxml2-devel packages.  The libxml2 sources can be downloaded
   via <reference href='http://xmlsoft.org/downloads.html'>the XMLSoft site</reference>.
</item>

<item tag='AXE'>
   It's not likely that you'll find precompiled AXE packages on
   the net, so you'll have to compile from source. Get the AXE
   sources from the <reference href='http://www.andromeda.nl/projects/AXE/AXE.html'>
   AXE page</reference>;
   you'll need version 0.3 or better.
   Instructions on compiling and installing AXE are given later in
   this document.
</item>
</description>


The following packages are optional and provide additional functionality
to GnuCoMo:

<description>
<item tag='GnuPG'>
   Recommended for encryption and signing of data that is transported
   over the network. Not used at this moment.
</item>

<item tag='XMLDoc'>
   We use XMLDoc to process our documentation. Download
   and installation instructions can be found on the
   <reference href='http://www.andromeda.nl/projects/xmldoc/xmldoc.html'>
   XMLDoc site</reference>.
</item>

<item tag='Apache'>
   [HELPME: basic instructions]
</item>

<item tag='Python + tkinter'>
   A GUI based configuration tool named MalfisInter (mi) is being
   worked upon. This tool is programmed in Python and requires XML
   and TkInter support.
</item>

</description>

</para>
</section>

<section>
<heading>Compiling</heading>

<para>
If you're lucky enough to find precompiled packages for your system and
have root permissions to install them, things are easy for you; otherwise
you would have to compile from source, which takes a bit more time if
you allready have the standard developer tools (C and C++ compilers,
make, (f)lex and yacc or bison) installed. You will need those tools
for GnuCoMo anyway.
You will also need the GNU automake and autoconf packages.
</para>
<para>
Another essential utility is the bzip compressor package; you will need
it to unpack the archives. bzip2 and bunzip2 come standard with Linux,
but may not be available on older Unix distributions. Sources for bzip2
can be found at http://sources.redhat.com/bzip2/.
Most of the packages mentioned in this document come with detailed
compilation and installation instructions and I recommend to read the
README and INSTALL files before compiling and installing them.
</para>

<subsection>
<heading>Compiling AXE</heading>

<para>
For compiling AXE you need the X-windows (X11) headers and
libraries. Under Linux you might need to install the XFree86-devel
package, but these headers are usualy available whenever the C compiler
is installed (also on propriatary unixes).
[TODO]
</para>

</subsection>

<subsection>
<heading>Compiling GnuCoMo</heading>

<para>
The usual <code>./configure; make install</code> will compile
the binary applications and install <code>gcm_input</code> and
<code>logrunner</code> into <code>/usr/local/bin</code>.
You will need to copy the web interface scripts and gcm_daemon
manually to the appropriate directories.
</para>


</subsection>
</section>

<section>
<heading>Setting up the database</heading>
<para>
Gnucomo won't do anything usefull without a PostgreSQL database, so you will have to
set this up first.
Make sure you have the PostgreSQL server running and you have a valid userid
and password to access the server.
You also need to have the permission to create new databases.
If not, you will need to log in as the database administration user (usually
a user called <emph>postgres</emph>) and create a new user for the database.
</para>

<para>
To create the initial Gnucomo database, you need to use the SQL script
<code>create.sql</code>, which is in the <code>src/database</code>
directory of the Gnucomo distribution.
Type the following commands to create the database:

<verbatim>

   createdb gnucomo
   psql gnucomo -f create.sql

</verbatim>

With the default installation of PostgreSQL on RedHat 8.0,
you will probably encounter an authentication problem when you try to
use the Gnucomo web interface. The problem will look somewhat like this:
</para>

<para>
<strong>
Warning: pg_connect() unable to connect to PostgreSQL server:
FATAL 1: IDENT authentication failed for user "arjen"
</strong>
</para>

<para>
Refer to PostgreSQL Administrator's guide, Chapter 4: Client Authentication.
You probably have this line in the /var/lib/pgsql/data/pg_hba.conf:

<verbatim>
   local   all     ident   sameuser
</verbatim>

(I know RedHat 8.0 does this). You need to change this into:

<verbatim>
   local      all           password
</verbatim>

This tells PostgreSQL to allow any UNIX user to log into the database
as any database user on a local socket, using his database password.

</para>
</section>

<section>
<heading>The Gnucomo configuration file</heading>

<para>
All Gnucomo applications use a single configuration file, named
<code>gnucomo.conf</code> by default.
There are three places where this configuration file is stored.
Gnucomo applications first look for the configuration file in <code>/etc</code>,
and if the file is not found there, it should be in
<code>/usr/local/etc</code>.
Furthermore, you can have an additional configuration file in your home
directory, named <code>$HOME/.gnucomo.conf</code>.
The configuration file in <code>/etc</code> or in <code>/usr/local/etc</code>
holds the system-wide parameters.
The configuration file in a user's home directory may hold user-specific
preferences.
The system-wide configuration is mandatory; any user-specific configuration
is optional.
</para>
<para>
The configuration parameters are organized as a two-level tree, written
in XML.
The first level of the tree constitutes the 'sections', referring to a
set of related paramaters.
The second level contains the parameters themselves.
Each parameter is a single value, i.e. there are no provisions to create
structured parameters.
Here is an example of what the Gnucomo configuration file may look like:

<verbatim>
   &lt;?xml version='1.0'?&gt;
   &lt;gnucomo version='0.0.8'&gt;
      &lt;database&gt;
         &lt;type&gt;PostgreSQL&lt;/type&gt;
         &lt;name&gt;gnucomo&lt;/name&gt;
         &lt;user&gt;arjen&lt;/user&gt;
         &lt;password&gt;guess-again:-)&lt;/password&gt;
      &lt;/database&gt;
      &lt;logging&gt;
         &lt;method&gt;file&lt;/method&gt;
         &lt;destination&gt;/var/log/gcm_input&lt;/destination&gt;
         &lt;level&gt;0&lt;/level&gt;
      &lt;/logging&gt;
   &lt;/gnucomo&gt;

</verbatim>

At the root of the tree is an XML element, called <code>gnucomo</code>.
The direct children of the root element in the example above are
<code>database</code> and <code>logging</code>.
These are the 'sections', corresponding to different aspects of the Gnucomo
system.
As you may have guessed, the <code>database</code> section holds information
we need to access the Gnucomo database on the PostgreSQL server and the
<code>logging</code> section specifies how much logging we want the
Gnucomo applications to generate and where we want that information to go.
</para>
</section>

<section>
<heading>Getting the web interface up and running</heading>
<para>
Your primary view onto Gnucomo is through the web interface.
Gnucomo uses a web server, e.g. Apache, with PHP scripts to implement
its user inetrface
At the present, the web interface is the only way to manage the
information in the Gnucomo database and view the results that
Gnucomo produces.
</para>
<para>
To be able to use the web interface, you need to copy the PHP
scripts from the <code>src/web</code> and <code>src/phpclasses</code>
directories into a directory accessible through the web server.
On most Red Hat Linux systems, this is usually in
<code>/var/www/html</code>.

</para>
</section>

<section>
<heading>And now...</heading>
<para>
If you like GnuCoMo [TODO]
Report success under Non-Linux [TODO]
</para>
</section>
</chapter>

<chapter>
<heading>Using Gnucomo</heading>

<para>
Once you have the database and the web interface set up, Gnucomo is almost
ready to go.
To put Gnucomo to work, you need to enter at least one object into the database
and create your methods to feed information to gnucomo.
</para>

<section>
<heading>The initial entries</heading>

<para>
Gnucomo maintains information about monitored systems such as servers,
work stations, switches and that sort of thing.
In Gnucomo, these are called <emph>objects</emph>.
The <emph>objects</emph> are distinguished by their name, usually
a hostname in your network. Or, better still, a fully qualified name
on the internet. 
You can use the web interface to create an <emph>object</emph> by logging
in and selecting the picture that looks like a computer cabinet
in the button bar at the top.
Enter the object's name, for example <code>server.gnucomo.org</code>,
and click the <strong>Create</strong> button.
</para>

</section>

<section>
<heading>Feeding log files</heading>

<para>
Once an object is known by Gnucomo, i.e. the object's name is stored in the
database, you can start to fill the database with log entries.
Log entries are stored by a variety of programs into one or more log
files on the object's harddisk.
For example, system messages on most Linux distributions are put
in a file called <code>/var/log/messages</code>.
There are several ways to transfer the contents of a log file into the
Gnucomo database, each of which involves <code>gcm_input</code>.
<code>Gcm_input</code> is the Gnucomo application that reads various kinds
of raw input and tries to scan for as much information as possible.
The usefull information that is extracted from the input is stored
into the Gnucomo database.
</para>

<para>
The most direct way to invoke <code>gcm_input</code> maually and read
the log file through its standard input:
<verbatim>
   gcm_input -h server.gnucomo.org &lt;/var/log/messages
</verbatim>
You'll have to explicitly specify the hostname of the object because
there is no way for <code>gcm_input</code> to know where the log
file comes from.
Take care not to feed the same log file another time through
<code>gcm_input</code>.
This will lead to duplicate log entries in the Gnucomo database.
</para>
<para>
Feeding information into the Gnucomo database can also be done through
a mail server, such as sendmail.
Create a mail alias that invokes <code>gcm_input</code> as a mailer program.
For example:
<verbatim>
   gnucomo:        "|gcm_input"
</verbatim>
With such an email alias, Gnucomo clients can simply send log files and
other reports to this email address:
<verbatim>
   mail gnucomo@server.gnucomo.org &lt;/var/log/messages
</verbatim>
</para>
<para>
The third method to feed log files into the Gnucomo database is by
using <code>logrunner</code>.
</para>
</section>

<section>
<heading>Viewing results</heading>
<para>
Use the web interface to review log entries, notifications and parameters.
</para>
</section>

</chapter>

<chapter>
<heading>Managing system parameters</heading>

<para>
Besides log entries which represent the transient events that occur in
an object, Gnucomo also maintains the state of the object.
The state of an object is defined by the set of parameters and
their values.
Examples of parameters are the file systems, users, processes, installed
packages, and so on.
By managing those parameters with Gnucomo, you can hold a firm grip on
the state of your systems.
Especially on large sites with many workstations, servers, switches
and other objects, managing the state of all those systems is not
a trivial task.
The following sections explain how Gnucomo can help.
</para>

<section>
<heading>Installed packages</heading>

<para>
Gnucomo can maintain a list of all packages installed on an object.
This list is stored in the database as parameters of class
<emph>package</emph>.
Each <emph>package</emph> parameter has one property: the version number
of the package.
Gnucomo keeps an historical record of the version number for each package,
so you will know when the version of a package changes.
Furthermore, you can compare the installed packages of two objects, so
you can easily find the differences between two installations.
</para>

<para>
To maintain the installed packages of an object, all you need is to
feed a complete list into the Gnucomo database through <emph>gcm_input</emph>.
The package list is a text file with a package on each line of the form
<code>packagename-version</code>.
Not the dash ('-') between the name and the version.
On many Linux systems that use the RedHat Package Manager (RPM), such
a list is easily obtained and fed into <emph>gcm_input</emph>:

<verbatim>
   rpm -qa | gcm_input -h example.gnucomo.org
</verbatim>

You will have to repeat this procedure at regular intervals.
Each time you feed a new package list to Gnucomo, Gnucomo will
compare the new list with the package parameters in the database
and notify you of any changes.
</para>
<para>
For keeping your systems up-to-date, it is often convenient to maintain
a repository of the most recent packages.
Gnucomo can help you with this task.
To enter the list of packages and updates from a distribution into the
Gnucomo database, you can create a virtual <emph>object</emph> and
create the <emph>package</emph> parameters for this object.
For example, to maintain a list of RPMs for RedHat Linux 7.3, create an
object (using the web interface) with the name <code>redhat-7.3</code>.
Using this object, enter the first batch of package parameters from a directory
listing of the CD-ROM.
However, you can not use the output from <strong>ls</strong> directly as
input for <strong>gcm_input</strong>.
You need to strip off two siffixes off the filenames to make it look like
a <code>rpm -qa</code> output.
The following script will do just that:

<verbatim>

#!/bin/sh
#
#   Turn an 'ls' listing of RPM files into an 'rpm -qa' listing
#   Reads a list of filenames, possibly preceeded by a directory and
#   strips the directory path from the beginning and the two suffices
#   from the end of each filename. For example, the name
#   "/mnt/cdrom/RedHat/RPMS/kernel-2.4.20-13.7.i686.rpm" gets turned
#   into a simple "kernel-2.4.20-13.7".

while read filename
do
   case $filename in
   *.src.rpm)
   ;;

   *)
      filename=`basename $filename .rpm`
      case $filename in
      *.athlon)
         rpm=`basename $filename .athlon`
      ;; 
      *.i386)
         rpm=`basename $filename .i386`
      ;; 
      *.i486)
         rpm=`basename $filename .i486`
      ;; 
      *.i586)
         rpm=`basename $filename .i586`
      ;; 
      *.i686)
         rpm=`basename $filename .i686`
      ;; 
      *.noarch)
         rpm=`basename $filename .noarch`
      ;; 
      esac
      echo $rpm
   ;;
   esac
done

</verbatim>

Suppose this script is stored as <code>ls-rpm</code>, you can apply it
like this:

<verbatim>
   ls /mnt/cdrom/RedHat/RPMS | ls-rpm | sort | uniq | gcm_input -h redhat-7.3
</verbatim>

You can repeat this command to enter additional packages into the database.
For example, to add packages from other CD-ROMs or from a directory where
you keep downloaded updates.
Remember, however, to use the <strong>-i</strong> option for "incremental"
with gcm_input.
Without this option, gcm_input will regard the list of packages as a <emph>full</emph>
list and remove any parameters that are not present in the new list.
</para>
<para>
With a list of available packages for a specific distribution in the Gnucomo
database, along with the list of packages installed on your systems, you
can easily obtain an overview of which packages you need to update.
To make this overview, use the web interface to compare the packages
on an object with the packages on the virtual object.
</para>
</section>

</chapter>

</book>
</doc>