2 <!DOCTYPE doc SYSTEM "/usr/local/xslt/doc.dtd">
3 <?xml-stylesheet type="text/xsl" href="/usr/local/xslt/html.xsl"?>
7 Gnucomo - Gnu Computer Monitoring Tutorial
8 Original author : Peter Roozemaal
9 Version : $Revision: 1.1 $
11 This document is prepared for XMLDoc. Transform to HTML,
12 LaTeX, Postscript or plain text with XMLDoc utilities and
13 XSLT sheets from http://www.andromeda.nl/projects/xmldoc/
18 <title>Gnucomo - User's Manual</title>
19 <para><picture src='logo.png' eps='logo' scale='0.7'/></para>
20 <author>Peter Roozemaal<code><mathfox@xs4all.nl></code></author>
21 <author>Arjen Baart <code><arjen@andromeda.nl></code></author>
22 <date>October 23, 2003</date>
24 <infoitem label="Version">0.1</infoitem>
25 <infoitem label="Organization">Andromeda Technology & Automation</infoitem>
26 <infoitem label="Organization">De Winter Information Solutions</infoitem>
35 <heading>What is GnuCoMo ?</heading>
38 The aim of the GnuCoMo (GNU COmputer MOnitoring) project is to build a
39 set of applications that will help administrators to monitor networks as
40 a whole for errors, attacks and security-breaches in a very user-friendly
41 way. It is free (GPL) software, sources are available and you are free
42 to modify it to suit your needs.
45 GnuCoMo collects system and application logfiles from the monitored
46 systems (this can be computers, routers and other devices) and stores them
47 in a database. This collected data is further interpreted and analyzed
48 until reports and alerts are generated. Because GnuCoMo can combine data
49 from multiple sources and longer timespans than common IDSs it is able
50 to recognize attacks that previously remained undetected.
53 Remeber, this manual is work in progress.
54 Although it contains some usefull pointers to help you get started
55 with Gnucomo, it is still quite incomplete.
56 You can help us improve this manual and, in fact, Gnucomo itself,
57 by sending your comments and suggestions to the mailinglist:
58 <code>gnucomo@dewinter.com</code>.
62 <heading>About this Document</heading>
65 This document aims to guide you through the installation of GnuCoMo on
66 a Unix (Linux) workstation. When we have completed the installation this
67 document will give a quick guided tour through the functionality GnuCoMo
68 offers. The procedures described here aim at getting the GnuCoMo demo
69 up and running as quickly as possible and may not be suited for your
70 production environment. I assume you have some experience with installing
71 software on your system.
75 This document is part of the Gnucomo package. Gnucomo is released under
76 the Gnu General Public License, see the file COPYING.
82 <heading>Reporting Bugs</heading>
85 If you think you have found a bug in GnuCoMo, this tutorial or the
86 other GnuCoMo documentation, please report it by sending an e-mail to
87 the mailinglist: <code>gnucomo@dewinter.com</code>.
88 Please include in your bug report:
91 <item>the GnuCoMo version you've found the bug in</item>
92 <item>what you did to provoke the error</item>
93 <item>the output (or error message) you got</item>
94 <item>(if relevant) which output you expected</item>
95 <item>processor type and kernel version you use</item>
96 <item>the versions of the packages that gnucomo depends on</item>
105 <heading>Installation</heading>
108 Most of the GnuCoMo installation can be done from an ordinary Unix
109 account; when you need root rights you will be told so explicitly. It is
110 however a good idea to create a gnucomo account.
114 <heading>Getting the software</heading>
117 GnuCoMo can be downloaded from the
118 <reference href='http://www.gnucomo.org/'>GnuCoMo website</reference>.
119 If you read this file you are likely to have a
120 copy of GnuCoMo on your system. If you are serious about using GnuCoMo
121 it would be a good idea to periodicly check the website for updates
125 To be able to install and run GnuCoMo you'll need several other packages:
128 <item tag='PostgreSQL'>
129 PostgreSQL is the database we use for GnuCoMo. Most
130 linux distributions provide ready to install packages for
131 PostgreSQL. We need at least the postgresql, postgresql-server,
132 postgresql-libs and postgresql-devel packages.
133 If you want to compile PostgreSQL from source: go to the
134 <reference href='http://www.postgresql.org/'>PostgreSQL homepage</reference>
135 and download the sources via one of the ftp sites. We need libpq++
137 Though we appreciate the performance improvements that the
138 PostgreSQL 7.3 server provides, we recommend sticking to the
139 PostgreSQL 7.2 client versions for now; at least until we've
140 solved the problems with the libpq++ libraries and PostgreSQL 7.3.
144 The C++ client interface for the PostgreSQL database server
145 is a separate project, not distributed with PostgreSQL.
146 This library replaces the old libpq++ library.
147 The place to find libpqxx is
148 <reference href='http://gborg.postgresql.org/project/libpqxx/projdisplay.php'>
149 the Gborg website</reference>.
150 Make sure you install libpqxx in a default library path or
151 set your LB_LIBRARY_PATH environment variable to include
152 the directory in which libpqxx is installed. The default
153 for libpqxx is /usr/local/libpqxx/lib.
157 PHP is used as programming language for gcm_deamon. If you
158 have packages you would want to install at least the php and
160 You can get PHP sources and documentation from the
161 <reference href='http://www.php.net/'>PHP website</reference>.
162 If you're compiling yourself, don't forget
163 to include PostgreSQL support.
167 We use XML for configuration that we can't (or don't want
168 to) store in the database and for documentation in XMLDOC
169 format. The libxml2 library usually comes with your linux
170 system and you would want to install both the libxml2 and
171 libxml2-devel packages. The libxml2 sources can be downloaded
172 via <reference href='http://xmlsoft.org/downloads.html'>the XMLSoft site</reference>.
176 It's not likely that you'll find precompiled AXE packages on
177 the net, so you'll have to compile from source. Get the AXE
178 sources from the <reference href='http://www.andromeda.nl/projects/AXE/AXE.html'>
179 AXE page</reference>;
180 you'll need version 0.3 or better.
181 Instructions on compiling and installing AXE are given later in
187 The following packages are optional and provide additional functionality
192 Recommended for encryption and signing of data that is transported
193 over the network. Not used at this moment.
197 We use XMLDoc to process our documentation. Download
198 and installation instructions can be found on the
199 <reference href='http://www.andromeda.nl/projects/xmldoc/xmldoc.html'>
200 XMLDoc site</reference>.
204 [HELPME: basic instructions]
207 <item tag='Python + tkinter'>
208 A GUI based configuration tool named MalfisInter (mi) is being
209 worked upon. This tool is programmed in Python and requires XML
219 <heading>Compiling</heading>
222 If you're lucky enough to find precompiled packages for your system and
223 have root permissions to install them, things are easy for you; otherwise
224 you would have to compile from source, which takes a bit more time if
225 you allready have the standard developer tools (C and C++ compilers,
226 make, (f)lex and yacc or bison) installed. You will need those tools
228 You will also need the GNU automake and autoconf packages.
231 Another essential utility is the bzip compressor package; you will need
232 it to unpack the archives. bzip2 and bunzip2 come standard with Linux,
233 but may not be available on older Unix distributions. Sources for bzip2
234 can be found at http://sources.redhat.com/bzip2/.
235 Most of the packages mentioned in this document come with detailed
236 compilation and installation instructions and I recommend to read the
237 README and INSTALL files before compiling and installing them.
241 <heading>Compiling AXE</heading>
244 For compiling AXE you need the X-windows (X11) headers and
245 libraries. Under Linux you might need to install the XFree86-devel
246 package, but these headers are usualy available whenever the C compiler
247 is installed (also on propriatary unixes).
254 <heading>Compiling GnuCoMo</heading>
257 The usual <code>./configure; make install</code> will compile
258 the binary applications and install <code>gcm_input</code> and
259 <code>logrunner</code> into <code>/usr/local/bin</code>.
260 You will need to copy the web interface scripts and gcm_daemon
261 manually to the appropriate directories.
269 <heading>Setting up the database</heading>
271 Gnucomo won't do anything usefull without a PostgreSQL database, so you will have to
273 Make sure you have the PostgreSQL server running and you have a valid userid
274 and password to access the server.
275 You also need to have the permission to create new databases.
276 If not, you will need to log in as the database administration user (usually
277 a user called <emph>postgres</emph>) and create a new user for the database.
281 To create the initial Gnucomo database, you need to use the SQL script
282 <code>create.sql</code>, which is in the <code>src/database</code>
283 directory of the Gnucomo distribution.
284 Type the following commands to create the database:
289 psql gnucomo -f create.sql
293 With the default installation of PostgreSQL on RedHat 8.0,
294 you will probably encounter an authentication problem when you try to
295 use the Gnucomo web interface. The problem will look somewhat like this:
300 Warning: pg_connect() unable to connect to PostgreSQL server:
301 FATAL 1: IDENT authentication failed for user "arjen"
306 Refer to PostgreSQL Administrator's guide, Chapter 4: Client Authentication.
307 You probably have this line in the /var/lib/pgsql/data/pg_hba.conf:
310 local all ident sameuser
313 (I know RedHat 8.0 does this). You need to change this into:
319 This tells PostgreSQL to allow any UNIX user to log into the database
320 as any database user on a local socket, using his database password.
326 <heading>The Gnucomo configuration file</heading>
329 All Gnucomo applications use a single configuration file, named
330 <code>gnucomo.conf</code> by default.
331 There are three places where this configuration file is stored.
332 Gnucomo applications first look for the configuration file in <code>/etc</code>,
333 and if the file is not found there, it should be in
334 <code>/usr/local/etc</code>.
335 Furthermore, you can have an additional configuration file in your home
336 directory, named <code>$HOME/.gnucomo.conf</code>.
337 The configuration file in <code>/etc</code> or in <code>/usr/local/etc</code>
338 holds the system-wide parameters.
339 The configuration file in a user's home directory may hold user-specific
341 The system-wide configuration is mandatory; any user-specific configuration
345 The configuration parameters are organized as a two-level tree, written
347 The first level of the tree constitutes the 'sections', referring to a
348 set of related paramaters.
349 The second level contains the parameters themselves.
350 Each parameter is a single value, i.e. there are no provisions to create
351 structured parameters.
352 Here is an example of what the Gnucomo configuration file may look like:
355 <?xml version='1.0'?>
356 <gnucomo version='0.0.8'>
358 <type>PostgreSQL</type>
359 <name>gnucomo</name>
360 <user>arjen</user>
361 <password>guess-again:-)</password>
364 <method>file</method>
365 <destination>/var/log/gcm_input</destination>
366 <level>0</level>
372 At the root of the tree is an XML element, called <code>gnucomo</code>.
373 The direct children of the root element in the example above are
374 <code>database</code> and <code>logging</code>.
375 These are the 'sections', corresponding to different aspects of the Gnucomo
377 As you may have guessed, the <code>database</code> section holds information
378 we need to access the Gnucomo database on the PostgreSQL server and the
379 <code>logging</code> section specifies how much logging we want the
380 Gnucomo applications to generate and where we want that information to go.
385 <heading>Getting the web interface up and running</heading>
387 Your primary view onto Gnucomo is through the web interface.
388 Gnucomo uses a web server, e.g. Apache, with PHP scripts to implement
390 At the present, the web interface is the only way to manage the
391 information in the Gnucomo database and view the results that
395 To be able to use the web interface, you need to copy the PHP
396 scripts from the <code>src/web</code> and <code>src/phpclasses</code>
397 directories into a directory accessible through the web server.
398 On most Red Hat Linux systems, this is usually in
399 <code>/var/www/html</code>.
405 <heading>And now...</heading>
407 If you like GnuCoMo [TODO]
408 Report success under Non-Linux [TODO]
414 <heading>Using Gnucomo</heading>
417 Once you have the database and the web interface set up, Gnucomo is almost
419 To put Gnucomo to work, you need to enter at least one object into the database
420 and create your methods to feed information to gnucomo.
424 <heading>The initial entries</heading>
427 Gnucomo maintains information about monitored systems such as servers,
428 work stations, switches and that sort of thing.
429 In Gnucomo, these are called <emph>objects</emph>.
430 The <emph>objects</emph> are distinguished by their name, usually
431 a hostname in your network. Or, better still, a fully qualified name
433 You can use the web interface to create an <emph>object</emph> by logging
434 in and selecting the picture that looks like a computer cabinet
435 in the button bar at the top.
436 Enter the object's name, for example <code>server.gnucomo.org</code>,
437 and click the <strong>Create</strong> button.
443 <heading>Feeding log files</heading>
446 Once an object is known by Gnucomo, i.e. the object's name is stored in the
447 database, you can start to fill the database with log entries.
448 Log entries are stored by a variety of programs into one or more log
449 files on the object's harddisk.
450 For example, system messages on most Linux distributions are put
451 in a file called <code>/var/log/messages</code>.
452 There are several ways to transfer the contents of a log file into the
453 Gnucomo database, each of which involves <code>gcm_input</code>.
454 <code>Gcm_input</code> is the Gnucomo application that reads various kinds
455 of raw input and tries to scan for as much information as possible.
456 The usefull information that is extracted from the input is stored
457 into the Gnucomo database.
461 The most direct way to invoke <code>gcm_input</code> maually and read
462 the log file through its standard input:
464 gcm_input -h server.gnucomo.org </var/log/messages
466 You'll have to explicitly specify the hostname of the object because
467 there is no way for <code>gcm_input</code> to know where the log
469 Take care not to feed the same log file another time through
470 <code>gcm_input</code>.
471 This will lead to duplicate log entries in the Gnucomo database.
474 Feeding information into the Gnucomo database can also be done through
475 a mail server, such as sendmail.
476 Create a mail alias that invokes <code>gcm_input</code> as a mailer program.
479 gnucomo: "|gcm_input"
481 With such an email alias, Gnucomo clients can simply send log files and
482 other reports to this email address:
484 mail gnucomo@server.gnucomo.org </var/log/messages
488 The third method to feed log files into the Gnucomo database is by
489 using <code>logrunner</code>.
494 <heading>Viewing results</heading>
496 Use the web interface to review log entries, notifications and parameters.
503 <heading>Managing system parameters</heading>
506 Besides log entries which represent the transient events that occur in
507 an object, Gnucomo also maintains the state of the object.
508 The state of an object is defined by the set of parameters and
510 Examples of parameters are the file systems, users, processes, installed
512 By managing those parameters with Gnucomo, you can hold a firm grip on
513 the state of your systems.
514 Especially on large sites with many workstations, servers, switches
515 and other objects, managing the state of all those systems is not
517 The following sections explain how Gnucomo can help.
521 <heading>Installed packages</heading>
524 Gnucomo can maintain a list of all packages installed on an object.
525 This list is stored in the database as parameters of class
526 <emph>package</emph>.
527 Each <emph>package</emph> parameter has one property: the version number
529 Gnucomo keeps an historical record of the version number for each package,
530 so you will know when the version of a package changes.
531 Furthermore, you can compare the installed packages of two objects, so
532 you can easily find the differences between two installations.
536 To maintain the installed packages of an object, all you need is to
537 feed a complete list into the Gnucomo database through <emph>gcm_input</emph>.
538 The package list is a text file with a package on each line of the form
539 <code>packagename-version</code>.
540 Not the dash ('-') between the name and the version.
541 On many Linux systems that use the RedHat Package Manager (RPM), such
542 a list is easily obtained and fed into <emph>gcm_input</emph>:
545 rpm -qa | gcm_input -h example.gnucomo.org
548 You will have to repeat this procedure at regular intervals.
549 Each time you feed a new package list to Gnucomo, Gnucomo will
550 compare the new list with the package parameters in the database
551 and notify you of any changes.
554 For keeping your systems up-to-date, it is often convenient to maintain
555 a repository of the most recent packages.
556 Gnucomo can help you with this task.
557 To enter the list of packages and updates from a distribution into the
558 Gnucomo database, you can create a virtual <emph>object</emph> and
559 create the <emph>package</emph> parameters for this object.
560 For example, to maintain a list of RPMs for RedHat Linux 7.3, create an
561 object (using the web interface) with the name <code>redhat-7.3</code>.
562 Using this object, enter the first batch of package parameters from a directory
563 listing of the CD-ROM.
564 However, you can not use the output from <strong>ls</strong> directly as
565 input for <strong>gcm_input</strong>.
566 You need to strip off two siffixes off the filenames to make it look like
567 a <code>rpm -qa</code> output.
568 The following script will do just that:
574 # Turn an 'ls' listing of RPM files into an 'rpm -qa' listing
575 # Reads a list of filenames, possibly preceeded by a directory and
576 # strips the directory path from the beginning and the two suffices
577 # from the end of each filename. For example, the name
578 # "/mnt/cdrom/RedHat/RPMS/kernel-2.4.20-13.7.i686.rpm" gets turned
579 # into a simple "kernel-2.4.20-13.7".
588 filename=`basename $filename .rpm`
591 rpm=`basename $filename .athlon`
594 rpm=`basename $filename .i386`
597 rpm=`basename $filename .i486`
600 rpm=`basename $filename .i586`
603 rpm=`basename $filename .i686`
606 rpm=`basename $filename .noarch`
616 Suppose this script is stored as <code>ls-rpm</code>, you can apply it
620 ls /mnt/cdrom/RedHat/RPMS | ls-rpm | sort | uniq | gcm_input -h redhat-7.3
623 You can repeat this command to enter additional packages into the database.
624 For example, to add packages from other CD-ROMs or from a directory where
625 you keep downloaded updates.
626 Remember, however, to use the <strong>-i</strong> option for "incremental"
628 Without this option, gcm_input will regard the list of packages as a <emph>full</emph>
629 list and remove any parameters that are not present in the new list.
632 With a list of available packages for a specific distribution in the Gnucomo
633 database, along with the list of packages installed on your systems, you
634 can easily obtain an overview of which packages you need to update.
635 To make this overview, use the web interface to compare the packages
636 on an object with the packages on the virtual object.