--- /dev/null
+<?xml version="1.0" encoding="iso-8859-1"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0 plus SVG 1.1//EN" "http://www.w3.org/2002/04/xhtml-math-svg/xhtml-math-svg.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml">
+<head><title>Welcome to the home of the GNUCOMO project.</title>
+ <link rel="stylesheet" type="text/css" href="gnucomo.css"/>
+</head>
+<body>
+<h1>GNUCOMO: GNU COmputer MOnitoring</h1>
+<p>
+<img src='logo.png' alt='Gnucomo Logo'/>
+</p>
+<h2>About the project</h2>
+<p>Welcome to the homepage of Gnucomo, the project to monitor the devices (mostly computers) in your network.
+The aim of the project is to build a set of applications that will help administrators to monitor
+networks as a whole for errors, attacks and security-breaches in a very user-friendly way.
+It resembles an intrusion detection system, but doesn't want to redo the efforts made by other projects.
+All types of possible entries are used as signals for detection of not normal situations.
+</p>
+<p>The gnucomo <a href='https://www.dewinter.com/mailman/listinfo/gnucomo'> mailing-list</a> can be used to ask questions or to join the project.</p>
+
+<h2>Goal and used technology</h2>
+<p>The project tries to achieve the following goals:</p>
+<ul>
+<li>Intrusion detection</li>
+<li>Detecting hacker attempts </li>
+<li>Early detection of system failures </li>
+<li>Exhaustion of system resources </li>
+<li>Capacity planning for future expansion </li>
+<li>Spotting bottlenecks in a system.</li>
+<li>Verifying system integrity </li>
+<li>Assistance with troubleshooting </li>
+<li>Perform post-mortem forensics </li>
+<li>Incident response system.</li>
+</ul>
+<p>Gnucomo is licensed according to the GNU Public License V2. We rely on the following free software to perform the tasks needed:
+</p>
+<ul>
+<li><a href="http://www.andromeda.nl/projects/">XML DOC</a> for transforming documentation to several formats like PDF, HTML.</li>
+<li><a href="http://www.postgresql.org/">PostgreSQL</a> for our main database. This is actually BSD-licensed since the revision this license is compatible with the GPL license.</li>
+<li><a href="http://gborg.postgresql.org/project/libpqxx/projdisplay.php">libpqxx</a> for the C++ interface to the PostgreSQL database server.</li>
+<li><a href="http://www.gnupg.org/">GnuPG</a> for secure transportation of the data when e-mail is used.</li>
+<li><a href='http://www.andromeda.nl/projects/AXE/AXE.html'>AXE</a> for the C++ modules used gcm_input (the application that read the data into the database.</li>
+</ul>
+
+ <h2>The core team</h2>
+ <p>
+ The core team has agreed to take responsibilities for parts of the projects.
+ </p>
+ <ul>
+ <li>Brenno de Winter: Project co-ordination, Website, Documentation, Mailinglist, Gnucomo test-server, Processing raw data in database into readable data.</li>
+ <li>Arjen Baart: Code review, XML-settings routines, Client rotation application, Transportation</li>
+ <li>Peter Roozemaal: Configuration for gnucomo-clients, security reviews overall architecture</li>
+ <li>Edwin Nadorp: Administrators interface</li>
+ <li>Andreas Sikkema: Windows logging client</li>
+ </ul>
+<p>
+<img src='devmeeting.jpg' alt='Core Team'/><br/>
+The core team on Gnucomo developer day 1.
+</p>
+
+<h2>Links</h2>
+<h3>Documentation</h3>
+ <p>
+ Gnucomo Development <a href='http://www.gnucomo.org/manifest.html'>Manifest</a><br/>
+ Gnucomo <a href='http://www.gnucomo.org/design.html'>Design documentation</a><br/>
+ Gnucomo <a href='http://www.gnucomo.org/manual.xhtml'>User manual</a><br/>
+ </p>
+<h3>Presentation</h3>
+<p>
+ <a href='http://www.gnucomo.org/presentation/bocholt_2002'>Presentation</a> held at the <a href='http://www.ccamp.de/'>CCAMP</a> in Bocholt on August 22nd 2002.
+ <br/>
+ <a href='http://www.gnucomo.org/presentation/amsterdam_dotnet_2003'>Presentation (in Dutch)</a> held at the dotnet event in the Amsterdam RAI.
+</p>
+<h3>Downloads</h3>
+<p>All documentation here is in XMLDoc format. Please be aware that this is still work in progress:
+</p>
+ <ul>
+ <li><a href="gnucomo-0.0.1.tar.bz2">Gnucomo-0.0.1</a> Oct 05, 2002 </li>
+ <li><a href="gnucomo-0.0.2.tar.bz2">Gnucomo-0.0.2</a> Nov 20, 2002 </li>
+ <li><a href="gnucomo-0.0.3.tar.bz2">Gnucomo-0.0.3</a> Dec 06, 2002 </li>
+ <li><a href="gnucomo-0.0.4.tar.bz2">Gnucomo-0.0.4</a> Feb 06, 2003 </li>
+ <li><a href="gnucomo-0.0.5.tar.bz2">Gnucomo-0.0.5</a> Feb 21, 2003 </li>
+ <li><a href="gnucomo-0.0.6.tar.bz2">Gnucomo-0.0.6</a> Jul 15, 2003 </li>
+ <li><a href="gnucomo-0.0.7.tar.bz2">Gnucomo-0.0.7</a> Aug 14, 2003 </li>
+ <li><a href="gnucomo-0.0.8.tar.bz2">Gnucomo-0.0.8</a> Sep 04, 2003 </li>
+ <li><a href="gnucomo-0.0.9.tar.bz2">Gnucomo-0.0.9</a> Dec 24, 2003 </li>
+ <li><a href="gnucomo-0.0.10.tar.bz2">Gnucomo-0.0.10</a> Oct 19, 2007 </li>
+ <li><a href="gnucomo-snapshot.tar.bz2">Gnucomo Development snapshot</a> Jan 11, 2007 </li>
+ </ul>
+
+<h3>Other</h3>
+<p>
+ A mailinglist for development has been created. All announces will be done here.
+</p>
+ <ul>
+ <li><a href='https://www.dewinter.com/mailman/listinfo/gnucomo'>Mailing-list</a>
+</ul>
+
+<h2>Known problems</h2>
+
+ <table>
+ <tbody><tr><th>Log Nr.</th><th>Subject</th><th>Synopsis</th><th>State</th><th>Prior</th></tr>
+
+<tr><td>17</td><td>GnuCoMo Computer Monitoring</td><td>ERROR in gcm_daemon: Attribute 'source_ip' specified more than once </td><td>SOLVED</td><td><img src="prior0.png"></td></tr>
+
+<tr><td>2</td><td>GnuCoMo Computer Monitoring</td><td>Log entries rejected by gcm_input</td><td>NOTED</td><td><img src="prior1.png"></td></tr>
+
+<tr><td>5</td><td>GnuCoMo Computer Monitoring</td><td>List of required and recommended software</td><td>ANALYZED</td><td><img src="prior1.png"></td></tr>
+
+<tr><td>10</td><td>GnuCoMo Computer Monitoring</td><td>Gcm_input may loose its input message</td><td>OPENED</td><td><img src="prior1.png"></td></tr>
+
+<tr><td>11</td><td>GnuCoMo Computer Monitoring</td><td>Indicate type of data to gcm_input</td><td>OPENED</td><td><img src="prior2.png"></td></tr>
+
+<tr><td>7</td><td>GnuCoMo Computer Monitoring</td><td>Notifications page</td><td>OPENED</td><td><img src="prior3.png"></td></tr>
+
+<tr><td>8</td><td>GnuCoMo Computer Monitoring</td><td>Related log_adv records on a log line</td><td>OPENED</td><td><img src="prior3.png"></td></tr>
+
+<tr><td>9</td><td>GnuCoMo Computer Monitoring</td><td>Input filename for gcm_input</td><td>OPENED</td><td><img src="prior3.png"></td></tr>
+
+</tbody></table>
+
+<br><br>
+
+</body>
+</html>
<?xml version="1.0"?>
<!DOCTYPE doc SYSTEM "/usr/local/xslt/doc.dtd">
<?xml-stylesheet type="text/xsl" href="/usr/local/xslt/html.xsl"?>
-<doc style="main.css">
+<doc style="gnucomo.css">
<!--
Gnucomo - Gnu Computer Monitoring Tutorial
Original author : Peter Roozemaal
- Version : $Revision: 1.2 $
+ Version : $Revision: 1.3 $
This document is prepared for XMLDoc. Transform to HTML,
LaTeX, Postscript or plain text with XMLDoc utilities and
<para><picture src='logo.png' eps='logo' scale='0.7'/></para>
<author>Peter Roozemaal<code><mathfox@xs4all.nl></code></author>
<author>Arjen Baart <code><arjen@andromeda.nl></code></author>
- <date>October 23, 2003</date>
+ <date>October 25, 2007</date>
<docinfo>
- <infoitem label="Version">0.1</infoitem>
+ <infoitem label="Version">0.2</infoitem>
<infoitem label="Organization">Andromeda Technology & Automation</infoitem>
<infoitem label="Organization">De Winter Information Solutions</infoitem>
</docinfo>
other GnuCoMo documentation, please report it by sending an e-mail to
the mailinglist: <code>gnucomo@dewinter.com</code>.
Please include in your bug report:
+</para>
<itemize>
<item>the GnuCoMo version you've found the bug in</item>
<item>the versions of the packages that gnucomo depends on</item>
</itemize>
-</para>
-
</section>
</chapter>
</para>
<para>
To be able to install and run GnuCoMo you'll need several other packages:
+</para>
<description>
<item tag='PostgreSQL'>
</description>
+<para>
The following packages are optional and provide additional functionality
to GnuCoMo:
+</para>
<description>
<item tag='GnuPG'>
</description>
-</para>
</section>
<section>
<code>create.sql</code>, which is in the <code>src/database</code>
directory of the Gnucomo distribution.
Type the following commands to create the database:
+</para>
<verbatim>
</verbatim>
+<para>
With the default installation of PostgreSQL on RedHat 8.0,
you will probably encounter an authentication problem when you try to
use the Gnucomo web interface. The problem will look somewhat like this:
<para>
Refer to PostgreSQL Administrator's guide, Chapter 4: Client Authentication.
You probably have this line in the /var/lib/pgsql/data/pg_hba.conf:
+</para>
<verbatim>
local all ident sameuser
</verbatim>
+<para>
(I know RedHat 8.0 does this). You need to change this into:
+</para>
<verbatim>
local all password
</verbatim>
+<para>
This tells PostgreSQL to allow any UNIX user to log into the database
as any database user on a local socket, using his database password.
-
</para>
</section>
<heading>The Gnucomo configuration file</heading>
<para>
+<label name='configuration'/>
All Gnucomo applications use a single configuration file, named
<code>gnucomo.conf</code> by default.
There are three places where this configuration file is stored.
Each parameter is a single value, i.e. there are no provisions to create
structured parameters.
Here is an example of what the Gnucomo configuration file may look like:
+</para>
<verbatim>
<?xml version='1.0'?>
</verbatim>
+<para>
At the root of the tree is an XML element, called <code>gnucomo</code>.
The direct children of the root element in the example above are
<code>database</code> and <code>logging</code>.
<code>logging</code> section specifies how much logging we want the
Gnucomo applications to generate and where we want that information to go.
</para>
+<para>
+Here is a full list of the parameters that are used by Gnucomo, subdivided
+in their various sections:
+</para>
+<itemize>
+
+ <item>
+ <strong>database</strong>
+ <para>
+ All parameters that provide Gnucomo applications access to the database.
+ </para>
+ <description>
+ <item tag='type'>
+ The type of the database server. Currently, only PostgreSQL is supported.
+ </item>
+ <item tag='host'>
+ The hostname of the system on which the database server runs.
+ </item>
+ <item tag='port'>
+ The TCP port used by the database server.
+ </item>
+ <item tag='name'>
+ The name of the database to use for Gnucomo.
+ </item>
+ <item tag='user'>
+ The user name with which to log in on the database server.
+ </item>
+ <item tag='password'>
+ The password used for logging in to the database server.
+ </item>
+ </description>
+ </item>
+
+ <item>
+ <strong>logging</strong>
+ <para>
+ The amount of logging generated by Gnucomo applications and where to send it.
+ </para>
+ <description>
+ <item tag='method'>
+ The method for sending log information.
+ </item>
+ <item tag='destination'>
+ Where the logging information is sent.
+ </item>
+ <item tag='level'>
+ Choose a higher level for more information.
+ </item>
+ </description>
+ </item>
+
+ <item>
+ <strong>logfile</strong>
+ <para>
+ Primarily used by <emph>logrunner</emph>.
+ A list of files that are scanned for system logging that in sent to the Gnucomo server.
+ There can be more than one <strong>logfile</strong> element in a Gnucomo configuration file.
+ </para>
+ <description>
+ <item tag='name'>
+ Name of the file to scan for logging, e.g. <code>/var/log/messages</code>.
+ </item>
+ <item tag='type'>
+ The type of the logging information.
+ Types supported by Gnucomo are <code>system log</code>,
+ <code>apache access log</code> and <code>apache error log</code>.
+ </item>
+ <item tag='fromhost'>
+ Hostname (not a FQDN) of the system for which the log entries are sent.
+ Only log entries that originate from this host are sent to the Gnucomo server.
+ This option applies to system log files that may have log entries for several hosts,
+ in case the system acts as a syslog server.
+ </item>
+ <item tag='filter'>
+ A regular expression to filter out log entries before sending to the Gnucomo server.
+ There may be several <strong>filter</strong> elements in a single
+ <strong>logfile</strong> element.
+ </item>
+ </description>
+ </item>
+
+</itemize>
</section>
<section>
The main gnucomo scripts go into this directory and the PHP class library can be copied
either in a subdirectory <code>classes</code> or <code>../phpclasses</code>.
Here's an example installation:
+</para>
<verbatim>
[gnucomo] # mkdir $DocumentRoot/gnucomo
[gnucomo] # cp src/web/* $DocumentRoot/gnucomo
[gnucomo] # cp -r src/phpclasses $DocumentRoot
</verbatim>
-</para>
</section>
<section>
<para>
The most direct way to invoke <code>gcm_input</code> maually and read
the log file through its standard input:
+</para>
<verbatim>
gcm_input -h server.gnucomo.org </var/log/messages
</verbatim>
+<para>
You'll have to explicitly specify the hostname of the object because
there is no way for <code>gcm_input</code> to know where the log
file comes from.
</para>
<para>
Feeding information into the Gnucomo database can also be done through
-a mail server, such as sendmail.
+a mail server, such as sendmail or postfix.
Create a mail alias that invokes <code>gcm_input</code> as a mailer program.
For example:
+</para>
<verbatim>
gnucomo: "|gcm_input"
</verbatim>
+<para>
With such an email alias, Gnucomo clients can simply send log files and
other reports to this email address:
+</para>
<verbatim>
mail gnucomo@server.gnucomo.org </var/log/messages
</verbatim>
-</para>
<para>
The third method to feed log files into the Gnucomo database is by
using <code>logrunner</code>.
+Where the previous methods can only feed complete logfiles as a whole,
+logrunner can feed partial logiles to the Gnucomo server.
+Logrunner keeps track of the growth of a logfile and converts the newly
+added log entries into an XML message suitable for gcm_input.
+A practical way to deploy logrunner is:
+</para>
+<verbatim>
+ /usr/local/bin/logrunner -1 | mail gnucomo@gnucomo.server
+</verbatim>
+<para>
+Note the '-1' option for logrunner. This option is needed to keep
+logrunner from generating multiple XML documents in one output stream.
+Which logfiles are scanned by logrunner is specified in the Gnucomo
+<ref to='configuration'>configuration file</ref>.
</para>
</section>
<heading>Viewing results</heading>
<para>
Use the web interface to review log entries, notifications and parameters.
+The <emph>Objects</emph> page shows statistics about the data that is collected
+for each monitored object.
+Clicking on the links in the statistics will display the specific data for that object.
+The icon of each object takes you to a form for editing information about that object,
+such as an identification code, physical location and services used by that object.
+</para>
+
+<para>
+After feeding the log entries of monitored objects to the Gnucomo server,
+these log entries can be analyzed by the Gnucomo analyzer daemon, <code>gcm_daemon</code>.
+The present version of Gnucomo does not yet support the operation of a real
+daemon, so you will have to start <code>gcm_daemon</code> explicitly.
+This performs a number of analyses on the log entries and updates the statistics
+for each object.
+When the <code>gcm_daemon</code> finds something out of the ordinary, it will
+create a <emph>Notification</emph>.
+Notifications are created to alert you to the fact that something may be wrong
+with that object.
+</para>
+<para>
+The checks made by Gnucomo are related to services that run on a monitored object.
+A few checks are predefined in Gnucomo.
+You can add more checks to tailor your situation through the <emph>Services</emph> page.
+Click on the icon of a specific service to edit the list of checks that are
+performed for log entries generated by that service.
+Each check consists of a <emph>pattern</emph> in the form of a regular expression
+and an action.
+For example, the following pattern matches a log entry from postfix which states that
+outgoing mail is delivered:
+</para>
+<example>
+ to=.+, relay=.+, delay=[0-9]{1,2}, status=sent \(250 .+\)
+</example>
+<para>
+The action is performed when the regular expression matches with a log entry.
+Each check is tagged with a rank number for defining the order in which the
+checks are applied.
+The first pattern that matches is used to perform the action.
+The very last check is a 'catch all' pattern, predefined in the Gnucomo database.
+When a pattern matches, one of four actions is executed:
+</para>
+<description>
+<item tag='ignore'>
+Simply ignore this log entry.
+</item>
+<item tag='notify'>
+Create a notification.
+The <emph>Issue</emph> for the notification is stated in the <emph>Argument</emph> field.
+</item>
+<item tag='abuse'>
+Add one to the number of abuses detected for the IP address or hostname that is mentioned in
+the log file.
+The argument for this action is almost always "$1", which is the first part of the pattern
+between parenthesis.
+Note that this requires a special form of regular expression in the pattern which will select
+certain parts as a sub expression.
+</item>
+<item tag='forgive'>
+Remove one abuse for the IP address of hostname in the argument.
+</item>
+</description>
+<para>
+A notification is always created with a specific <emph>Issue</emph> which states
+what Gnucomo thinks is going wrong.
+A number of issues are predefined in the Gnucomo database.
+You can create your own issues and have Gnucomo generate notifications with these
+issues on the Issues page.
+</para>
+</section>
+
+<section>
+<heading>Abuse and intrusion detection</heading>
+<para>
+Gnucomo maintains a list of IP addresses that have committed some kind of abuse.
+This can be attempts to send spam, attempts to intrude a system from the internet
+or any other kind of malicious action.
+These abuses are detected by creating the proper patterns in checks for log entries
+as discussed in the previous section.
+Each time an abuse is detected in the log, one abuse is added to the record for
+that <emph>Object</emph> and IP address.
+A reference to the log entry in which the abuse was detected is also maintained.
+When the number of abuses for a certain IP address exceeds the limit (32 in this
+version of Gnucomo), the status for the IP address is set to 'dropped'.
+You can review the list of abusing IP addresses by clicking on the 'Abuse list'
+in the <emph>Objects</emph> page.
+</para>
+<para>
+The most useful application of the abuse list is to maintain a firewall
+and block all IP addresses that have the 'dropped' status.
+A short shell script will do this job:
</para>
+<verbatim>
+#!/bin/sh
+#
+# Create a firewall script from the gnucomo abuses table
+#
+
+psql -h samos -t gnucomo arjen -c "select source from object_abuse
+ where status='dropped' and objectid=$1"|grep -v '^$'>/tmp/gnucomo-abuses
+
+while read ADDRESS
+do
+ echo iptables -I INPUT -s $ADDRESS -j DROP
+done < /tmp/gnucomo-abuses
+</verbatim>
+
</section>
</chapter>
Not the dash ('-') between the name and the version.
On many Linux systems that use the RedHat Package Manager (RPM), such
a list is easily obtained and fed into <emph>gcm_input</emph>:
+</para>
<verbatim>
rpm -qa | gcm_input -h example.gnucomo.org
</verbatim>
+<para>
You will have to repeat this procedure at regular intervals.
Each time you feed a new package list to Gnucomo, Gnucomo will
compare the new list with the package parameters in the database
You need to strip off two siffixes off the filenames to make it look like
a <code>rpm -qa</code> output.
The following script will do just that:
+</para>
<verbatim>
</verbatim>
+<para>
Suppose this script is stored as <code>ls-rpm</code>, you can apply it
like this:
+</para>
<verbatim>
ls /mnt/cdrom/RedHat/RPMS | ls-rpm | sort | uniq | gcm_input -h redhat-7.3
</verbatim>
+<para>
You can repeat this command to enter additional packages into the database.
For example, to add packages from other CD-ROMs or from a directory where
you keep downloaded updates.
projects / gnucomo.git / commitdiff