Spam scanning investigation

[gnucomo.git] / ChangeLog

??? 12, 2020 - Release 0.0.13
====================================

gcm_input
----------

 - logrunner is statically linked so it can be copied to different systems,
   32 bit or 64 bit, without library dependencies.
 - Save the input in an XML file if it can not be stored in the database
   because the database is not available or the XML parser returns an error.
   The directory where the XML file is stored is defined in the configuration file,
   element "spool/directory".
   This solved PR10.

gcm_maintenance
-----------

 - Cleanup abuse records only for subnets smaller than /16.

gcm_daemon
-----------

 - Resolve a hostname before adding to the abuses.

Database
---------

 - Added a new group daemon and created additional permissions to the database.
 - Cleaned up the COLUMN recognized from the TABLE logng

web interface
-------------

 - Put the password for logging into the database between quotes (').
 - When removing an object, clean up logs, parameters and notifications.



Dec 12, 2007 - Release 0.0.12
====================================

gcm_input
---------

- New options added for logrunner:
   -l <file>: Read input from logfile <file>.
   -s <file>: Use <file> as the status file instead of the default.
   -v Verbose output

- Logrunner outputs debug information only when the -v option
  is added on the command line.

- Buffer overflow problem in logrunner with input lines of more than 4096
  characters fixed.

gcm_daemon
-----------

- Added a new script gcm_maintenance.php to cleanup the database
  and check referential integrity. Purging old log entries is
  removed from the gcm_daemon script.

web interface
-------------

- Added fields to edit the range of a DYNAMIC property.

- The index.html page in now obsolete and redirects to index.php

- Added an option to perform one action on multiple notifications at once.

- When blocking a subnet, maintain the references to the log table from
  all IP addresses in that subnet.

Nov 22, 2007 - Release 0.0.11
====================================

Database
---------

- Added the class definition for a filesystem parameter.
- New issue type: 'property out of range'.

gcm_input
---------

- Logrunner can select a specific host from a logfile, in case
  syslogd acts as a log server for multiple systems.
  This adds a new parameter (logile/fromhost) to the configuration
  file.

- Logrunner uses the official hostname (FQDN) instead of the
  hostname returned by gethostname().

- Handling of parameters is greatly improved.
  When creating a new parameter from an XML report which is fed into
  gcm_input, the class definition is used as a template to fill in
  the default values for the properties.
  The class is also used as a template when a new property is added
  to an existing parameter.

- DYNAMIC properties are now handled properly. Instead of making a
  'changed property' notification, the value is checked against the
  defined range for the property. An 'out of range' notification
  is created when this condition is detected.

- Added a new filter which can directly read the output
  of the UNIX df command. A brief description is added in
  the user manual.

- Bugfix: Segmentation fault when reading an rpm package list
  with empty lines.

- New program: spamdetect.
  Expirimental utility to log manually reported spam and have
  Gnucomo detect the spammer's IP address.

- Added a number of usefull scripts, mainly for use in Gnucomo clients.

web interface
-------------

- The buttonbar at the top of each page is now a fixed 'div' element
  instead of a framed page.
  Contributed by Edwin Nadorp.

PHP classes
-----------

- Added a PHP5 module for the configuration class.

Test
----

- The test script, gcmtest can run individual tests as opposed to running
  all tests in sequence.

Oct 19, 2007 - Release 0.0.10
====================================

- Improved method for finding PostgreSQL libraries in configure script.

Database
---------

- Fixed field declaration for PostgreSQL 7.4
- New tables: log_abuse, object_abuse and service_pattern
- Added general service patterns in the service 'ANY'
- Added new issue types.
- Log_adv and derived tables removed.

logrunner
---------

- Status files in /var/lib
- Rewrote most functions and datastructures into C++
- Use the Gnucomo (XML) configuration file

gcm_input
---------

- Accept '/' and '.' characters within the service name in a system log.
  When extracting the service name, the '/' character is not considered
  part of that name, though.
- Textual changes in parameter notifications
- Experimental start of database OO abstraction layer.
- The '-' character is part of the service name.
- Skip the Email header when reading XML input.

gcm_daemon
-----------

- Bugfixes in the analysis of sendmail logs.
- Send email about open notifications to an object's users.
- Added log analysis for spam and abuse in sendmail log entries.
- Added pattern check on log entries with the service_pattern table.
- Create notifications from log entries with pattern matching.
- Record the date when adding the number of abuses for an IP address
  in the abuse list.


web interface
--------------

- Added a form to edit user data.
- Added a view for the log analysis with a link from the log page.
- New page: Parameter classes administration.
- New page: Abuse list
- Added an interface to edit check patterns to the services page.
- Manually edit parameters.
- View logs from abusing IP addresses.
- Added an interface for editing issues.
- In the abuse list, IP addresses can be whitelisted.
- Improved interface for editing check patterns.

Test
----

- Added test data for Problem Report 15, 16 and 17

Documentation
---------------

- Added a list of related projects and introduced the concept of
  dynamic parameters in the manifest.

Dec 24, 2003 - Release 0.0.9
====================================

Database
---------

- Changed the type of log_adv_daemon_email.delay and log_adv_daemon_email.xdelay
  from time to interval. These delays can be more than 24 hours.

libgnucomo
----------

- Catch an exception if we can not setup a database transaction.


gcm_input
---------

- SQL_Escape(): Backslashes are correctly escaped with another backslash
- XML_Entities(): transform non-ASCII characters into hexadecimal entities.
- Do not add another parameter_notification record is the notification
  already exists for that parameter.
- Catch exceptions from the database library.
- Major redesign. All input is handled through XML. Raw input data is first
  transformed into an XML document for further processing.
  A collection of polymorphic classes handle the transformation of various
  input formats into XML.
- Classifying input data is done with a finite improbability calculation.

gcm_daemon
-----------

- Create separate notifications for different objects in service_check().
  Fixes problem report 23.
- Change the notation for delays from "days+hh:mm:ss" to "days hh:mm:ss. This
  fixes problem report 14.
- In linux_daemon_sendmail(), check the existence of '=' before using it as a
  field separator. Fixes problem report 15.


web interface
--------------

- Optionally show or hide removed parameters from the parameter
  comparison page.

Documentation
---------------

- Added a User Manual, based upon the draft TUTORIAL


Sep 04, 2003 - Release 0.0.8
====================================

Database
---------

- Fixed a few typos in the database creation script.
- BUGFIX: Secondary indices on log_notification were unique.
- Additional information in the 'usr' table: 'display_name' and 'email'.
- Added new issues and services.

gcm_input
---------

- Fixed a namespace problem in message.cpp
- Fixed a gcc 2 vs. gcc 3 problem in gcm_input.cpp
- Reject log entries that are found to be invalid.
- A date without the time for the '-d <date> option will
  assume midnight on that date.

gcm_dameon
----------

- In PHP, the method configuration::read() will also read
  a user-specific configuration from the home directory if a 
  PHP script is not run through the web server.
- Accept command argument '-c config' to use an alternate
  gnucomo configuration.
- New PHP methods:
   db::Result() - Returns the result from the last query.
   db::Field()  - Returns the value of a single field.
   db::new_notification() - Create a new notification in the database.
- Check the log table against the servies running on an object and
  create notifications if a service is not supposed to be available
  or is not known at all.

libgnucomo
----------

- Changed the gnucomo_database class to the new PostgreSQL
  library, libpqxx
- Overloaded gnucomo_database::Field() to include the Result
  from a query as an argument.

web interface
--------------

- New page to enter and modify services.
- The objects page provides an interface to edit the list of
  services and users for an object.

Test
----

- Reduced the amount of output from test scripts.
- Added output data that is expected from test scripts.
- New test: tests the database upgrade with gcm_daemon
- Overall test script: gcm_test. Runs all tests in sequence.


Aug 15, 2003 - Release 0.0.7
====================================

Database
---------

- Added index to the history table to improve performance.

gcm_input
----------

- Added a new section 'logging' with three configuration parameters:
   method       - Output method to use for logging.
   destination  - Name of the log output destination.
   level        - Log level: Verbose output if greater than 0.
- Added '-i' option for incremental parameter updates.
- Debug output to the log stream instead of cerr.
- Fixed namespace problems in XPath searches of the DOM.
- Moved string utility functions to a separate file.
- Different kinds of log files are parsed by a collection of objects
  of different classes, derived from the base class line_cooker
  Depending on the message content or the message_type element in
  XML, one of these objects is selected.
- Logrunner is integrated with gcm_input. Although its functionality
  is still limited, a connection between logrunner and gcm_input
  is beginning to form.

gcm_daemon
----------

- BUGFIX: Print an error message if a parameter does not have
  any history.
- BUGFIX: undefined variables and indices when processing sendmail logs.

web interface
--------------

- BUGFIX: Convert special characters for HTML (<, >, and &) into
  their entities.
- BUGFIX: Removed parameters were somtimes shown on the wrong side
  of the parameter difference page.
- Added performance measurement to the parameters page.
- Added editing of detailed object information.

Jul 15, 2003 - Release 0.0.6
====================================

- Gcm_input reads IRIX system logs.
- Gcm_input exits without reading any input if the database connection fails.
- Gcm_input extracts the hostname out of the 'From:' or 'Message-Id:' line
  of an email header.
- New member C++ function gnucomo_database::is_conected().
- New columns in the table 'log_adv_daemon_email': size, pri, relay,
  status_details and dsn.
- Added several indices for the table 'log_adv_daemon_email'.
- In phpclasses/db.class.php: Added the database connection string as
  an argument to the function copy_db_class.
- Gcm_daemon processes logs from sendmail.
- Fixed the PHP member function db::db_connect(). The Postgres connection
  string is now passed as an argument to that function.
- Gcm_input reads cooked log entries from an XML input stream.
- New database tables: notification_check, notification_check_buffer,
  notification_check_line and object_statistics.
- PHP function db::query() returns the result index.
- PHP function db::num_rows() accepts a result index as argument
  (default = 0 -> use result from the previous query).
- New PHP function db::fetch_object().
- Gcm_daemon gathers statistics on parameters, notifications, etc. for all objects.
- The Objects web page uses the new object_statistics table.
- The Log web page shows the log one day at a time.
- Removed parameters are displayed in a shaded style.

Feb 21, 2003 - Release 0.0.5
====================================

- The notifications web page keeps a record of each time a notification
  is displayed. Either on a listing or in detail.
- Improved the table layout of the objects web page.
- In the users page of the web interface, you can add a new user and
  make him/her a member of a group.
- Users can change their password through the users page of the web interface.
- Gcm_input also detects packages that are removed from the system.
- In gcm_input, determining the version number of a package in a RPM
  list is improved. Only the last one or two parts of the string that
  begin with a '-' and a number are considered the version.
- The C++ function gnucomo_database::new_notification() uses the SQL
  function currval() to obtain the identification number
  of the most recently created notification.
- The C++ class gnucomo_database prints the query on cerr, along
  with the error message when the query results in an error.
- Removed the 'Change Password' image from the main menu.
  Changing passwords is integrated in te users page.
- Added the 'form' class on table and td elements. This class is intended
  for borderless tables that are used to layout HTML forms.
- A new style 'h2.error' for error reports in the stylesheet.
  Put borders around the tables and cells in the web interface.
  Added right-alignment for numbers and fixed width for timestamps.
- Gcm_daemon maintains the last notification identifier.
- Setup the groups and database permissions. The three groups
  are: view, ops and admin.
- Added two new entries to the action table.
- All web interface pages use the page class. This provides for a
  uniform session and database handling.
- Use our own error handler for PHP errors and warnings in the web interface.
- Added log, notification and parameter counters to the 'object' table.
  Counting these things at the time a user interface needs them is
  too slow. Other programs, like gcm_daemon en gcm_input should prepare
  these counters for quick retrieval.
- Show the total number of Log entries, parameters and notifications
  on the object page of the web interface.
- Added new tables to the database : log_adv_daemon and log_adv_daemon_email.
- Made gcm_daemon a 'proper' executable.
- Paths of included files in PHP scripts are more flexible.
- Gcm_daemon recognizes important start and stop events of various daemons
- Gcm_daemon checks for exiting daemons in the log entries.
- Directory structure of PHP scripts reorganized. PHP scripts that are included in
  both gcm_daemon and the web interface are now in the directory src/phpclasses.


Feb 05, 2003 - Release 0.0.4
====================================

- Added style parameters for diff-like table views and textarea elements.
- Display and handle notifications in the web interface.
- Display the difference of all package class parameters for two objects
  in the web interface.
- New PHP class added: 'page'. This is a (sort-of abstract) base class
  for all Gnucomo web interface pages. It hanldes the session, login,
  opening the database and the head and tail of the web page.
- gcm_input creates notifications when a new package is discovered
  in a 'rpm -qa' list or when the version of a package is changed.
- Bug fix: action.statuscode was in upper case.
- Added new tables to the database: parameter_class and parameter_notification.
- New C++ function: gnucomo_database::new_notification(). retruns the id number
  of the newly created notification record.
- Added new records to the 'type_of_issue' table.
- Changed semantics of actionid 9 in the 'action' table.
- Bugfix in gcm_input: The hostname in a system log may contain
  digits as well as letters.
- Added new fields to the 'type_of_issue' table.

Dec 06, 2002 - Release 0.0.3
====================================

- Directory structure is reorganized and prepared for GNU configure.
- gcm_input sets the value of log.processed to FALSE when inserting a
   new log entry into the database
- When a syslog entry arrives from last year, gcm_input subtracts one from the
   year of arrival to create the year of the log entry.
- Gcm_input reads output from "rpm -qa" and enters packages in the parameter table.
- Changes to log and log_adv tables described in the manifest.
- Added description of gcm_daemon in the design.
- Added new program (in PHP): gcm_daemon.
- Added a check on the database version in gcm_damon.
- Database-login in gcm_damon can be done using TCP/IP or UNIX-sockets.
- Gcm_daemon processes iptables-records including ICMP support.
- Returning values with ICMP (bounced) are seperate and linked to the same logfile.
- Which each new version records that haven't been recognized are processed again
  through gcm_daemon.
- Added program-specific database user and password in config file.
- Added new arguments to gnucomo_config::Database(): user and password.
  If empty, default values are taken from the config file.
- Implemented a slightly functional web interface that shows some
  output from the database.