© Anton Voronin (anton@urc.ac.ru), 2000-2001.
Contents:
TAS is designed to gather and process the traffic statistics from
PC or Cisco routers (actually, with slight modifications - from any
traffic accounting capable devices) - on IP level and from specific applications
on application level.
The application level is needed because some "intermediate" services
(like http-proxy servers, news servers or mail relays) "hide" actual
user's traffic from IP level. For example, a client requests a large file
from the abroad via your http proxy server. On IP level you can notice
only the traffic between the client and your proxy server. So if you wish
to know all traffic flows initiated by or destinated for your clients
(either for billing, for setting traffic limits or just for estimating your
network usage per each client), you have to account the traffic on application
level as well. TAS can work with the following applications: squid, sendmail
and MailGate.
TAS is written completely in Perl and consists of the following
components:
The first four programs collect accounting data picked up from routers or
specific applications. AcctMax does a specific processing
required for IP data before it is processed by AcctLog. AcctLog
builds arbitrary reports according to the rules specified in its configuration.
AcctJoin summarizes daily databases into current month databases.
Periodic scripts are responsible for running other TAS components,
send the reports to operator and archive them.
Accounting data is stored in Berkeley DB tables. I know, it is not very smart
idea to use db for this task because it leads to consequent search of
the full database when selecting data for building reports. But it is very
simple and convinient to summarize the data in hash tables bacause it
eliminates key duplications (in comparison to storing data in plain text
files).
After you have unpacked the archive, you'll see the Makefile. You don't need
to build or configure anything before install. To install the TAS just type:
Fetches IP accounting data from routers. Runs via cron every several minutes.
The program accepts the following command line switches:
This script requires you run trafdump utility (from trafd package) on
routers from cron for each network interface as often as you run AcctFetch
on your accounting server.
Apart from that, you have to configure passwordless SSH access from your
accounting server to all of your PC routers. It implies your accounting server
is a trusted host, i.e., it does not give shell access to regular users and
is highly protected from network attacks.
On the server add the following to root's crontab:
Obtains FTP/HTTP accounting data by analysing squid's log files that should be
directed to its standard input.
Runs once per day from periodic scripts. Squid's access.log has to
be rotated exactly at 00:00. To achieve this you can add the following to
root's crontab on the machine where your squid runs:
Gets SMTP accounting data from sendmail via syslog. The following lines should
be added to /etc/syslog.conf:
Gets FTP/HTTP/NNTP accounting data from MailGate via syslog. The following lines should
be added to /etc/syslog.conf:
It is recommended to gather IP accounting data from all the routers of your
backbone network to account all the data flows in it. However in this case
you'll get duplicated data for flows that go through more than one router.
AcctMax fixes this by keeping data for each IP flow only for router that
has maximum value (in kbytes) for that flow.
It should be run from daily periodic script before processing the data.
Joins an accounting database (source table) with another one (destination table)
adding values to ones of the corresponding records of the destination table.
-v command line switch makes it to print out progress information
(the number of records processed).
The main and most complicated part of the TAS. It processes the given database
of specific type of traffic and creates reports consisting of
tables whoes captions, columns, rows, selection criterias and computation rules
are specified in its configuration file (by default, /usr/local/etc/Scripts/AcctLog.conf).
Hosts for which AcctLog builds a table are usually aggregated into groups.
A group may be any level domain, any size network or an individual
host. Also groups of group lists may be defined. See the
Configuration chapter for details. The host groups
make the leftmost column (row headers) of the table.
If needed, addresses of hosts that make a separate groups may be resolved into
names.
To describe each table of the report you can operate with host categories
and host category expressions, traffic direction, traffic measurement units,
and traffic tags. See the Configuration chapter for details.
-f command line switch lets user to specify an alternative configuration file.
-v command line switch makes it to print out progress information while
processing the database (the number of records processed).
All tables for each type of traffic are computed in a single pass, which makes
AcctLog robust and efficient.
Calls AcctSquid (to obtain accounting data from yesterday's squid logs),
AcctSendmail and AcctMailgate (to make sure they have been called before
processing and have rotated their current day databases even if there was no
log record since 00:00), AcctMax (to eliminate data duplications for IP traffic),
AcctLog (to process accounting data), and AcctJoin (to summarize
yesterday's data into the current month databases); archives AcctLog's
reports into /usr/local/www/data/acct and sends a copy to admins.
As this is a shell script it can be easily modified on admin's taste.
If your OS doesn't have periodic system, just call this script from cron
every day at about 2:00.
Calls AcctLog to process accounting data for the past month, archives its
reports into /usr/local/www/data/acct, and sends a copy to admins. Also
archives the past month's database, naming it so that it reflectts the year
and the month number and compresses it. Then removes archives older that one
year.
As this is a shell script it can be easily modified on admin's taste.
If your OS doesn't have periodic system, just call this script from cron
every 1st day of a month at about 5:00.
TAS uses three configuration files - /usr/local/etc/tas/tas.conf for
AcctFetch, AcctSquid, AcctSendmail and AcctMailgate
programs, /usr/local/etc/tas/AcctLog.conf for AcctLog program
and /usr/local/etc/tas/accounting.conf for periodic scripts.
tas.conf has a single parameter $prefix that defines a directory
where the accounting databases reside.
AcctLog.conf has three complex parameters:
@local_nets,
@lists,
and %tables.
Format:
Format:
List entries may be hostnames, domains, IP-addresses, IP-nets or other
list names (be careful not to create loops when nesting lists).
By default, if a list entry is a domain, a network or a nested list, all
hosts that satisfy it are aggregated into this domain or this network
or this list name when the list is used to describe the table's category
(i.e., its leftmost column).
If a domain is started with '?', hosts are aggregated into it's subdomains.
If a domain is started with '*', hosts are not aggregated.
If an IP-network is started with '*', hosts are not aggregated.
If a nested list name is started with '*', hosts it is then split according to
their own configuration.
Format:
Table caption is any text that is printed above the table in a report.
Category expression is an expression on the specified hosts categories
(see below) with + and - operations. Each subsequent argument of the
expression is an addition to the previous arguments of the same sign and
the exception from the previous arguments with the different sign, e.g.:
Category expressions are used to describe tables and columns.
All the hosts that are covered by the table's expression, make up the
table's Group column (the column of row headers). They may be or may
not be aggregated into list names, domains or IP nets - this depends
on the configuration of group lists that are used as the expression's
arguments.
Column's expression limits the traffic sources and/or destinations (depending
on the traffic direction specified for the column) for which to count
the traffic in each row of the table.
A hosts category may be either an IP address, a domain name, a list name,
the word total or the word each. Example of a category expression:
sort_column is a number optionally prefixed by + or - sign. It is a number
of table column (conting the Group column which contains headers of the rows)
by which to sort the table rows. If the number is positive, rows are sorted
ascending or alphabetically; if it is negative, rows are sorted descending
or backward-alphabetically.
Resolve flag may be either true or false. It specifies whether or not
to resolve host addresses in te Group column, if they are not aggregated to
any list or domain and if they belong to the nets you specified in
the @local_nets list.
Column caption is any text to put into the upper (caption) cell of the column.
Traffic direction should be either to, from or both keyword. It
specifies traffic of which direction to count in the column (to means
from hosts covered by the column's category expression to hosts in the Group
column for each row, i.e., to hosts covered by the table's category
expression, from means the reverse direction, and "both" summarizes the
traffic in both direction).
Measurement units should be either kbytes or items keyword. In the first
case the amount of traffic is counted, while in the latter case the traffic
items are counted. Items are the packets (for IP traffic), email
messages (for SMTP traffic from sendmail) or requests (for FTP, HTTP or NNTP
traffic obtained from squid or MailGate).
Tag list is a comma-separated list of tags by which to select accounting data.
For IP and sendmail statistics, tags are the names of routers or mail relays
from which the data was obtained, for squid statistics they identify request
status (DENIED, MISS, HIT, OTHER), and for MailGate statistics they identify
access protocol (FTP, HTTP or NNTP). The special tag * means any tag, and
is appropriate for most cases.
File accounting.conf also has three scalar parameters. All they should
be explicitly defined.
Whom to mail the report. Example:
recipient=root
Where to put the reports for archive. Example:
Where the accounting databases reside. Example:
For hosts that are covered by @local_nets list use domains to group
hosts by specific domains (for example, by client domains), but also
use network addresses to group unresolvable hosts into them.
The most time consuming operation of the TAS is report building. To make
it more efficient, the following measures have been taken:
I doubt that there's any way to increase processing speed even more
(at least in current realization, when a full consequent database search is
used for selection).
In the future it is planned to get rid of DNS resolution of addresses into
names and grouping by names when building a report. Instead AcctLog
should connect to an MySQL database
that keeps all the information about clients, find out who owns the given
address, and so be able to aggregate hosts by clients in the report tables
rather than by ip nets or domain names. Of course, DNS resolution
and grouping will be kept as an option.
Also the results of traffic computation for each client have to be
automatically put into the client database, not only into the report.
Accounting data itself also needs to be stored in a real database, like MySQL,
rather than in db tables because it would let to eliminate a full sequential
retrieval of all records when building a report.
For domain names it would be possible if names were stored in reverse form
(upper-level domain to the left).
For IP addresses it would be possible to select by the first N octets of an
address (where N * 8 is less than or equal to the length of mask that
identifies the subnet to which the selected addresses should belong) and only
then to apply the mask and compare with the subnet address.
Even simpler it might be achieved using some other database system that supports
indexing by arbitrary functions of the fields.
Now is the same as 1.1
The first release.
Introduction
Model of work
Installation
make install
By default all components are installed under /usr/local. If you want to
use any other prefix (for example, /usr/local/tas), then type:
make PREFIX=/usr/local/tas install
After the files are copied you need to do some installation steps manually.
See the next chapter for each TAS component.
The TAS components
You can modify it according to your taste and conditions. In particular, don't
forget to replace traflog's -o argument with the format name you have
specified in your traflog.format file.
ip accounting-threshold 32768
ip rcmd remote-host root X.X.X.X root enable
where X.X.X.X is your accounting server's ip.
And for each interface:
ip accounting output-packets
from to packets bytes
i.e.,
212.192.192.138 205.11.187.54 51 32411
On each call it has to show the data since the previous call.
It may be based on trafd package.
For example (by Alexey Zelenin):
#! /usr/bin/perl
system( '/usr/local/bin/trafsave ed0' );
system( '/usr/local/bin/trafsave ed2' );
system( '/usr/local/bin/trafsave ed3' );
system( '/usr/local/bin/trafsave ed4' );
push ( @ARGV, '/usr/local/bin/traflog -i ed0 -n -o zelya |' );
push ( @ARGV, '/usr/local/bin/traflog -i ed2 -n -o zelya |' );
push ( @ARGV, '/usr/local/bin/traflog -i ed3 -n -o zelya |' );
push ( @ARGV, '/usr/local/bin/traflog -i ed4 -n -o zelya |' );
while (<>) {
chop;
next if split(' ', $_, 5) < 4;
$key = "$_[0] $_[1]";
($num1, $num2) = split( / /, $count{$key} );
$num1 += $_[2];
$num2 += $_[3];
$count{$key} = "$num1 $num2";
}
while ( ($key, $value) = each %count ) {
( $from, $to ) = split( ' ', $key );
($num1, $num2) = split( / /, $value );
printf(" %s %s %d %d %s\n", $from, $to, $num1, $num2);
}
*/5 * * * * /usr/local/sbin/AcctFetch -c cisco1,cisco2,cisco3 -p pc1,pc2,pc3
0 0 * * * /usr/local/sbin/squid -k rotate && sleep 30 && gzip -f /var/log/squid/*.log.0 2>/dev/null
If you have caching http/ftp proxy server running on a different machine,
then you need to make it's logs available to accounting server via NFS.
!sendmail
*.* |/usr/local/sbin/AcctSendmail
If you have mail relay(s) on different machine(s), then
you have to pass their logs to accounting server's syslog.
!QueryServer
user.* |/usr/local/sbin/AcctMailgate
If you have MailGate runnung on a different machine, then
you have to pass its logs to accounting server's syslog.
Configuration
@local_nets = ( "net1", "net2", ... );
where each net should be specified as net_address/mask_len notation.
%lists = (
"listname1" => [ "entry1", "entry2", ... ],
"listname2" => [ "entry1", "entry2", ... ],
...
);
List names may contain only US ASCII letters, underscores and spaces.
Words each and total are reserved and cannot be used as the list names.
%tables = (
"traffic type 1" => [ # set of tables
[ # table description
"table caption",
"category expression",
sort_column,
"resolve flag"
[ table's column
"column caption",
"category expression",
"traffic direction",
"measurement units",
"tag1,tag2,..."
],
[
...
],
...
],
...
],
"traffic type 2" => [
...
],
...
);
Traffic type is tag identifying the set of tables. You will specify it in
AcctLog's command line. Usually you need to have a separate set of tables
for each traffic type, so you may name the tags accordingly - "ip", "squid",
"sendmail" and "mailgate".
"A-B-C+D"
This expression covers all the hosts that belong to category A
but not the hosts that belong to categories B and C except the hosts
belonging to category D (all category D hosts are also covered by the
expression even if they belong to A or B category).
"total-listname1+listname2-10.20.30.40-host.domain.com"
total and each imply all possible addresses, but when used in a table's
category expression, total aggregates all of them in a single group
"total", while each makes each host to be a separate group (i.e., a
row header).
storage=/usr/local/www/data/acct
prefix=/var/account
Configuration tips
Performance
On my PII 266MHz server to build a report consisting of 6 tables each of
8 columns from a database of more than 100,000 records (which is a daily
amount of IP traffic statistics I currently have), AcctLog spends
23 minutes. To process monthly database it takes 2-3 hours.
Planned enhancements
History of changes
Another "Use of unitialized value" warning has been fixed in AcctLog.
Many typos fixed in this documentation page.
Fixed a very stupid misprint in AcctJoin, made during the changes
of Jan 5, 2001, that caused to store number of packets instead of
number of bytes when adding data to the month database.
Fixed a minor bug in AcctLog that caused "Use of uninitialized value"
warning in some situations.
Fixed typo in AcctFetch.
SEEK_SET definition is now correctly taken from IO::Seekable module.
Removed unused variables from AcctSendmail and AcctMailgate.
Databases are not locked now - a stampfile is now locked instead.
AcctFetch now opens ip-database only after the data was fetched from
a router and then closes it before fetching data from another router.
Fix for bug with renaming current-day databases that were not processed
in time for some reason.
Accounting.conf now lets a user to specify which types of traffic to
process (directives process_ip, process_squid, process_sendmail,
process_mailgate), so there's no need any more to modify periodic
scripts if statistics for some types of traffic is not gathered.
Russian text accidentally remained in periodic scripts, was translated
into English.
Fix for squid's accounting database rotation failure in monthly
periodic script.
Fix for uninitialized value warning in AcctLog.
Configuration directory is now /usr/local/etc/tas instead of
/usr/local/etc/Scripts.
New configuration file tas.conf has been added, it lets to specify
a directory where accounting databases should reside.
Two configuration parameters added to accounting.conf: "compress_rotated"
to specify whether or not to compress rotated month databases and "keep"
to specify number of months to keep the data.
File permissions fixed for executable scripts so that they could be
run by any user. Thanks to Andreas Klemm (andreas@klemm.gtn.com).
The data fetched from a router now is accumulated in memory and
committed to a database only after the fetch is completed. Commitment
during fetching sometimes led to a db table damage (I don't know why).
Download