In this document we give an introduction and overview of Ourmon and it's individual filters.
Ourmon is an open-source tool for network monitoring and anomaly detection. It runs on FreeBSD, Linux, and Solaris. Goals include:
The front-end packet capture engine has three forms of filters: 1. hardwired (programmed in C), 2. user-programmable using the BSD Berkeley Packet Filter (BPF), and 3. top N style filters that produce sorted lists of largest flows, interesting IP hosts, or top talker TCP/UDP ports. User-programmed BPF filters or BPF filter sets may be added to the system by the user, and allow considerable customization of the system. For example, a user might choose to create one or more graphs that watch local subnets, local hosts, or local ports on local hosts. Hardwired, and BPF filters use the RRDTOOL package for the display and baselining of data. Top N filters use a supplied histogram-drawing program and log one week's worth of data (typically off-line in the ourmon/logs directory) for later analysis. Some summarization-style logs are available from the web pages.
The back-end system displays network information on the web using either:
Recently ourmon has been modified to enhance its abilities to detect network anomalies associated with various forms of attacks including TCP syn scanning, UDP scanning, DOS attacks, and distributed zombie attacks. A number of features have been added, some of which use the BPF to give us an overall network view of TCP and ICMP control data, some of which use new top N filters to show information about particular host IP systems engaged in scanning, and some of which provide RRDTOOL graphs for carefully chosen and proven useful metadata. We will go over these particular filters below in a section of their own entitled anomaly detection filters.
The ourmon architectural flow diagram is intended to give a rough
analysis of data flow through the ourmon system.
For network setup, the ourmon "probe" box is assumed to be directly
connected to an Ethernet switch. The switch must have port mirroring
turned on so that promiscuous mode filtering on the ourmon front-end "probe" can see all
desired network packets passing through the Ethernet switch. Network packets
are passed to the NIC card of the front-end probe, and in FreeBSD are stored
in the BPF buffer. (In Linux, the details are different, but there is still
a kernel buffer associated with packet flow to the front-end application).
Ourmon architecturally consists of two parts, known as the front-end probe, written in C, and the back-end graphics engine, mostly written in Perl. The front-end is a form of network analyzer that reads network packets from an Ethernet interface out of a kernel buffer, and runs each packet through a set of filters. (It should be noted that the front-end program is also called ourmon). For each filter, byte counts or packet counts are periodically stored in an output report file, and that information is passed to the back-end. The back-end then takes the output of the front-end and creates web-based graphics. It may optionally log top N flows to ourmon logs (that are similar to syslog in style) and produce some daily "top of the top" reports on the logs.
The front-end uses the Berkeley Packet Filter and pcap library to fetch packets from a promiscuous mode Ethernet device. The filters used are specified in an input configuration file, called ourmon.conf . One may add or remove filters as desired. 68 bytes maximum per packet (as with tcpdump) are captured. Thus only the protocol parts of packets are actually captured via the BPF. Ourmon at this time does not examine L7 data. This allows (L2) Ethernet and (L3) IP addresses, and L4 ports to be examined. This is an important trade-off. As a consequence, because it does not look at layer 7 data, ourmon can deal with higher speed flows when compared to an IDS signature-based tool like Snort. On the other hand, one can only use it to analyze up through layer 4, and no layer 7 analysis is currently possible. From a security point of view, it could easily make sense to run both ourmon and Snort side by side (but on different hosts please). This is because Snort might miss a new attack, like a high-speed DOS attack. Ourmon would show that something abnormal happened simply because it does NOT understand the data. For example, the pkts graph could show an unusual number of counted packets and/or drops. On the other hand, ourmon is certainly not going to see a known attack signature in the data part of a UDP packet.
Internally the front-end program looks at the Ethernet header, IP source and destination addresses, IP next protocol, and TCP and UDP ports, or in the case of ICMP, major and minor ICMP codes. Each configured-in filter is executed per packet. Thus if there are 5 filters in use, they will be executed in order, one after the other on each packet. In general, the average filter will count bytes or packets and represents this with integer counters. Top N filters keep hash lists of flows or IP addresses associated with counters. At the end of the sample period (every thirty seconds) the output is written out in a very simple format, called the mon.lite file, counters are reinitialized to zero and dynamically allocated lists are also freed. See mon.lite for an example. This file is then processed by the back-end to create various kinds of graphics available from a web server. The "mon.lite" file may be viewed as a summarization of a great deal of packet information into a small intermediate and condensed form.
The back-end is Perl-based. The back-end consists of several programs. The most important backend program is called: omupdate.pl. It creates, and updates graphs with the help of the RRDTOOL library. For top N display, it also employs a tool called drawtopn to draw histograms. There are some additional sorting and logging functions that will not be covered here.
Omupdate.pl both creates and updates per-filter RRD (RRDTOOL) databases, depending on which set of filters you wish to use. Omupdate.pl must be driven by a crontab entry (and is typically called twice a minute). It takes the current copy of the mon.lite file, processes it a bit, and stores information in various per-filter rrds (RRDTOOL log databases), as well as creates the current set of RRD-based graphics. It is possible, and probably a good idea, to run the front-end and back-end on two separate hosts. How "mon.lite" is copied from the front-end to the back-end is an "out of band" problem, and is not solved by the ourmon system. But it is an easy thing to do and ssh or the wget application can be used to solve the problem. One very nice side effect of using RRDTOOL is that all the RRD filters, including BPF-based filters, produce a year's worth of baseline data in graphical format. Thus per-filter, one gets current (daily), weekly, monthly, and yearly pictures.
In the next two sections, we will discuss most of the individual filters in detail.
Please note that ourmon as a system supports three kinds of filters:
1. hardwired filters, that are done in C code. These have
names and specific functions. There are not many of them.
2. user-programmable BPF filters. Ourmon supports arbitrary user-space
(as opposed to kernel-space) Berkeley Packet Filter (BPF)-based filters.
One can have 1-6 BPF-filters per RRDTOOL graph (more than 6 produces a cluttered
graph, and in fact 6 is probably too many). These filters use
the same language specification mechanism used by tcpdump.
It is currently possible to have up to six BPF expressions per RRDTOOL style graph.
At the time of writing the one deployed ourmon front-end system has 60 BPF expressions in it total
and in general does not lose packets.
3. top N filter mechanisms that produce various kinds of top N information
including a traditional flow monitors. Other top N lists exist focused
on anomaly detection. These lists typically show information associated with
individual IP source addresses.
In the next section we begin a detailed discussion of the various filters supplied by ourmon. Note that filters have names; for example, the filter that displays different IP level protocols is called "fixed_ipproto" . Filter names are important because they tie the inputs and outputs of the ourmon system together. Filters are named in the ourmon.conf file, and output appropriate to that name appears in the mon.lite file, and again appears in rrdtool libraries, log files, or png images created by the back end. It should also be pointed out that data associated with a filter presented by the back end, is usually interpreted as packets or bytes, and may therefore be presented as bits/sec, or pkts/sec. In some cases, a data item may be presented as data/period. Period means the ourmon sample period of 30 seconds.
In this section we provide a brief overview of the web page layout. The layout is hierarchical with two levels consisting of a main page (index.html) and any number of secondary detail pages. The primary function of the main page is to provide current status -- the goal is to tell you what is going on now . Secondary pages which are accessed as links from the main page show more runtime statistical details. In general the graphics-engine back-end of ourmon makes reports and graphics. It does not (yet) dynamically update any html pages. The design and layout of the html pages can (and should at least when using the user-customizable BPFs) be altered according to your own needs.
The main page includes RRDTOOL current graphs for the current day in terms of hours, the "top-most" top N histogram graphs, and various reports including reports for the last sample period and some hourly/daily summarizations. For example, this would include the RRDTOOL basic pkts graph at the top layer. From the main page, find basic network information and look right underneath it for the probe #1 pkts/drops graph. Essentially the graph has a label and both graph and label are links as well. In general the label (when a hypertext link) will take you to the info.html page in order to provide more details on the features of the graph. The graph will take you in turn to a secondary details page that provides more runtime statistical details. Ourmon's notion of help consists of links that jump back and forth between the main index.html page, the info.html help page, and the secondary detailed statistics pages referenced by index.html. (This system works best with mozilla/firefox).
Top_n graphs are often presented as histograms. On the main page find the ASCII heading top talker (top_n) flows based on IP source . Underneath it is the histogram graph that shows the top 10 IP-based flows. It is a link too and if you click on it, you go the to related secondary pages that should show more pics with 10 bars per picture. The first graph shows the top 10 flows. The second graph shows the next 10 flows, etc.
The index.html page is structurally broken up into several sections including:
Here we only wish to discuss the quick jump directory as the other sections should prove self-explanatory. The quick jump directory consists of two tables of links into various parts of ourmon. The first table consists of links that are either security related or have bearing on whether or not ourmon is actually running. The second table entitled main page sections simply divides the main page up into various sub-sections and can be used to directly jump to those sub-sections. For example, in the first table we have the TCP-based "worm" portreport.txt report that gives information about noisy TCP hosts. One also finds the similar UDP-based udpreport.txt that gives information about noisy UDP hosts. Both of these links are very fundamental to security and may show current attacks. The following links will take you to the relevent "info" section below for various important security links in the two tables:
In general one uses a web browser to browse ourmon. There are a few things to note. The top-page comes in two flavors, index.html and indexstatic.html. index.html self-updates every 30 seconds (which may be annoying). indexstatic.html should have the same content but does NOT update every thirty seconds. All the second-level web pages do update every thirty seconds, but ASCII reports obviousally are not written in html and do not update themselves. You must use your web browser reload button to update an ASCII report. Just remember that the sample period is thirty seconds long, and an occasional refresh if looking at an ASCII page is helpful.
Now we will discuss the filters in detail.
For example, you can always choose to lose a filter. The top N flow filter without a doubt,
is likely the biggest CPU consumer. It may also be better to have ourmon running with
only one Ethernet device as opposed to two (even though two is possible).
This may be the case due to BPF buffering schemes. Note that the pkts
(caught and dropped) counters are zeroed out (in SNMP terms, pkts is a GAUGE),
at the mon.lite write period time. All of the individual ourmon
filters are currently
zeroed out at mon.lite output time. Thus the counters start over
from zero for the next round (and at the moment they are more or
less all GAUGEs anyway).
Typical "mon.lite" output is as follows:
The mon.lite output looks like this:
The mon.lite output looks like this:
The mon.lite output looks like this:
In the ourmon.conf file, BPF filters default to displaying bits per second
(bpf-bytes is used above to force this mode). The config file
has two modes which can be toggled on and off between filter sets (but not
inside a filter set) using the following two commands:
The first filter above is supplied with the default ourmon config file
and serves to graph pkts/sec for basic IP protocols, including any IP protocol itself,
TCP, UDP, ICMP, and "other" protocols not of an IP nature.
We get one RRDTOOL graph with five lines in it.
Any use of "bpf" as the first
config token starts a new BPF graph.
The name of the graph is "protopkts". "bpf" and "bpf-next" tags are used
to introduce (label, BPF expression) pairs. For example,
The "ports" filter set also gives us one RRDTOOL graph with 5 lines in it.
The name of the graph (BPF filter set) is "ports".
The BPF tag called "bpf" indicates to ourmon that we are starting
a new BPF filter set graph. "ports" is the name and label of the entire graph
and is used in the backend for creating unique filenames both in the web and
rrddata directories. This first bpf line also includes the first line label "ssh" and a
BPF filter specification associated with that line label. The BPF expression
"tcp port 22" is designed to capture all secure-shell traffic on port 22.
The subsequent "bpf-next" lines add additional lines
to the graph for p2p traffic (kazaa, bigtorrent, edonkey, gnutella, and the like --
by definition p2p apps can use whatever port they want, so this isn't perfect,
it's just an informed guess), the web, ftp, and email.
Each bpf-next line has a line label and filter specification.
A BPF graph is either terminated by another filter or by a BPF-noxtra label
which tells ourmon to not collect and graph bytes that fail to
match any of the BPF filter specifications in this graph. One
may choose to have remaining bytes shown in the final "xtra" graph,
or choose to ignore them (which means xtra is *still* shown in the graphs but has a zero
value as nothing is ever stored in its counter).
The supplied ourmon.conf file has many examples of BPF filters, and you
should make your own. For example, you could graph subnet traffic for
3 subnets, 10.0.1.0/16, 10.0.2.0/16, and 10.0.3.0/16 as follows:
As a runtime optimization that only involves the front-end probe,
we have developed a sub-system we all CBPF.
The goal of CBPF is to allow the user to increase the number
of BPF filter-set expressions while at the same decreasing the odds
that packets will be lost. CBPF allows us to
hand code C functions to replace some commonly used BPF expressions
in the front-end. Unlike the BPF expression mechanism, CBPF is
not general. There is only a limited set of expressions
that can be used. The syntax in the config file is also slightly
different and in some cases cumbersome. Therefore BPF expressions will exist
that cannot be replaced by the existing set of CBPF functions.
However the runtime advantage of a CBPF function over its equivalent
BPF expression is significant. Typically CBPF is at least ten times
faster. One can use CBPF to replace common subnet expressions (net IP)
and also in our supplied BPF filter-sets for ports and P2P applications.
(See the supplied ourmon.conf file). In the configuration file,
CBPF tags where they exist may be dropped into the user BPF filter-set
at any point, replacing an equivalent BPF expression. CBPF config tags
can be used to replace either "bpf" or "bpf-next" as follows:
Commentary: the BPF lacks a range operator. BPF expressions
can consist of logical ORs but this results (particular in
an attempt to catch the bittorrent default port range, which
may actually extend to port 6899) in a very inefficient
set of BPF instructions. Keep in mind that the BPF is interpreted
at runtime, and CBPF is compiled C code. In general, BPF
expressions have linear cost. Longer expressions cost more
at runtime. Note that this CBPF operator (as it true in
similar cases) can be viewed as a set of logical ORs in terms
of its operands.
Commentary: Mixed sets of UDP and TCP ports can be OR'ed together.
Ten pairs of (u port) or (t port) combined are allowed maximum.
The above is the functional equivalent of the following:
Commentary: The CBPF ping tag takes no expression string.
The front-end ourmon program performs a hash function, and creates
a dynamic hash list of flows at runtime. At report time,
it sorts the list with quicksort, and writes out the
top N total IP "flows" (src to destination), top N TCP, top N UDP, and top N ICMP flows.
Note that the numbers are simply 30 second
byte counts represented as bits/sec.
The drawtopn program is used to create png histograms
for the results. In addition, logging is done for the four flow
types in separate logging files (this is true for all top N filters).
(These syslog-style logs are available for analysis by the administrator,
but they are not available on the web).
In addition "top of top" reports
are generated for the topn_ip, topn_udp, and topn_icmp filters,
on an hourly or daily basis for a week. The sorting functionality
provided allows one to see the top flows over a period of time.
Top of top reports are also created for topn_ip, topn_udp, and topn_icmp
that report on the biggest IP source and destination addresses flow generators for these
three flow types.
The mon.lite output is as follows:
There are a couple of important design ideas here driving this work.
One is simply the observation that it can be useful to look at network
control data to observe a baseline and then detect anomalies in that baseline.
There is in general much LESS network control data than data itself (one hopes).
There are less ICMP error messages, than UDP data (usually). There
are less TCP control packets in terms of SYNS/FINS/RESETS compared to data itself.
As two more concrete examples of this notion, one would expect that
TCP SYNS might somehow be related graphically to TCP FINS. Large
numbers of SYNS that have nothing to do with the FIN count thus
indicate that something abnormal is happening. One can also look
at ICMP error data, and see that for example, large numbers of ICMP
host unreachables are being returned because of an active UDP
scanning system. Second-order data may also be of interest from
the viewpoint of network control. For example, instead of focusing
on individual top network flows, we can focus on the overall count
of network flows. Perhaps one has on the order of 1000 ICMP flows per
second. 100000 ICMP flows per second would then indicate that
something was very wrong. (This actually happened due to an outbreak
of the Welchia/Nachi worm on one campus in Fall 2003).
We also are in the process of a long-term experiment using
various kinds of weights to detect scanners. There are three
new top N list mechanisms that in some cases use weights
to try and detect scanning systems, and in some cases simply
sort on integer counts. We have a top N syn list that currently
does both of these things. It sorts on the top N ip src sending
TCP syn packets, and at the same time, produces various metrics
that for example tell us the percentage of TCP control packets compared to
the overall percentage of TCP packets sent and received by
that host during the sample period. We call this weight the
work weight . The work weight appears
to be very useful as it allows us to rule out most P2P applications
that often also send large numbers of TCP SYNS during the sample period,
but in fact, generate less TCP SYNS + FINS + RESETS / total pkt count
than most SYN scanning worms. The top N syn mechanism
also produces several other outputs including the tworm graph ,
which represents a total count of anomalous ip sources in terms of SYN counts,
and thus allows one to see distributed attacks occuring in parallel,
tcpworm.txt report which can be viewed via several different outputs most
notably via the so-called
port signature report
.
Both of these topn syn list outputs are produced as a side effect
of a weight metric where we only capture ip sources from the syn tuple
when they satisfy the following worm metric :
In other new top N filters, we also
count the total number of ICMP error packets sent back to a host,
and a weighted measure of UDP errors as well.
We also have a conventional IP scan catcher that sorts on the top N IP
source addresses with the most unique destination IP addresses. This scan
filter also produces graphs that show the mapping of an IP source
to N L4 TCP or UDP destination ports.
This work is "not done" and is only preliminary. We hope to improve it over time.
Ourmon if so configured stores a TCP syn tuple with various counters
that look at the two-way nature of TCP traffic from an IP source address.
This syn tuple includes information about SYNS sent by an ip source, the total number
of SYNS+ACK packets sent by an ip source, FINS returned to that ip source,
RESETS returned, ICMP errors returned, total pkts sent from the IP source, total
pkts returned to the IP source, counts of unique IP destinations, counts of unique
L4 TCP ports, and a novel destination port sampling scheme.
The port sampling scheme stores the first N TCP destination ports seen associated
with SYNs and stores packet counts for SYN packets only sent to those ports.
This is a sampling scheme as only the first N ports seen are stored. Not all destination
ports are stored. The SYN+ACK count packets are only those packets from a given ip source that are sent
as the second packet of the TCP 3-way handshake. In TCP state machine terms, these
packets represent a service on a host. Thus it is fair to say that a percent score
here of 100% means a host is a server.
As one example, the topn_syn list shows individual IP source addresses sending the most TCP syns
in a period.
The following outputs are currently produced by the topn_syn list tuple:
Note that three different top N mechanism currently rely on each other. The IRC
summarization mechanism uses the topn_syn work weight. The topn_syn list
itself also uses data from the topn_scan IP and port lists. All three of
this lists should be turned on together.
Two major weight functions are used in various places above. These are called
the work weight and the worm weight respectively.
The work weight approximates the percentage of TCP control packets captured divided
by the total number of packets sent both ways. The work weight for each ip source is roughly:
The front-end collects a topn syn list which may of course have 1000s of "syn flows" in it.
It sorts the syn list twice, once for the topn_syn report, and once for a second
output file now by the ourmon front, which is the tcpworm.txt file in raw form.
(Raw form means it is not yet processed for human readability which is done by the back-end).
The information given in the tcpworm.txt file consists of that set of ip source addresses
showing up with more syns than fins returned to them. The metric used there is the
worm weight . At this point the worm weight is computed by saying
Let us first look at the setup in the ourmon.conf file and then we will consider the
results in more detail:
The first command
topn_syn turns on the basic topn_syn graph.
The argument to topn_syn (60) specifies how many hosts
you want to capture in terms of pictures. This argument should have the value 10, 20, 30.. to 100
maximum. 60 will produce six histogram graphs with the top 60 "syn" producing systems shown in two
graphs with 10 ip source addresses per graph.
topn_syn_wormfile turns on the tworm RRDTOOL graph output and also the tcpworm.txt report.
In other words, output based on the worm metric is produced. There are actually
several front-end outputs including the tworm graph, which graphs integer counts showing
the number of external ip sources and internal ip sources appearing in the tcpworm.txt file,
and the tcpworm.txt file itself, which like the mon.lite file is passed to the back-end
for further processing. Note that we assume you will use this function, and it is probably not wise to turn
it off.
The second argument to topn_syn_wormfile is a directory path which specifies
where to place the tcpworm.txt report in the front-end file system.
This argument may be overridden with the -D option to the front-end ourmon probe. -D
causes all ourmon probe output files to be placed in a specified directory, typically
/home/mrourmon/tmp. If two systems are used, one as the front-end probe, and the
other as the back-end graphics engine,
the administrator must arrange that the tcpworm.txt file will be copied to the back-end,
and further processed by shellscripts produced in the configuration process. Tcpworm.txt
is the rough output that is further processed by the back-end and becomes the port signature report,
and other outputs when processed by the back-end.
We will discuss these reports more below. So tcpworm.txt is a rich source of information and at this time there
are a number of back-end reports based on its data.
topn_syn_homeip specifies what you consider to be the "internal" subnet used for "us" in the tworm.html output.
This allows the front-end to decide what is "us" versus "them" in terms of the tworm RRDTOOL graph.
The first argument is a network IP and the second argument is a netmask. If you are using ourmon
on a host-basis and not watching an entire network, you can set this value to be your host IP,
and then use a /32 netmask.
topn_syn_p2pfile causes the creation of the p2pfile.txt probe output file. This file is the subset
of all syn tuples that has an application flag set which more or less means the set of hosts using
well-known P2P applications including Bittorrent, Gnutella, Kazaa, and Limewire/Morpheus. It also
includes hosts using Internet Relay Chat. This file is processed by the back-end to create the p2p port report,
which is viewable on the IRC web page .
Finally tworm_weight can be used to specify a value ranging from 0..100 percent that is used to filter
the number of hosts shown in the tworm RRDTOOL graph. 70 is probably a reasonable number here as in
our experience hosts with a higher value are very likely to be worm/viruses, although worm/viruses can
appear with a lower weight. For example a common attack against one of the SQL-slammer ports (1433)
may actually result in two-way data exchange. This is because SQL servers are returning Layer 7 data that
is basically saying: "the password guess is wrong". As a result, the work metric for an attacking IP source
may be lower if data (albeit negative data) is being returned by the attacked host.
If this switch is not used, the default filter weight is 0%. This means no filtering is done.
There is at this time no way to filter the portreport itself or the tcpworm.txt file as an input.
The topn_syn
graph shows syns, fins, resets, and the work weight metric
for individual IP source addresses sorted by the IP source sending
the most syns in the sample period. The topn_syn back-end graph information per ip source address
might be presented as follows for two ip addresses 10.0.0.1 and 10.0.0.2.
The IP address is the IP source address for the SYNS being sent. The SYN count is given
below the other information and in the first case above is 4779 meaning 4779 syns in the sample period.
f stands for "FIN". r stands for "RESET". These
represent TCP FIN and RESET counts sent back to the IP source address.
t represents the total TCP packets sent to and from the address
in question. The work weight is also given as a percent with one flag, which may be 'w' or 'W'.
As a visual aid, if the percent is greater than 90%, we give it a "W" flag.
If the percent is between 50..90%, we give it
a 'w' flag.
With the work weight, false positives are possible. In general, one
should use tcpdump or some other scanner to capture packets
from a believed suspicious IP host and verify its behavior.
Although false positives are possible,
we have observed the following about the work weight system:
The above weighting system is also used in the port signature report and its companion
tcpworm.txt output file which has more details in it per ip source.
mon.lite output is as follows for topn_syn:
mon.lite output for tworm appears as follows:
The portreport may be regarded as a simple summary of the more verbose tcpworm.txt report.
It is called the TCP "portreport" or "port signature report" because it includes a small sample of TCP destination
ports from the IP source in question. Thus it gives a set of destination ports
which we call a "port signature". This port signature may allow you to see a new virus
at work and often some virus/worms have distinctive "port signatures". For example, see
ports 5554 and 9898 as found below in a small example of real port report output.
This is a signature of the dabber worm.
Here we only discuss the portreport or "port signature" report as the longer version
is for the most part more detailed in terms of SYN tuple information.
The port signature report has three sections. The first section
is the port signature report itself which is is sorted in ascending order in terms of IP source
address. The second section is based on a daily "db" file database that tells you
if the current set of destination ports seen are "new" in this port sample, or "old".
This database is started again every night at midnight.
New simply means so far today we have not seen that particular set of ports before. "old" means
we have seen that particular port set before. For example, note the destination ports
given below (5554, 9898). This set might have appeared in previous portsignature reports
for the day, or it might be new for this particular report. New port signatures are
stored in the event log as well (see below for more information). The final chunk
of information in the port signature report is a simple condensed help reminder
in case you forget what the port signature fields mean or the meaning of the EWORM flags used
as well.
With the main port signature, we sort first by IP source address from small to large.
Thus one can thus easily note attacks that come from the same IP source "locale",
including attacks from the same IP subnet possibly synchronized with some bot mechanism.
Each line represents culled information for one IP source deemed anomalous in terms
of its SYN output. Let's look at some examples:
The ip source address is given, followed respectively by the flags, application flags, work, SA/S,
L3 destination and L4 destination, sampled dst/total dst, and port signature tuple fields.
The flags field attempts to convey whether the "work" in question is two-way or simply
one way from the IP source and provides details about the nature of TCP control data as well.
The total set of flags is as follows with a rough explanation given for each:
Scanners or worms commonly produce WORM or WOM, although if a network administrator chooses too, he or she
might produce more RESETS from local routers and/or hosts, and this could be useful in detection
of scans. The flags field is explained in more detail in the verbose tcpworm.txt output file.
The work weight as before roughly represents the number of TCP control packets divided by all
TCP packets sent to and received from the IP source address in question. 100% means more or less
all control and no data. From experience we suggest that work weights in the range of 60% or more
should be deemed highly suspicious. Although there are rare cases of noisy clients that
for some reason cannot reach their server OR email servers that are trying to reply to spam
(which will always fail). We have observed over many months that in general anything
with a weight over 60% is "anomalous" and in most cases not benign.
Low work weights may mean you have a noisy web server
that is sending large numbers of small packets per connection to a client.
On the other hand, low work weights may be associated with P2P applications, Layer 7 password attacks on web servers
(port 80 as a destination port may indeed be significant, but it could be an attack or
a benign web client use), or irc server bots with a high rate of JOIN message churn.
The latter very well may be a trojan. Thus it is important to note:
A low work is not necessarily a benign indication.
The application flags field (which is also used in the p2p version of the port report and in fact
defines that report) gives various hints in some cases derived from L7 scanning about
the nature of the applications on the host in question. In general the P2P applications
here (and IRC) indicate the start of a peer to peer exchange (not a client/server exchange).
Thus it is possible that a host may appear to have a "worm" or TCP scanner when using
Gnutella and not show a "G" flag simply because it is completely failing to ever
find a real Gnutella peer and start a Gnutella peer to peer exchange.
The hints flags currently include the following:
The SA/S field expresses the total percent (0..100) of SYN+ACK packets typically sent as the
second packet of the TCP 3-way initial handshake divided by the total number of SYN
packets sent from the IP source in question. There are three possible thresholds here.
0 means the system in question is a client. 100 means the system in question is a server.
A number in-between (which is commonly found with P2P systems) shows that the system
in question has both server and client functionality. One should correlate this information
with the port signature information, in particular when the sample space of 10 possible
ports is filled up. A low work weight, 100% for SA/S and 10 out of 10 ports in the port signature
typically indicates a server (which is typically either a web server or a server undergoing a L7
attack returning error messages like "404 not found", or "password failed").
One interesting facet of this field is that occasionally one will see a work weight of
100% and an SA/S value of 100%. This can only mean that the host in question is performing
SYN/ACK scanning.
The L3D/L4D destination field is derived from the ourmon probe scanner module.
The L3D field shows the exact number of unique L3 IP destinations during the sampled period.
Although in practical experience this field is not misleading, it should be noted that there
is no guarantee that the IP destinations are indeed TCP peers (they might be UDP peers). (This
is a bug and will hopefully be fixed in a future release). The L4D field shows the unique
count of L4 TCP ports seen during the sample period. These fields can often be used to
give you an approximation of the behavior of the IP host in question especially if it is a scanner.
The sdst/total field gives some measure of how well the port signature mechanism worked.
The sdst number represents the total number of sampled packets stored as counts in the port signature tuple.
The port signature tuple samples the first 10 destination ports seen coming from the IP source
in question during a sample period. Each tuple consists of a TCP destination port, and the packet count associated
with that destination port. The total field consists of the total number of packets coming
from that IP source. If the dst value is much less than the total and all 10 port tuple
buckets are full, then this is a good indication that the port signature field failed to
catch much in terms of its samples. These two numbers are not expressed as a percent because
the two values can give you some idea of how many packets per sample period are being
sent by the IP source in question. For example if you see that the work weight is 100%,
and that packets are being sent to say 5554, 9898 as above, you can then estimate from the numbers
given above that probably 3 syns per second are being sent.
The port signature field itself results from a possible sample size of 10 TCP destination ports
captured for the IP source. The destination port is the first field in the port 2-tuple.
Note that in order to help spot similar attacks, the ports are sorted in ascending order
from left to right.
We can sometimes tell by the port number itself that an attack is occuring. For example,
in general one should be suspicious of any of the following Microsoft ports in the UDP or TCP
port signature report.
Currently the batch report gives the total number of instances for the IP source in question,
and also presents a small sample (5) of port signatures. The port signatures presented here
are of the form [port_destination, packet count].
topn_icmperror generates two separate top N lists, one for ICMP errors associated
with individual IP source addresses, and an additional list
for UDP-based IP hosts. The former is important for catching
scanners generating ICMP errors (which may or may not be generating ICMP packets
themselves). The latter is focused on UDP-based anomalies
which may include UDP scanners, DOS attacks, or badly-behaved UDP-based applications.
The rationale for these top N lists is the same -- they focus on hosts that
generate large numbers of ICMP errors.
In the mon.lite file, these lists are shown as follows:
In the mon.lite file udperror_list, the tuple means:
(ip source, a computed weight:
UDP pkts sent, UDP pkts returned, ICMP unreachables returned, ICMP Ping packets sent,
L3 IP destination count, L4 UDP destination port count,
and a port count plus port signature tuple). The port count indicates how
many port signatures are in the port tuple, 1 to 10 total. Each tuple
consists of a L4 UDP destination port followed by a packet count.
Port signatures are a sampling technique; that is, we do not try to
capture all ports, only the first N seen. L3 and L4 destination fields
and the port signature fields are thus roughly similar to the same fields
in the TCP portreport.txt report.
Here however the weight is used for sorting. The weight is basically:
In the graphical representation for the udperror_list,
we sort on the computed weight as mentioned before.
The flag system in this case is as follows:
The ASCII UDP port report is created by omupdate.pl and is more verbose
than the histogram-based graphical system. The report presents the per IP source information
in a form similar to the TCP port signature report. Port packet tuples
are of the form [destination port, frequency]. In other words, packet counts
are not given here, but instead are given as a percent of the total packets
captured in the port signature sample. For example, one might see the following:
This example is likely a worm. 138 UDP packets were sent and none were returned. 28 ICMP unreachables were
returned. The only port used is a Microsoft file share port, and we see that all 138 packets
were aimed at port 137. Ports 137 and 1434 (SQL) should be viewed with suspicion.
In real life, the UDP report suggests that scanning and other anomalies are not as common as TCP-based
equivalents, but they do exist. Low-level scans (with low weights) for ports 137 and 1434 are common.
Given the apparent sporadic nature
of UDP attacks (compared to 7x24 TCP attacks), we have three additional mechanism for help in analysis. This includes the
udpweight graph discussed in the next section, the system
system event log and the new trigger system,
which can be setup to automatically capture the top UDP weight based on a pre-configured
threshold. See triggers - automated packet capture for more information
on the latter.
The system event log is automatically configured to capture any UDP anomaly in this list
with a weight greater than 10,000,000. Basically omupdate.pl stores the first line of the ASCII
UDP port signature report in the event log if such an event occurs. The threshold in question
is configured in bin/omupdate.sh as a parameter (-u) passed to omupdate.pl and may be changed if
so desired.
The topn_port_scans filter presents three separate graphs, but in general,
looks at single IP sources
sending packets to many unique L4 TCP and UDP destination ports.
We sort on the maximum L4 destination ports. There are three graphs because
the ip_portscan graph counts both TCP and UDP ports (and does not discriminate between the two),
while the tcp_portscan and udp_portscan graphs only show TCP and UDP destinations respectively.
Thus both topn_scans and topn_port_scans are 1-N in terms of their basic mapping.
The ourmon.conf configuration is setup as follows:
Therefore the graph representation is not any different from the mon.lite outputs.
We simply present an IP source address, with its respective destination
count. Results are sorted in order of unique destinations.
It should be pointed out that information here may be correlated
with information shown in the UDP errors or TCP syn top N mechanisms.
The top N flow mechanism may also be of use for network security
in terms of anomaly detection. There are fundamentally two
different sets of graphs shown here. First of all, it has proven
very useful to use a RRDTOOL graph to display the count of
all four kinds of flows. (The count is the tag that follows
the flow tag in the mon.lite file. It is the count of unique
flows of that type, of which only the top N are provided
as tuples in the mon.lite output).
We call this the "flow_count" graph.
This graph shows small and large attacks and classifies them
as to whether they are TCP, UDP, or ICMP in origin. It shows
the attacks simply because "scanning" means variation in IP destination,
and/or L4 ports (or ICMP major/minor numbers) which are classified as separate flows as a result,
even though they may be flows of one packet only. This graph
has proved to be a particular good indicator of network-wide scanning
attacks, including single instances of worms like the UDP slammer.
The companion graph, "topnstat.html", show the counts
of inserts in the flow hash list, which again are due to separate flows.
Inserts result in malloc(3) calls and this graph also tracks attacks
fairly well. (However long term it may prove to be redundant
to the flow count graph).
In addition, it should go be pointed out that the ICMP top N flow graph
may be useful. In particular, the ICMP flow "top of top" report
has a tendency to reveal hosts that are engaged in long term scanning
simply because they pile up extraordinary amounts of errors. (It may
also show a long term ping of course). These
scanners may be both TCP or UDP-based. In the TCP case, errors
may occur due to administrative prohibited ICMP unreachables,
ICMP redirects, or TTL exceeded errors. In the UDP case, UDP scanners,
may in addition pile up ICMP port unreachable errors over time.
As a result, the top of the top report here is useful for detecting
scanning systems as well. ICMP unreachables, TTL exceeded, and
routing redirects may be treated with suspicion.
Typically a trigger event occurs when a runtime current value
as seen with a graphics-engine graph exceeds some
administratively set ourmon probe configuration-time threshold integer.
The threshold integer is calculated in terms of back-end values,
either in bits per second, or packets per second.
A per trigger type dump directory must be specified (say /usr/dumps)
and created before the ourmon probe is started.
Each runtime trigger when enabled places a unique instance of a tcpdump
dumpfile filled with the specified count of packets in an instance file.
The instance file name is always of the form:
Each trigger has some
sort of implicit BPF expression capability
associated with it, which is not user specifiable at this time.
For example the UDP weight trigger
dynamically determines the top IP host address, and stores count
packets of the form: "host IP and (udp or icmp)", thus storing
UDP packets and any ICMP errors associated with the host IP that
caused the trigger to activate.
Current trigger types include:
In ourmon.conf, triggers in general should be placed at the
end of the configuration file as they assume that certain
filters exist and are turned on. They have the following
general syntax:
trigger_name threshold packet_count dump_dirname.
threshold - specify a threshold in bits/sec or pkts/sec or units
depending on the trigger type. When the named filter reaches that threshold,
packet capture will begin. The threshold is always specified in
terms of a backend graphics-engine value.
Packet dumping is turned off when either the count is exceeded
or the trigger threshold flow count is less than the current runtime value
-- which ever comes first. Packet dumping after being triggered cannot recur
between trigger on and trigger off times.
For example assume the packet capture count is 10000, and the threshold value is 2000.
If at mon.lite creation time, the value is 2000, a trigger_on event
is started and 10000 packets may be stored. If the count is exhausted and
the threshold is still > than the value, no more packets will be stored in
the file. Thus it is not possible for one trigger_on event to cause
more than one output file concurrently. When an event is retriggered,
a separate file with a different timestamp is created.
Trigger_on and trigger_off messages are put into mon.lite as "elog"
messages and are recorded in the back-end daily
event log.
As a result
it is possible to easily see when trigger on and off events have
occurred. The trigger dump filename is passed to the event log.
Note that if the probe and back-end graphics engine run on separate
computers, due to the imprecision of synchronization, a given
on or off message may NOT appear. However usually both messages
will appear, or at least the on or off message will appear.
Trigger messages in the event log appear as follows:
The trigger on message means that the trigger for tworm is on because
the current total "worm" count is 20 and has exceeded the configured
threshold of 10. The trigger off message indicates that packet
capture is done. The trigger will capture and store packets
into the file:
Note: at this time ourmon has no way to determine if the
dynamically created file has been created successfully
or if there is runtime storage room within the file system
in question. If the file cannot be created an elog message
will be sent. The ourmon probe will not exit.
It can be extremely useful to use the
saved back-end portreport.txt file for analysis here (see
logging
below). The relevant TCP port report file here
may give important clues about the nature of the attack
in terms of TCP destination port numbers, the number
of IP destination addresses, or the IP source addresses
involved in the attack. This information may
be useful in helping both to ignore irrelevant TCP syns
gathered during the attack and for searching the tcpdump
file for the IP addresses or ports in question (simply
tack on a BPF expression at the end of the tcpdump search).
For example, assume that the portsignature.txt
file shows that port 6666 was attacked, then one can do:
# tcpdump -n -r "dumpfile" tcp port 6666
This helps to narrow down the packet dump to a more
relevant set of packets.
This trigger is based on the
udperror_list (topn_icmperror). More information
can be found in the following sections:
Note that this threshold should be set the same both
in the front-end config file and in the backend
omupdate.sh file. The latter file causes the offending
line from the UDP port signature report to be placed in
the elog event file. As a result it becomes simple to identify
the IP source address of the offending UDP system. Also
the UDP destination ports in question are also supplied.
The capture filename has the form:
topn_udp_err.< timestamp >.dump
This trigger will capture and store UDP packets sent to and from
the top host in udperror_list. The internal BPF expression used is:
"ip_src and (udp or icmp)".
The event log entries for topn_udperror will appear roughly as follows:
The dump filename is created by concatenating the major and minor labels
along with a prefix of "bpf" and a timestamp to make the dump file unique.
For example, a dump for the above could have the following name:
The capture filename has the form:
IRC information is enabled in the front-end probe by
the ourmon.conf config tags:
The web
page has several kinds of data including:
The IRC summarization format may be broken down in the following way.
First the file itself consists of three sub-sections including:
The statistical layout for various kinds of formatting is explained below:
The foo channel
has one host in it with a total message count
of 161 messages broken down into total
counts for JOINS, PINGS, PONGS, and PRIVMSGS.
The total number of channels for the host
in question is shown.
The maximum work weight
for all instances is also shown. We also
show whether we believe the host is an
IRC server or IRC client. Note that a real
channel/host breakdown minimally has at least
two hosts in it (one client and one server).
The most important sections of the IRC summarization in security terms are probably
the very first channel sub-section that sorts channels based on the number of wormy hosts
and the very last part of the report that shows wormy hosts themselves.
We call channels with too many wormy hosts "evil channels". This simply means a channel
that may be botnet controlled.
The second important security section appears at the absolute bottom of the irc summarization
and gives a list of hosts that may be infected according to the work weight.
Channel names in IRC are case insensitive. Thus it is possible that a channel
named "exploit" and "EXPLOIT" are the same channel. This is probably true if
they have the same set of IP hosts associated with them. At this time our report
mechanism makes no attempt to join these channels into one channel. This may
be deemed a bug. However our reason for not doing so is to simply show how
the channel name is being spelled. This may be fixed in a subsequent release.
The event log is summarized in a
weekly summary .
The current
daily event log
is updated
hourly and at midnight becomes the event log for the previous day.
Summarized data at the bottom of the main web page includes:
Another search technique may be of use specifically
in the portreport directory. If a large external multiple host
attack occurs one can easily find the associated port
report simply by sorting on the size in lines of the port report file.
This is because each IP source address in the port report
gets its own line.
pkts: caught:9440 : drops:0: caught2:0 : drop2:0
fixed_ipproto
The fixed_ipproto filter is very simple. It simply
counts up TCP versus UDP versus ICMP bits, dumping any other IP protocol
into the "xtra" bin.
Typical "mon.lite" output is as follows:
fixed_ipproto: tcp:67402888 : udp:30976940 : icmp:23158 : xtra:47623:
A sneaky technique is used in the mon.lite file in that in some cases
mon.lite counts bytes per period, and omupdate.pl converts bytes to bits per second.
fixed_tcp3
The fixed_tcp3 filter accepts two TCP dst OR src ports. It then
counts packets displaying bits/sec with src/dst port1 versus port2 versus any
remainder packets, dumping any others
into the "xtra" bin. Note that xtra is all other bits, not just
TCP (this may be a silly idea).
Ourmon (the front-end) takes a configuration
file. The entry for this filter might take the following form:
fixed_tcp3 119 80
What this means is that we are interested in capturing USENET NEWS (port 119)
and HTTP traffic (port 80). As supplied, this filter captures email (25) versus
web (80) traffic.
Typical "mon.lite" output is as follows:
fixed_tcp3: 119:39908731 : 80:11033941 : xtra:47506497:
fixed_cast
The fixed_cast filter is performed at the Ethernet header
layer, not the IP layer, if and only if the supplied link device
is an Ethernet device. It displays bits/sec.
It classifies packets as Ethernet multicast,
Ethernet unicast, and Ethernet broadcast, based on the Ethernet
destination address. Note that this filter is potentially useful
for observing possible broadcast storms, whatever the cause, or multicast
routing meltdowns. At this time, assuming Ethernet inputs,
"xtra" packets should be 0.
The configuration file looks as follows:
fixed_cast 127.0.0.0 255.0.0.0
This is because we require an IP net/mask pair, just in case
one of the input interfaces is not Ethernet-based (the localhost device).
These addresses are ignored if Ethernet is used, but they must
still be supplied (just leave them alone in the supplied configuration).
fixed_cast: mcast:191214 : ucast:98257955 : bcast:0 : xtra:1440:
fixed_size
The fixed_size filter is performed at the Ethernet header
layer, not the IP layer, if and only if the supplied BPF device
is an Ethernet device. It displays pkts/sec. This filter counts packets within
four fixed byte bucket sizes, where the packet is <= 100 bytes (tiny), <= 500 (small), <= 1000 (medium),
or <= 1500 bytes (big). These packets may include errors.
The configuration file looks as follows:
fixed_size
fixed_size: tiny:51732 : small:13795 : med:13544 : big:32876:
fixed_l2proto
The fixed_l2proto filter is performed at the Ethernet header
layer, not the IP layer, if and only if the supplied link device
is an Ethernet device. It displays pkts/sec.
This filter counts packets according to the L2 ethernet header
for IP protocols only, including IPv4, IPv6, and ARP. Other
packet types are considered as "xtra". Note that this filter
might show an ARP storm.
The configuration file looks as follows:
fixed_l2proto
fixed_l2proto: ip:647819 : ipv6:1 : arp:48 : xtra:19:
user designed RRDTOOL-based BPF graphs
The user-mode BPF filters are a powerful part of
ourmon and allow programmable back-end RRD-based graphs.
They allow the user to design his or her own RRDTOOL graphs.
For example, we might have the following BPF filter set
examples in our ourmon.conf
filter specification file:
bpf-packets
bpf "protopkts" "ip" "ip"
bpf-next "tcp" "tcp"
bpf-next "udp" "udp"
bpf-next "icmp" "icmp"
bpf-bytes
bpf "ports" "ssh" "tcp port 22"
bpf-next "p2p" "port 1214 or port 6881 or port 4662 or port 6346 or port 6257 or
port 6699 or port 6257"
bpf-next "web" "tcp port 80 or tcp port 443"
bpf-next "ftp" "tcp port 20 or tcp port 21"
bpf-next "email" "tcp port 25"
bpf-noxtra
bpf-bytes
bpf-packets
These two commands cause an entire BPF filter set to
produce either bits/sec or pkts/sec.
bpf-next "tcp" "tcp"
means the label is "tcp" followed by the BPF expression which is also "tcp" in this
case. Let us look at a more complex example.
bpf-bytes
bpf "subnets1" "net1" "net 10.0.1.0/16"
bpf-next "net2" "net 10.0.2.0/16"
bpf-next "net3" "net 10.0.3.0/16"
bpf-noxtra
Or you can easily make up graphs that might watch a local server using expressions like:
bpf "hostA" "total" "host 10.0.0.1"
bpf-next "email" "host 10.0.0.1 and port 25"
bpf-next "web" "host 10.0.0.1 and (tcp port 80 or tcp port 443)"
bpf-noxtra
See the INSTALL file for more information on BPF graph customization.
CBPF BPF Optimization
cbpf "ctag" "graph-label" "line-label" "cfilter"
cbpf-next "ctag" "per-filter-label" "cfilter"
The "ctag" is used to signal to the configuration the syntactic form
of the CBPF filter. The "graph-label" and "line-labels" have the
same function as with the pure BPF filter set syntax. However "ctag"
and "cfilter" are totally different from BPF. Here are usage examples
for the existing set of CBPF tags shown with equivalent BPF
expressions:
1. network tag ("net")
cbpf "net" "subnets" "subnet8" "10.0.8.0/24"
bpf "subnets" "subnet8" "net 10.0.8.0/24"
cbpf-next "net" "subnet8" "10.0.8.0/24"
bpf-next "subnet8" "net 10.0.8.0/24"
2. tcp port range ("tcprange")
cbpf-next "tcprange" "bittorrent" "6881 6889"
bpf-next "bittorrent" "tcp and (port 6881 or port 6882 etc.)"
cbpf-next "udprange" "someports" "6881 6889"
bpf-next "someports" "udp and (port 6881 or port 6882 etc.)"
4. udp and tcp mixed single ports ("ports")
cbpf-next "ports" "edonkey" "u 4665 u 4672 t 4665 t 4661 t 4662"
bpf-next "edonkey" "(udp and port 4665 or udp port 4672) or (tcp and (port 4665 or port 4661 or port 4662))"
cbpf-next "tcpflag" "syn" "s"
cbpf-next "tcpflag" "fin" "f"
cbpf-next "tcpflag" "rst" "r"
bpf-next "fin" "tcp[tcpflags] & tcp-fin != 0"
bpf "tcpcontrol" "rst" "tcp[tcpflags] & tcp-rst != 0"
6. ping ("ping")
cbpf-next "ping" "pingline"
bpf-next "bpfping" "(icmp[icmptype] == icmp-echoreply) || (icmp[icmptype] == icmp-echo)"
cbpf-next "icmp" "ccportun" "3 3"
bpf-next "bpfportun" "icmp[icmptype] == icmp-unreach && icmp[icmpcode] == 3"
8. ip next protocol ("ipproto")
cbpf-next "ipproto" "tcp" "tcp"
bpf-next "tcp" "tcp"
cbpf-next "ipproto" "esp" "50"
bpf-next "esp" "ip proto 50"
topn flow graphs
The following links show the outputs for the various topn flow graphs:
topn_ip
topn_tcp
topn_udp
topn_icmp
The topn_ip filter provides a traditional top N flow
point of view for IP (any IP flow), TCP, UPD, and ICMP flows.
It shows the top N flows in bits/sec. An IP flow is defined
as a 5 tuple having this form: (IP src, IP dst, next IP protocol, L4 src port,
L4 dst port). TCP and UDP flows of course do not have the next IP protocol field.
ICMP flows display major and minor ICMP codes as opposed to L4 ports.
The ICMP major value is displayed as the L4 "source port", that is,
it is put on the left-hand side of the flow.
The ICMP minor value is displayed as the L4 "source port" on the right-hand
side of the flow.
topn_ip : 6954 : 131.252.208.43.65529->131.252.120.170.119(tcp): 18320510 :
128.223.220.30.40165->131.252.208.43.119(tcp): ETC ...
topn_tcp : 5596 : 131.252.208.43.65529->131.252.120.170.119: 18320510 : ETC...
topn_udp : 1257 : 209.70.46.6.27968->131.252.77.153.6974: 269300 : ETC ...
topn_icmp: 2: 131.252.3.1.0->131.252.2.1.0: 5234: 131.252.2.1.8->131.252.3.1.0: 5234: ETC ...
Note that the number following the topn_ip tag value above is the count of
distinct IP flows seen during the sample period. This is not the same as the top N flows
shown as tuples in the mon.lite file. It is a count of the unique flows seen during
the sample period, all of which have been stored in the hash list itself. But of course,
not all of them are printed out if the number of flows exceeds the top N value (currently
hardwired to 9, but this will change in the next release).
The flow count itself is currently
graphed in an RRDTOOL-style graph as it is very useful for anomaly detection.
See below in the anomaly detection section for more information on that graph.
topn ports graph
The following links show the output for the topn ports graph:
topn_tcp_port
topn_udp_port
The topn_port
filter displays the top N ports used in TCP and UDP
flows. The top N ports are sorted by packet byte count and expressed in bits/sec.
Up to 100 ports may be displayed with 10 ports per graph
for each TCP and UDP graph (which are separate).
The back-end topn_tcp_port graph has the following format in its graphs:
port/L4 src_count/L4 dst_count
mbits
The top port is displayed followed by the L4 src_count and
dst_count in the "legend" or top part of the label on the graph.
The port value may be either a src or destination L4 port.
For each port below the legend, we present bits/sec which is used for sorting the tuples.
(Effectively the port is the key, and the bit count is used for sorting).
The src_count/dst_count denote how many times the L4 port was a
source/destination *port*. Thus it may be possible to determine that a particular
port is only being used as a destination (or source) port.
Also the src and dst counter is a packet counter, not a byte counter.
The entry for this filter in the configuration file may be as follows:
topn_port 20
This indicates how many top port tuples should be written to the mon.lite file.
mon.lite output is as follows:
tcp_ports: 8472 : 80:56316509:49409:39209 : 6881:20155127:13459:13166 : ETC...
udp_ports: 2237 : 49156:3834617:3693:2758 : 49302:3834617:2758:3693 : ETC...
The number value followed the filter tag in the mon.lite output (tcp_ports : number : ...)
represents the number of distinct port tuples seen during the sample period.
N (e.g., 20) 4-tuples follow with the tuple format: (port, byte count,
L4 src port packet count, L4 dst port packet count). omupdate.pl rearranges this information
into a tuple more suitable for display.
Anomaly Detection
Ourmon provides a fair number of anomaly detection filters using both
supplied BPF filters, top N filters, and a few carefully chosen meta-data
RRDTOOL-based graphs. We will discuss the top N filters here first,
including some interesting features, and then go on to the supplied BPF
filters. It should be pointed out that in general the BPF filters
look at the big picture. For example, the TCP control BPF filter set
shows the overall number of TCP SYNS, FINS, and RESETS in a network.
On the other hand, the top N TCP syn filter shows the top IP hosts
sending out TCP SYNS. The former gives you a network-wide picture.
The latter helps show individual hosts that may be taking part
in an attack (or running gnutella).
TCP SYNS sent - TCP FINS returned > 20.
This mechanism seems to do a good
job of showing distributed zombie attacks simply because the number
of hosts matching the SYNS - FINS metric increases over some normal
bound by a significant number. The metric can be viewed as a low-pass
filter that leaves out "normal" applications that typically produce
roughly equivalent amounts of SYNS and FINS in a sample period. We
have also done some normalization experiments and can state that
unusual numbers of TCP SYNS aimed at TCP ports 445 and 139 do not
occur with normal Microsoft file sharing.
TCP Syn List Overview
The following links provide outputs for information derived from the topn tcp syn list:
topn_syn histograms
tworm RRD graphs
the TCP port signature report
detailed TCP port signature/tcpworm.txt report
hourly summarized port signature analysis report
the TCP port signature new scanner log
the p2p application version of the port report
In this section we discuss the TCP syn list tuple and various outputs associated with it.
We will first present a general overview ot the TCP syn list mechanism, then give
it configuration, and finally discuss each output presented above in turn.
1. the topn_syn list,
2. the tworm RRDTOOL graph,
3. the detailed tcpworm.txt output report,
4. the port signature report (which is a condensed parallel view of the previous item).
5. the ourmon scanner log which records new instances of port signatures in a day.
6. the port signature hourly summary batch reports.
7. the p2p port signature port which shows hosts using major P2P applications
like bittorrent.
SYNS sent + FINS sent + RESETS returned / total 2-way packet count.
Typical benign hosts score low work weights of 0%. A work weight of 100% means all control and no data and may
be deemed truly "anomalous". Signicantly anomalous hosts may have a work weight between 50..100%.
Intuitively we are measuring the amount of TCP control packets versus the total number of packets.
If there are large amounts of data the work weight will be low. Obviousally a SYN
scanner will score a 100% work weight. Of course an anomaly may represent a client that has no server.
Or it may represent a badly coded client or poorly performing TCP application. For example it is not unusual to
spot hosts using Gnutella with a high work weight because the Gnutella application is failing to
find Gnutella peers.
However in many months, we have seen only a handful of cases of such anomalies that were not worms,
and 1000s of cases that were worms. More details on the work weight are given below
when we talk about the port signature report.
SYNS - FINS > 20.
If this weight is satisfied, an ip source
tuple is added to the collection in the tcpworm.txt file. Note that this is a different
view of the original syn list data when compared to the topn syn list itself. The topn syn list
in the mon.lite file represents a sort of the syn list and gives us the top N syn sending systems. The tcpworm.txt
file gives us the complete set of ip sources where there were more SYNS than FINS by some number N.
As a result, the top N syn list is bounded as of course it is a top N list. The tcpworm.txt list is not bounded.
However typically barring large distributed attacks, it is far smaller than the total number of IP sources
in the complete topn N syn tuple list.
TCP Syn list ourmon.conf configuration
topn_syn 60
topn_syn_wormfile /home/mrourmon/tmp
topn_syn_homeip 10.1.0.0 255.255.0.0
topn_syn_p2pfile /home/mrourmon/tmp
tworm_weight 80
TCP syn list outputs -
topn_syn graph
10.0.0.1 f: 193: r: 247 t: 7254 72%:w
4779
10.0.0.2 f: 0: r: 20 t: 2035 100%:W
2015
syn_list: 9582 : 10.251.188.65:915:0:896:896:21:4654:3604:0:0:0:12:1:1:1433,4654,: next tuple ...
Note that the raw syn tuple is the same in both the mon.lite topn syn list,
the tcpworm.txt raw format file, and the p2pfile.txt files all produced by the front-end.
The first number after the syn_list tag is the total number of syn entries in the hash list.
It is followed by the specified number of top N syn tuples (60 say in our example).
The syn tuple, gives (IP src, syn count, syn+ack count, fins returned, fins sent,
reset count, total sent, total
returned packets, icmp errors returned, application flags (see p2p below),
a count of edonkey hits, total unique IP destinations sent, total unique TCP destination
ports seen, number of syn L4 destination ports sampled, and then two-tuples
that make up the port signature. Each tuple in the port signature is of the form (dst port,
packet count). The packet count here tells us that some number of packets including SYNS
and ordinary non-SYN packets were sent by the ip source in question to the L4 destination port.
TCP syn list outputs -
The tworm graph
The tworm graph attempts to capture the number of "wormy" hosts according to hosts
put in tcpworm.txt at one time. It is graphing the number of hosts that appear
in the tcpworm.txt file (which is the same as the number of lines in the file).
Hosts placed in this file are deemed to be "noisy" in that for some reason
they generate more TCP SYNS than TCP FINS.
Spikes in this curve may correspond to automated distributed
bot attacks which may be performing a DOS attack or simply scanning for exploits in parallel.
By default the tworm count information counts all IP sources appearing in the tcpworm.txt
file and classifies them as to whether or not they appear to be from the internal network,
or from an external network. (If internal versus external doesn't make sense, best to do something
like make the internal network 10.0.0.0 with netmask 255.0.0.0, thus making all IPs external).
The mon.lite config variable
tworm_weight 80
may optionally be used to filter the tworm count by the work metric. Thus one can approximate
the number of "real" worms as opposed to noisy P2P hosts or noisy web servers.
tworm: 9: 3: 6:
This tuple is placed in the mon.lite file and processed by omupdate.pl in the backend. This produces
the tworm RRDTOOL graph. The three numbers in turn represent:
1. a total count of "worms" (by default
the count here is the number of ip hosts found in the tcpworm.txt file.
2. the total count of systems in the home system that appear in the tcpworm.txt file.
3. the total count of external systems not in the home subnet.
We have observed external attacks in the 1000s made on one campus. These attacks are real
and this mechanism is very useful. It is fair to view this graph as a botnet attack detector.
TCP syn list outputs -
the port signature report
The ourmon system presents two processed versions of the front-end tcpworm.txt
file called "tcpworm.txt" and "portreport.txt" respectively. These files
represent different views of the TCP syn tuple information. (The back-end
tcpworm.txt report actually includes the portreport.txt information as well,
but for reasons of quick web lookup, the portreport is broken out into a separate report
file.) Both files are updated every thirty seconds but as they are ASCII outputs,
you must hit "reload" yourself on your web client.
ip src: flags apps: work: SA/S: L3D/L4D:
sdst/total: port signature
10.82.196.58 (WOM) 100: 0: 423/1
107/107: [5554,40][9898,59])
192.168.245.29 (O) B 6: 67: 622/438
277/411: [6880,0][6881,42][6883,6][... more]
E - An anomalous amount of ICMP errors are being returned to the IP source.
W - The work weight is greater than or equal to 90%. A W means the IP source
is highly suspicious.
w - The work weight is greater than or equal to 50% but less than 90%.
O - very few fins are being returned. O stands for "output" (or "ouch"?).
R - TCP resets are being returned.
M - no TCP packets are being returned to the IP src in question from receivers.
So for example, M here means there is no 2-way exchange of data.
B - a Bittorrent application was detected.
G - A Gnutella application was detected.
K - a Kazaa application was detected.
M - a Limewire/Morpheus application was detected.
I - an IRC application was detected.
e - an Edonkey/emule application was detected.
E - is used to indicate that packets are sent to destination port 25. The goal
here is to alert you to a possible spammer or a benign email server.
H - packets are being sent from port 80 or port 443. The sender could
be a noisy web server or nmap for that matter.
It should be noted that in general the applications flag field does not
usually find false positives, although the Edonkey flag is more prone to false positives
than the other fields. Also note that the E for email and H for web server flags
are only based on the existance of port 25 as a TCP port for email,
and ports 80 and 443 as destination ports for web servers. An attacker may very well be scanning
on those ports. The use of E and H based on TCP ports compared to B and G is a mixed metaphor, but we felt
it was useful to have a mechanism that would give us possible application clues and that
clues about email and web are important indeed.
[5554,40][9898,59])
For example [5554,40] means destination port 5554 was being scanned. The second field in
the port 2-tuple gives the overall percent or frequency of packets for that port in the total number
of sampled ports. In this case 40% of the total port count of packets were aimed at port 5554,
and 59% were aimed at port 9898. (For the actual numbers look in tcpworm.txt).
It should be noted that this is only a sample. It is not unusual to
see all 10 port tuples "full" of something that seems to be evenly distributed at 10%.
Such occurances are often due to web-based "noisy" client/server relationships and may be
benign. However in some cases this may represent a remote scanner that is simply walking the port space.
Scanners may be spotted by looking in the scanning log as their port signatures will be "new"
and will change over time. (Of course they may also show up in the top N scanner part of
ourmon as well).
135-139, 445, 1433, 1434
In typical use one does not see large discrepencies in ordinary
use of the Microsoft File Share distributed file system or with the use of a SQL server.
The work weight here simply does not matter (and high is still bad!), and in fact with port 1433,
a low work weight is a bad sign because it means hosts are your site may be responding at Layer 7
to attacks. Ports 80 and 443 of course may or may not represent attacks on web servers.
A low work weight with port 80 might mean that attacks are being launched at a web server
and it is returned failure messages at Layer 7. It might also mean that a web server
is sending lots of small connections back to a web client and in that case is benign.
Certain other port combinations represent well-known viruses as (5554, 9898) is an example.
If you see new port signatures, it can be useful to search google for suspicious port combinations
(as well as the dshield site and web pages offered up by various anti-virus vendors).
TCP syn list outputs -
the tcpworm report
The tcpworm report is essentially an expanded "translated" version of the raw tcpworm.txt file.
It is sorted by the syn list; that is IP sources with more syns will appear before
IP sources with less syns. We may divide it up into three parts: 1. per IP source statistics,
which include all the information the front-end has gathered about a specific IP source, 2. various
summary statistics., and 3. the port signature report, which is also included in here. The port signature
report is included so that one can use a text editor and simply pattern-match back and forth
to learn more details about IP sources in the port signature report deemed of interest.
TCP syn list outputs -
the p2p report
The p2p report (found on the irc page) is a separate front-end output and is another topn syn
feature. It is functionally all syn list IP src-based tuples that have set the application
flag field barring port flags for email and web services. In other words, it is
the set of IP hosts seen during the sample period that initiated P2P exchanges for
Bittorrent, Kazaa, Gnutella, Morpheus/Limewire. It also includes any host that
sent or received IRC messages including JOIN, PRIVMSG, PING, or PONG. See the previous
portreport section
for more format information. This file uses the same format as the TCP portreport,
but is an entirely different subset of IP sources taken from the entire set of
hosts sending TCP SYN packets. Note that there is no port signature database
associated with this file. The port signature database is only used with the TCP portreport.
TCP syn list outputs -
the TCP port signature new scanner log
The TCP scanner log is a daily log that provides
new port signatures -- where new means a TCP port signature that has not been observed
since midnight of the current day.
New port signatures are stored in a db database that is turned over every night at midnight.
Port signatures are put in the scanning log with the following form:
Tue Nov 2 08:00:29 PST 2004: new worm signature from:192.168.186.238 [5554][9898][16881]
A TCP scanner seen scanning over a long period may have multiple entries in the log.
Note that this log is potentially updated every 30 seconds.
TCP syn list outputs -
batch port signature reports
The ourmon back-end generates an hourly report that is a summarization of the TCP portreport.
The summarization uses a weight supplied in the back-end that averages all instances of
the TCP work weight found so far during the day. Only IP sources with all
tuples averaging at least equal to or above the supplied weight are shown in the summarization.
The goal is to leave out sporadic weights (of say email servers) where once in a while an IP source
will have a high work weight, but generally does not have a high work weight.
If you wish to tune this weight modify the back-end bin/batchip.sh script where it
calls ombatchsyn.pl to make this log. As this is a batch report, reverse DNS lookup is
performed. The output filename is "allworm_today.txt".
icmp and udp error list
The following links show various outputs associated with the topn icmperror list:
topn_icmperror
topn_udperror
udp port signature report
udpweight graph
ourmon system event log
The icmperror_list top n mechanism is specified in
the ourmon.conf file as follows:
topn_icmperror 20
10, 20, 30 ... 100 individual IP hosts may be specified with 10 outputs
per picture in units of 10.
icmperror_list: 20779 : 10.0.0.10:984:0:4:984:984:0:0: 10.0.0.2:125:5416:3:128:106:0:19: ETC.
udperror_list: 13435 : 10.1.2.3:5922:464:401:35:0:144:104:2:53,293,32768,8: ETC.
As usual, the number of list entries in the hash list follows the tag.
For the icmperror_list, tuples are as follows: (ip src address: icmp total error
count: tcp pkt count: udp pkt count: ping pkt count: icmp unreachable count:
icmp redirect count: icmp ttl exceeded count). This means that
for a given IP source address, we show the total number of ICMP errors,
where errors are the sum of ICMP unreachables, redirects, and TTL exceeded
packets returned to that IP src during the sample period. We also
attempt to give clues as to whether or not the system is sending
TCP packets, UDP packets, or ICMP ping packets. The list is
sorted by the total number of ICMP errors.
In the graphical histogram representation for the icmperror_list
we use a flag-based system as follows:
T: TCP packets sent by an IP source.
U: UDP packets sent by an IP source.
P: ICMP echo request packets sent by an IP source.
Un: ICMP unreachable packets sent back to that IP source.
R: ICMP redirect packets sent back to that IP source.
X: ICMP ttl eXceeded packets sent back to that IP source.
We only show a flag:count if the result for the IP source address
is non-zero. The list again is sorted by the total number of ICMP
errors.
In the next few sections,
we will discuss the UDP list in more detail as it has several features.
topn_udperror
and
udp port signature report
The UDP port signature report comes in two flavors. As the UDP port signature
information is fundamentally a top N list we include a histogram-based graphic,
which is simpler in form than its more verbose ASCII cousin. The former
may be called the "topn_udperror list" (graphic) and the latter is simply
called the "udpreport". The udpreport layout is somewhat similar to the TCP report,
but it is sorted by weight, not IP address. However the UDP port signature field
of the ASCII report is completely similar to the TCP port signature field.
UDP pkts sent - UDP packets received * ICMP errors.
It is quadratic and we have seen instances of this value above
one billion (a DOS attack). The ICMP errors consist of ICMP unreachables, redirects, and TTL exceeded pkts
sent back to the IP source in question. ICMP (port) unreachables weigh slightly more and will raise the weight.
U: UDP packets sent by an IP source.
R: UDP packets sent back to that IP source.
Un: total ICMP unreachable errors sent back to that IP source.
P: total ICMP PING packets sent from the IP source.
As usual, the weight is shown below the label that starts
with the IP source address. Tags are shown only if they have non-zero
counts.
ip src: weight: udp_sent: udp_recv: redirect:
ping: L3D/L4D: sdst/total: port signature
10.82.196.58 7672 138 0: 28
1 10/1 1: [137,100])
udpweight graph
The udp weight graph
shows the weight of the single UDP error list top talker per sample period.
This allows you to more easily estimate the time of a UDP attack and
also to gain some feeling for the statistical variance in the weights
shown in the UDP port signature report. Thus you may be able to
decide to increase or decrease the UDP weight threshold passed to omupdate.pl
or given to the ourmon probe to trigger automated packet storage.
See the previous section for more information.
topn scans
The following links show output associated with the topn scan lists:
ip_scans
ip_portscan
tcp_portscan
udp_portscan
The topn_scans filter (producing graph ip_scan above)
counts packets sent from a unique IP source to multiple IP destination addresses
during the sample period. Therefore we can say it is 1-N in terms of IP source
to IP destination mapping. Here we sort on the maximum unique IP destinations.
topn_scans 20
topn_port_scans 20
The number supplied should vary from 10..100 by values of 10. 20
will produce two topn scan graphs with the usual histogram.
There are four mon.lite outputs
(discounting STATS which are used for internal tuning).
mon.lite output is as follows:
ip_scan: 18908 : 10.0.0.1: 2340 : 10.0.0.3 : 2295 : ETC.
ip_portscan: 18563 : 10.0.0.1: 2366 : 10.0.0.4 : 2242 : ETC.
tcp_portscan: 16303 : 10.0.0.1: 2366 : 10.0.0.4 : 2242 : ETC.
udp_portscan: 2526 : 10.0.0.2 : 242 : 10.0.0.5 : 182 : ETC.
The topn_scans filter produces one output called ip_scan.
The topn_port_scans filter produces three outputs, called
ip_portscan, tcp_portscan, and udp_portscan, respectively.
Each output has the number of tuples following the tag,
and each tuple is a 2-tuple of (ip source address, unique destination count).
The destination count is the number of unique IP or L4 port destinations.
topn flows and anomaly detection
The following links show various aspects of the topn flow mechanism
and anomaly detection:
topn_ip flow RRDTOOL count graph
topn_ip insert count
topn_icmp
topicmp_today.txt
BPF graphs and network anomaly detection
A number of BPF filter sets are provided in the default configuration that show overall network
error and control information.
BPF network errors
The
graph above shows a total
count of TCP resets, ICMP unreachables, ICMP pings, and ICMP ttl
exceeded errors. TCP resets often correlate with TCP scanning attacks.
ICMP unreachables may correlate with a UDP attack.
BPF view of ICMP unreachable packets
The bpf-unreachable BPF filter set breaks out a number of different
kinds of ICMP unreachable errors. Network, host, port, and administrative
prohibited unreachable packet counts are shown. Keep in mind that UDP-based attacks
may cause large numbers of port unreachables. TCP and UDP attacks may
produce administrative unreachable errors if ACLs in Firewalls or Routers
return those kinds of errors. Yes, Virginia, it might just be a good idea
to do that to detect scanners, as opposed to returning nothing.
BPF view of TCP control packets
The bpf-tcpcontrol BPF filter set shows a breakdown of network-wide
TCP control packets, including SYNS, FINS, and RESETS. This may
be of use for spotting SYN anomalies as well as other kinds of anomalies.
It can sometimes be possible to spot an attack on this graph,
and then look through the topn_syn logs
(as well as the tcpworm.txt and tworm graphs)
to determine more information about the source of an attack, possibly
including IP source addresses.
Triggers - automated packet capture
The automated trigger mechanism allows the probe to dynamically store
a set of "interesting" packets which may be filtered internally
with various BPF expressions and then written out to a tcpdump style dump file.
trigger_tag.timestamp.dmp.
The timestamp combined with the trigger_tag makes the filename
unique. In general, only one trigger of each trigger type may be
active at a time. In this release, there are two kinds of
triggers. At this time we include a trigger for tworm events
and an additional trigger for UDP weight (udp error) events.
More triggers types may be released in the future.
tworm - a trigger may be set on the total worm count.
topn_icmperror (udplist) - a trigger may be set on the top udp error weight for a specific IP host.
bpf_trigger - a trigger may be set on any BPF expression in a BPF filter set.
drop-trigger - a trigger may be set on the pkts filter when the number of drops exceeds
a certain threshold.
It is important to note that triggers may be more useful in terms of
the packets captured if the particular BPF expression
is more specific. This problem can be called the "signal to noise problem".
For example, a BPF expression that captures all TCP packets may not show
anything useful. On the other hand, the udp error weight trigger only
captures UDP packets for a particular IP source. Thus the tcpdump
capture file in
question is more likely to show the exact nature of an attack.
packet_count - terminate when count packets are stored.
dump_dirname - directory in which to store the dump file.
This directory should be created before the ourmon probe is run.
Tue Mar 1 09:37:01 PST 2005: ourmon front-end event: tworm trigger on, current count: 45, threshold 40, dumpfile: /usr/dumps/tworm.<03.01.2005|09:36:53>.dmp
Tue Mar 1 09:38:01 PST 2005: ourmon front-end event: tworm trigger OFF, current count is 20, threshold: 40
/home/mrourmon/dumps/tworm.<03.01.2005|09:36:53>.dmp.
The contents of this file may be viewed with tcpdump.
Since the timestamp typically includes shell metacharacters one
can usually cut and paste the timestamp name in between double
quotes (or just use * globbing) as follows:
# tcpdump -n -r "tworm.<03.01.2005|09:36:53>.dmp"
tworm trigger
In the ourmon configuration file this trigger
has the following syntax:
trigger_worm threshold packet_count dump_dirname
The trigger_worm trigger is associated with the
tworm graph .
One should
determine a suitable configuration threshold by
watching this graph over time. Note that the threshold applies
to the tworm total count (not us, not them, the total ).
If the total value for tworm counts (us+them) exceeds the supplied
threshold value, a packet dump will begin for count packets in the
dump_dirname in a dynamically created filename.
Only TCP SYN packets are stored.
The filename has the form:
tworm.< timestamp >.dmp.
udp weight trigger
In the ourmon configuration file this trigger has the following syntax:
udperror_trigger threshold packet_count dump_dirname
icmp and udp error list
udp weight graph
udp port signature report
In terms of the back-end the threshold value is simply
the UDP weight calculated for the first host in the udperror_list.
This value is graphed in the udp weight graph .
We suggest that the threshold be set at 10000000 to start.
It should be tuned down or up depending upon the incidence
of false positives versus "interesting events". In general
UDP scanning does occur and this mechanism will catch outrageous
incidents. It may also catch disfunctional P2P applications or
multimedia games. Of course what you do with that information,
will depend upon your security policy.
Fri Mar 4 01:17:33 PST 2005: udpweight threshold exceeded:10.0.0.1 14841330 7284 1194 1218 0 10: [1649,0][7674,80][22321,18][33068,0][40167,0][54156,0][55662,0][61131,0][64734,0][ 65021,0]
Fri Mar 4 01:17:33 PST 2005: ourmon front-end event: topn_udp_err:
trigger on, current count: 14841330, threshold 10000000, dumpfile: /usr/dumps/topn_udp_err.<03.04.2005|01:17:05>.dmp
There are two event log messages associated with the UDP error event.
The first message is simply the first line of the udp port report
and is generated by the backend. The offending IP is given, along
with various statistics taken from the UDP port reporting including
ports. The second line is generated by the probe and gives the
trigger capture filename. Note that since udp port signature
files are also logged, it may sometimes be useful to refer to the port signature
file at the time of the attack. Given that the port signature
line for the incident in question is in the log this may not be the
case. On the other hand, a remote multiple system attack is always
a possibility and thus the port signature file could provide additional
details.
bpf trigger
In the ourmon configuration file this trigger
has the following syntax:
bpf_trigger "filterset_label" "bpf_line_label" threshold packet_count dump_dirname
The BPF trigger capability works with any individual BPF expression in a BPF filter set.
The bpf_trigger that matches a BPF expression must be declared in the config file
after the filter set declaration. The trigger expression first names the filter
set and then names the BPF line label as a two-tuple - thus determining the BPF
expression used for the trigger. All packets stored match the BPF expression.
The threshold value is either packets or bytes depending upon the declaration of
the filter set and as always is expressed in terms of back-end graphics output.
If the user bpf filter set was declared as follows:
bpf-packets
bpf "protopkts" "ip" "ip"
bpf-next "tcp" "tcp"
and the desired target BPF expression is "tcp", then the bpf_trigger
to match that expression would be:
bpf_trigger "protopkts" "tcp" 1000 10000 /usr/dumps
which means that the trigger should be turned on to store 10000 packets
when a threshold of 1000 packets per second is reached.
bpf_protopkts_tcp.<06.03.2005|01:02:37>.dmp
drops trigger
In the ourmon configuration file this trigger
has the following syntax:
drop_trigger threshold packet_count dump_dirname
The drop trigger is associated with the
pkts graph . It is triggered by
the (first) drops counter. The assumption here is that this
trigger is not normally triggered, but only triggers due to
large DOS attacks. It is possible that the trigger might
thus catch a DOS attack (especially if the DOS attack
is the major cause of packets).
The supplied packet count value should probably not be small as it is quite
possible that this filter might store "good" packets as opposed
to bad packets.
drop.timestamp.dmp
The threshold for this trigger is based on packet counts.
IRC information
IRC information is presented on its own web page.
IRC data is reported for two reasons, both security related.
It has long been known that IRC may be used by "hackers"
as the control plane for hosts that have been captured and
become part of a remote botnet. Client botnet hosts may
in turn be used for launching attacks on local (or remote) hosts
including denial of service attacks, and "fan-out" attacks using
various well-known exploits. Thus client IRC hosts may act
as TCP-based scanning hosts (UDP activity is not ruled out).
Second it is always possible that IRC may be used for the dissemination
of "warez" (meaning multimedia data including video and audio).
Whether this is allowed or not within an administrative domain
depends on the local security policy. Ourmon looks at statistical
data for IRC and does NOT look at PRIVMSG content. The main
goal is to identify IRC channels that appear to be under the control
of computers as opposed to people.
topn_irc 0
topn_irc_file /home/mrourmon/tmp
These two switches should both be used and never be
given separately. (0 is a placeholder for a possible future feature).
The latter switch directs where
the front-end output irc.txt file should be stored.
It may be overridden by the use of the -D parameter
with the ourmon probe. The ourmon probe uses a lightweight
layer 7 scanner to look for certain IRC messages including
JOIN, PRIVMSG, PING, and PONG and gathers a small set of
RRDTOOL statistics based on global counts for those messages.
It also gathers two sets of tuples including irc host
tuples and channel tuples. The host tuple
contains statistical information about IRC usage for a given host participating in IRC communication.
The channel tuple contains statistics about IRC channels including
the set of hosts participating in the channels. IRC information
gathering is not based on any TCP destination port and ignores ports.
The information produced also includes data taken from the topn Syn
module and includes a TCP syn work weight for each IRC host.
Thus it is possible to determine if an IRC host may be infected
by a worm/scanner/bot. Tuple information for IRC hosts and channels
is placed in the per sample period irc.txt file and handed over
to the back-end for further processing.
1. The p2p port signature report .
See that section above for more information. The p2p
report is included here simply because it also includes
information about IRC hosts. IRC hosts are given
an I as a flag in the application flags section. Note
that this file thus contains all IRC hosts visible during
the sample period.
2. The next section gives a processed version of the raw irc.txt
file. (This work is done by bin/irc.pl). A 30-second sample
period report is given, followed by a week's worth of daily summarizations.
Note that the first summarization is for the current day and as usual
is performed hourly during the current day. It is highly likely
that the summarizations for the current day and yesterday are the most
important items due to the sporadic and slow communication patterns
for IRC as a whole. We will explain the file format for the IRC
output report below in more detail.
3. RRD graphs for total IRC message counts are then provided.
Note that JOIN spikes here may indicate some sort of IRC "war" amongst competing IRC groups.
1. global statistics, 2. channel statistics, and 3. IRC host statistics.
Channel statistics are given in major sections including:
channels sorted by wormy hosts
channels sorted by max messages
channels with associated host IPs (and host stats)
channels with no PRIVMSGS, only JOINS
channels with any wormy hosts
Host statistics follow:
most busy server in terms of max messages
hosts with no PRIVMSGS, only JOINS
hosts with any sign of worminess
channel
msgs
joins
privmsgs
ipcount
wormyhosts
evil?
exploit
3
0
3
5
4
E
We have a channel named "exploit" with a total of 3 messages (all PRIVMSGS). Five hosts appeared
in the channel and 4/5 had a work weight greater than 60%. (The work weight in the IRC summarization
is not an average but the maximum work weight seen as we are looking for any signs of suspicious
behavior). An E flag is awarded if a channel has more than one wormy host and at least half of
its hosts (discounting servers which may be botnet controllers) are infected. The small e flag
is given if a channel only has one host with a "bad" work weight.
channel
ip_src
tmsg
tjoin
tping
tpong
tprivmsg
maxchans
maxworm
Server?
foo
10.0.0.1
161
30
67
63
1
1
99
S
Ourmon Logging
Logging in ourmon takes various forms.
1. RRD data is stored in the base back-end rrddata directory. The
file rrddata/ourmon.log records RRD errors and should
be consulted when one is attempting a new user BPF filter-set,
especially when the filter-set graphics do not appear!. The RRD
data gives a year's worth of baselined data for RRD graphics.
Note that RRD data files reach their maximum size at creation time
and do not grow larger over time.
2. The event log
gives significant events in the life of the ourmon system
including probe reboots and important security events.
More information is available below.
3. Web summarizations
represent hourly (and eventually daily) reports
available via the ourmon web pages. For example
various top N summarizations and IRC summarizations
are available. Most summarizations are available at the
bottom of the
main web page .
IRC summarizations
are on a separate page.
In general one week's worth of summarizations are available.
4. Basic log information is stored in the back-end logs directory.
This information includes top N logs, front-end files like mon.lite,
and important security files including the TCP port signature report
in the logs/portreport directory, and the UDP port signature
report in the logs/udpreport directory. Back end summarization
scripts run at hourly intervals and summarize this data displaying
some (but not all) of it in the form of web-pages including IRC, top
N, and other kinds of summarizations. Note that after running
ourmon for one week, in general the logs directory does not grow
any bigger as the current day is rolled over at midnight
(and the new data is zeroed out). Note that log files
are typically NOT web summarizations. They represent 30-second
sample reports. Taken together they can be used to produce
a web summarization which typically is done on an hourly or
daily basis.
At this time the back-end logs directory includes the following:
portreport - these are the back-end processed TCP port
signature report files (portreport.txt) and may be crucial to
security analysis.
udpreport - these are the back-end processed UDP port signature
reports. These files may be useful for security analysis.
p2preport - these are the back-end processed TCP
syn p2p portreport files and show which systems are doing IRC,
and various forms of p2p activity.
daily files (Mon Tue etc) - these are top N log files and in some cases
are used for daily summarization.
irc - these are the back-end processed per sample irc report files.
rawirc - these are the front-end irc.txt files and are used for
the back-end web summarization.
mon.lite - these are the front-end mon.lite files.
tworm - these are the front-end tcpworm.txt files.
Event Log
The event log records important events in the life of ourmon
including probe reboots and nasty back-end errors. It also
icludes important security events that in general should
not occur too often (we hope). At this time security events
include:
1. tcp worm trigger events
2. udp weight trigger events
3. udp weight port signature threshold events which provide
the IP source address of the UDP offender.
Web summarizations
Web summarizations are produced for some
top N data
and other data as well.
Summarizations
are a form of logging in which statistics are aggregated together
and summarized in various ways. Daily summaries are updated
at the hour and typically rolled back a day at midnight producing
a week's worth of summaries. Summarized data is stored
in the ourmon web directory of course and not in the logs directory.
However the logging directory data is used to produce the
web summarization.
Other summarizations exist and are discussed elsewhere including
the
IRC data
and the
TCP scanner log .
(The work weight parameter (-W) may be modified
by changing bin/batchip.sh on the line that begins:
$OMBATCHSYN -W 100 -N -c 1000 etc. )
Sorting within the above restrictions
is done by the maximum number of syns. Data is represented in
the same fashion as with the top N syn summarization above.
All of these summarizations are roughly similar to the basic 30-second
port report, but differ in the format of the output. For each
IP source we have three lines, where the first line gives statistics
averaged across all samples (filtered in), the second line gives time information,
and the third line gives a port signature sorted by the busiest sampled port
(busiest in terms of the most packets sent to that port).
Details for the three lines are summarized below:
ip source, followed by work weight given with min, average, and max values,
followed by average sa/s, L3 ip dst, and L4 port dst counts, syn/fin/reset counts,
and total packets sent and received. Note these counters generally are *averages*.
the total number of instances, followed by start and stop times (first and last times seen in the set of daily instances).
the port sigs are given sorted by max packets. Here each tuple gives the destination port,
followed by the total count (not an average) of the packets seen sent to that port.
Security Analysis and Logging
A couple of tricks that can be used for searching the log
directory data should be pointed out. For example assume
that you have an IP source address of interest and you wish
to search the TCP portreport logging directory to see if that
IP source is mentioned. Assume that you know friday
is also a day of interest.
# cd /home/mrourmon/logs/portreport/Fri
# find . | xargs grep 10.0.0.1
The use of xargs here combined with grep allows you
to easily search the log directory for the IP source
address 10.0.0.1. This technique of course can be used
in other logging directories as well.
# cd /home/mrourmon/logs/portreport/Fri
# find . | xargs wc -l | sort
Wc is used to count the lines in the file, and sort will
sort out the results showing you the biggest file.