by Jeffrey Fulmer <jdfulmer@armstrong.com>
December 2000
Table of Contents
Siege is an http/https regression testing and benchmarking utility. It was designed to let web developers measure the performance of their code under duress, to see how it will stand up to load on the internet. It lets the user hit a web server with a configurable number of concurrent simulated users. Those users place the webserver "under siege." The duration of the siege is measured in transactions, the sum of simulated users and the number of times each simulated user repeats the process of hitting the server. Thus 20 concurrent users 50 times is 1000 transactions, the length of the test.
Performance measures include elapsed time of the test, the amount of data transferred ( including headers ), the response time of the server, its transaction rate, its throughput, its concurrency and the number of times it returned OK. These measures are quantified and reported at the end of each run. Their meaning and significance is discussed below.
Siege has essentially three modes of operation, regression,
internet simulation and brute force. It can read a large
number of URLs from a configuration file and run through
them incrementally ( regression ) or randomly ( internet
simulation ). Or the user may simply pound a single URL
with a runtime configuration at the command line ( brute
force ).
The format for invoking siege is: siege options...
siege supports the following command line options:
' -V '
' --version' Print version information to the
screen.
' -h '
' --help' Print the help section. This presents a
summary of the options discussed in this section of the
manual.
' -v '
' --verbose '
Verbose output. With this option selected, siege will
print transaction information to the screen. This includes
HTTP protocol type, the return code and the page it
requested: HTTP/1.1 200
OK:
/cgi-bin/whoohoo.cgi?first=Homer&last=simpson
This option is especially useful for charting progress in
regression or internet modes when the program is hitting a
large number of assorted URLs.
' -c NUM '
' --concurrent=NUM '
Concurrent users ( requires argument ). This option
allows the user to stress the web server with NUM number of
simulated users. The amount is limited only by the
computing resources available, but realistically a couple
of hundred simulated users is equal to many times that that
number in actual user sessions. The number you select
represents the number of transactions your server is
handling. It does NOT represent the number of concurrent
sessions. Remember, real users take some time to actually
read the page that they've requested....
' -u URL '
' --url=URL '
The URL option allows the user to stress a single URL
configured as an option on the command line. This is the
brute force mode described in the introduction. With this
option the developer can pound a single URL with a desired
number of simulated users. Typically this is used to see
how a new program holds up under duress. You can stress
cgi programs by placing information in a query string,
i.e., www.whoohoo.com/cgi-bin/whoohoo.cgi?first=Homer&last=simpson
' -i '
' --internet '
This option is used with a configuration file, that is a
file containing many URLs. With this option in place, each
user randomly hits any one of the URLs in the file each
time it hits the server. Much like you can't tell the
users of your website which pages they can view, you have
no control over which pages siege will hit in internet
mode. With this option set, there is no guarantee that
every page in the file will be "viewed." Run this option
in verbose mode the first couple times to better understand
the internet simulation.
' -t NUM '
' --times=NUM '
With option, you instruct siege to run the test NUM number
of times. That means that EVERY simulated user repeats its
activity NUM times.
' -f FILE '
' --file=FILE '
The default configuration file, the file with all your
URLs is SIEGE_HOME/etc/urls.txt. You can use this option
to instruct siege to use a different configuration file:
siege
--file=serverb.txt
' - l '
' --log '
This option instructs siege to log the statistics to
SIEGE_HOME/var/siege.log. Each new statistics set is
appended to the log.
' - m MESSAGE '
' --mark=MESSAGE '
This option allows you to mark the log file with a separator,
to differentiate your log file entries with header information.
It is not necessary to use both the -m option and the -l option.
-m assumes -l so it marks and logs the transaction. If the MESSAGE
has spaces in it, make sure that you put it in quotes.
' -d NUM '
' --delay=NUM '
Each siege simulated user is delayed for a random number
of seconds between one and NUM. If you are benchmarking
performance, it is recommended that you use a 1 second
delay ( -d1 ). The default value is three (3 ). This
delay allows for the transactions to stagger rather then to
allow them to pound the server in waves.
Siege understands the following URL format:
[protocol://] [servername.domain.xxx] [:portnumber] [/directory/file]
Currently, siege only supports http and https protocols. HTTP is the
default protocol and therefore does not require a protocol specification. Frankly
siege allows you to cheat. The minimum requirement is this: servername.
That's it. So if you're in the same domain as a server named shemp
and shemp is in your host file or it is in DNS, then: siege -u shemp will
stress http://shemp.yourdomain.net/index.html ( assuming that is the server
specified index ). If you want to lay siege to https servers, then it is
necessary to specify the protocol. In the above example, siege -u https://shemp
Will lay siege to https://shemp.yourdomain.net/index.html given the assumptions stated above.
In order to run a regression test or an effective internet
simulation, you are going to have to run through the URLs
on the server you are testing. To accomplish that, place
the URLs in a configuration file. The default file is
SIEGE_HOME/etc/urls.txt. In that file list the URLs one per
line:
# comments behind
hashes
http://homer.whoohoo.com/index.html
http://homer.whoohoo.com/howto.jsp
http://homer.whoohoo.com/cgi-bin/hello.pl?first=bart&last=simpson
# and on and
on....
When invoked without the URL option [ -u URL | --url=URL ],
siege looks for URLs in a file. It reads it into memory
and runs through the URLs. Normally siege starts at the
beginning of the file and works it way through it
sequentially. If you specify internet mode [ -i |
--internet ], then it selects URLs randomly. An
alternative file can be selected at run time with the [ -f
FILE | --file=FILE ] option.
When siege is invoked with logging enabled [-l/--log], the program records the transaction in PREFIX/var/siege.log where PREFIX is the directory in which siege was installed. ( see INSTALL for instructions. ) The transaction logged is similar to standard output display at the end of every siege run. However, the information is arranged in comma separated text for easy import into a spread sheet. The logging option enables you to maintain history and chart progress over time.
In order to group runs by conditions such as URL, server or even protocol,
the -m "message" / --mark="message"> option was added. This
places the mark "message" in the log file. So that if you switched your
testing from http to https, you could mark the log with "start HTTPS testing"
-m/--mark assumes logging and makes the -l/--log option unnecessary.
Below is a sample log file:
Elap Data Resp Trans
Date & Time, Trans, Time, Trans, Time, Rate, Thruput, Concur, OKAY
02/22/01 08:05:47, 10, 10.64, 36570, 4.25, 0.94, 3437.03, 3.99, 10
02/22/01 08:14:47, 250, 267.31, 914250, 4.33, 0.94, 3420.19, 4.05, 250
02/22/01 08:23:35, 25, 26.56, 91425, 4.24, 0.94, 3442.21, 3.99, 25
02/22/01 08:27:58, 25, 8.08, 96825, 0.48, 3.09, 11983.29, 1.50, 25
02/22/01 08:28:22, 25, 8.60, 96825, 0.57, 2.91, 11258.72, 1.67, 25
02/22/01 08:28:37, 25, 7.59, 96825, 0.36, 3.29, 12756.92, 1.19, 25
02/27/01 00:40:01, 250, 61.86, 5024760, 1.35, 4.04, 81227.93, 5.46, 230
02/27/01 00:44:39, 250, 63.77, 5024760, 1.51, 3.92, 78795.04, 5.92, 230
02/27/01 00:48:00, 250, 62.05, 5024760, 1.43, 4.03, 80979.21, 5.75, 230
**** start HTTPS testing ****
02/28/01 19:54:28, 250, 32.65, 1655500, 0.21, 7.66, 50704.44, 1.59, 250
02/28/01 19:57:20, 250, 33.14, 1655500, 0.28, 7.54, 49954.74, 2.12, 250
02/28/01 20:02:23, 250, 28.14, 1655500, 0.11, 8.88, 58830.85, 0.95, 250
02/28/01 20:04:25, 250, 26.08, 1655500, 0.02, 9.59, 63477.76, 0.24, 250
02/28/01 20:04:56, 250, 26.30, 1655500, 0.03, 9.51, 62946.77, 0.28, 250
Performance measures include elapsed time of the test, the
amount of data transferred ( including headers ), the
response time of the server, its transaction rate, its
throughput, its concurrency and the number of times it
returned OK. These measures are quantified and reported at
the end of each run. The reporting format is modeled after
Lincoln Stein's torture.pl script.
This is a sample of siege output:
Hardy: $ siege -u
shemp.whoohoo.com/howto/Admin.html -d1 -t10 -c25
** Siege 1.00 2000/12/13
23:42:16
** Preparing 25
concurrent users for battle.
The server is now under
siege...done
Transactions: 250
hits
Elapsed
time: 14.67 secs
Data
transferred: 448000 bytes
Response
time: 0.43 secs
Transaction
rate: 17.04 trans/sec
Throughput: 30538.51
bytes/sec
Concurrency:
7.38
Status code
200: 250
Problem: siege: FATAL: Unable to create IPC
message queue
This error is usually caused by an improper termination of
the program, siege was not given a chance to clean up after
itself and it has an old IPC message still on the system.
You are going to have to clean it up manually:
This command will enable you to identify the msg
$ ipcs -q
------ Message Queues
--------
key msqid
owner perms used-bytes messages
0x00000fcc 5120
jfulmer 666 0 0
As you can see, there is a message queue still on this
system. We don't want it there, so we will have to delete
it.
$ ipcrm msg
5120
resource deleted
Problem solved! This example was generated on a GNU/Linux system. Unfortunately these commands vary across UN*Xen. You may have to read the ipcs and ipcrm man pages on your system for the exact syntax.
Problem: Siege won't compile on my
system.
Send the compiler output, system information to the author
<jdfulmer@armstrong.com> In the meantime, you might
consider running Lincoln Stein's torture.pl perl script.
If you are trying to compile it on windows, don't bother.
Siege using the UNIX fork() function which is not available
on Windows.
The latest version of siege is available via anonymous
ftp.
Mirrored at nerf-herder.net
AIX( powerpc-ibm-aix4.2.1.0 )
* Siege has been built successfully using IBM C for AIX
version 5
* The protocol bug on AIX 4.x has been fixed.
GNU/Linux( i[56]86-pc-linux-gnu )
* Siege was originally developed on SuSE GNU/Linux with
gcc, there
are no known issues on this platform.
HP-UX ( hppa2.0w-hp-hpux11.00 )
* Siege has been built on this platform with both the HP
ANSI C
compiler and gcc. There are no known issues at this
time.
Solaris( sparc-sun-solaris2.[678] )
* Siege has been built successfully using gcc.
* The protocol bug on Solaris 2.x has been fixed.
OpenBSD( i[345]86-openbsd2.5 )
* Siege has been built successfully using gcc.
* Currently the compiler complains about struct
descrepancies and
BSD doesn't like mktemp. These problems should be
remedied soon.
Jeffrey Fulmer -- Designed and implemented Siege in
his
position as Webmaster for Armstrong World Industries.
http://www.armstrong.com
[ jdfulmer@armstrong.com ]
Please consult the file, COPYING for complete license information.
Copyright (C) 2000 Jeffrey Fulmer <jdfulmer@armstrong.com>
Permission is granted to anyone to make or distribute
verbatim
copies of this document as received, in any medium,
provided that
the copyright notice and this permission notice are
preserved, thus
giving the recipient permission to redistribute in turn.
Permission is granted to distribute modified versions of
this
document, or of portions of it, under the above
conditions,
provided also that they carry prominent notices stating
who last
changed them.