http://tsung.erlang-projects.org/

Tsung User's manual

Version: 693
Date : 7th October, 2006

Tsung user's manual

Contents

1  Introduction

1.1  What is Tsung ?

Tsung (formerly IDX-Tsunami) is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, SOAP PostgreSQL, and Jabber servers.

It is distributed under the GNU General Public License version 2.

1.2  What is Erlang and why is it important for Tsung ?

Tsung's main strength is its ability to simulate a huge number of simultaneous user from a single CPU. When used on cluster you can generate a really impressive load on a server with a modest cluster, easy to set-up and to maintain.

Tsung is developed in Erlang and this is where the power of Tsung resides.

Erlang is a concurrency-oriented programming language. Tsung is based on the Erlang OTP (Open Transaction Platform) and inherits several characteristics from Erlang: More informations on Erlang on http://www.erlang.org and http://www.erlang-projects.org/

1.3  Tsung background

History: Tsung has been used for very high load tests: Tsung has been used at:

2  Features

2.1  Tsung main features

2.2  HTTP related features

2.3  Jabber related features

2.4  PostgreSQL related features

Warning: This pluging is still experimental.

2.5  Complete reports set

Measures and statistics produced by Tsung are extremely feature-full. They are all represented as a graphic. Tsung produces statistics regarding: Note that Tsung takes care of the synchronization process by itself. Gathered statistics are «synchronized».

It is possible to generate graphs during the benchmark as statistics are gathered in real-time.

2.6  Highlights

Tsung has several advantages over other injection tools:

3  Installation

This package has been tested on Linux, FreeBSD and Solaris. It should work on Erlang supported platforms (Linux, Solaris, *BSD, Win32 and MacOS-X).

3.1  Dependencies

3.2  Compilation


 
./configure
 make
 make install

3.3  Configuration

The default configuration file is ~/.tsung/tsung.xml ( there are several sample files in /usr/share/doc/tsung/examples).

Log files are saved in ~/.tsung/log/ . A new subdirectory is created for each test using the current date as name (~/.tsung/log/20040217-09:40 for ex.)

3.4  Feedback

Use the Tsung mailing list (see https://lists.process-one.net/mailman/listinfo/tsung-users) if you have suggestions or questions about Tsung.

4  HTTP benchmark approach

4.1  benchmarking a Web server

  1. Record one or more sessions: start the recorder with: tsung recorder, and then configure your browser to use Tsung proxy recorder (the listen port is 8090). A session file will be created. For HTTPS recording, use http://{ instead of https:// in your browser.
  2. Edit / organize scenario, by adding recorded sessions in the configuration file.
  3. Write small code for dynamic parts if needed and place dynamic mark-up in the scenario.
  4. Test and adjust scenario to have a nice progression of the load. This is highly dependent of the application and of the size of the target server(s). Calculate the normal duration of the scenario and use the interarrival time between users and the duration of the phase to estimate the number of simultaneous users for each given phase.
  5. Launch benchmark with your first application parameters set-up: tsung start
  6. Wait for the end of the test or stop by hand with tsung stop (reports can also be generated during the test (see § 9) : the statistics are updated every 10 seconds). For a brief summary of the current activity, use tsung status
  7. Analyze results, change parameters and relaunch another benchmark

4.2  benchmarking a proxy server

By default, the HTTP plugin is used to benchmark HTTP servers. But you can also benchmark HTTP Proxy servers. To do that, you must add in the options section:
 
  <option type="ts_http" name="http_use_server_as_proxy" value="true"></option>

5  PostgreSQL benchmark approach

It's the same approach as HTTP: first you start to record one or more sessions with the recorder: tsung -p pgsql recorder

This will start a proxy listening to port 8090 and will proxy requests to 127.0.0.0:5432.

To choose another port and/or address: tsung -L 5432 -I 10.6.1.1 -P 5433 -p pgsql recorder

This will start a proxy listening to port 5432 and will proxy requests to 10.6.1.1:5433.

6  Jabber benchmark approach

This paragraph explains how to write a session for Jabber.

There are two differences between HTTP and Jabber testing:
  1. There is no recorder for Jabber, so you have to write your sessions by hand (an example is provided in 8.6.2).
  2. the jabber plugin does not parse XML; instead it uses packet acknowledgments.

6.1  Acknowledgements of messages

Since the jabber plugin does not parse XML (historically, it was for performance reasons), you must have a way to tell when a request is finished. There are 3 possibilities:
ack=local
as soon as a packet is received from the server, the request is considered as completed. Hence if you use a local ack with a request that do not require a response from the server (presence for ex.), it will wait forever (or until a timeout is reached).
ack=no_ack
as soon as the request is send, it is considered as completed (do not wait for incoming data)
ack=global
synchronized users. its main use is for waiting for all users to connect before sending messages. To do that, set a request with global ack (it can be the first presence msg:
 
   <request> <jabber type="presence" ack="global"/> </request>
)

You also have to specify the number of users to be connected:
 
<option type="ts_jabber" name="global_number" value="100"></option>

To be sure that exactly global_number users are started, add the 'maxnumber' attribute to 'users'
 
    <users maxnumber="100" interarrival="1.0" unit="second"></users>

If you do not specify maxnumber, the global ack will be reset every global_number users

6.2  Status: Offline, Connected and Online

You can send messages to offline or online users. A user is considered online when he has send a presence:initial message (before this message , the state of the user is connected).

If you want to switch back to connected before going offline, you can use a presence:final message:

presence:final does two things:
  1. It removes the client from the list of Online users, and moves them into the list of Connected users.
  2. It sends a broadcast presence update of type='unavailable'.
presence:final is optional.

warn: this is new in 1.2.0, in earlier version, only 2 status were available: online and offline; a user was considered online as soon as it was connected.

6.3  Authentication

Below are configuration examples for the possible authentication methods. Note: the regular expressions used here are only examples - they may need to be altered depending on how a particular server implementation composes messages (see also  8.5.1 for password settings).

7  Using the proxy recorder

The recorder has two plugins: one for HTTP and one for PostgreSQL.

To start it, tsung -p PLUGIN recorder. http is the default plugin. For PostgreSQL, use pgsql.

The proxy is listening to port 8090. The recorded session is created as ~/.tsung/tsung_recorderYYYMMDD-HH:MM.xml; if it doesn't work, take a look at ~/.tsung/log/tsung.log-tsung_recorder@hostname

To stop it, use tsung stop_recorder.

Once a session has been created, you can insert it in your main config file, either by editing by hand the file, or by using an ENTITY declaration, like:
 
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [
 <!ENTITY mysession1 SYSTEM "/home/nniclausse/.tsung/tsung_recorder20051217-13:11.xml">
]>
...
<sessions>
  &mysession1;
</sessions>

8  Understanding tsung.xml configuration file

The default encoding is utf-8. You can use a different encoding, like in:
 
<?xml version="1.0" encoding="ISO-8859-1"?>

8.1  File structure

Scenarios are enclosed into Tsung tags:
 
<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [] >
<tsung loglevel="info">
...
</tsung>

If you add the attribute dumptraffic="true", all the traffic will be logged to a file. Warn: this will considerably slow down Tsung, so use with care. It is useful for debugging purpose.

The loglevel can also have a great impact on performance: For high load, warning is recommanded . Possible values are: For REALLY verbose logging, recompile tsung with make debug and set loglevel to debug.

8.2  Clients and server

Scenarios start with clients (Tsung cluster) and server definitions:

For non distributed load, you can use a basic setup like:
 
  <clients>
     <client host="localhost" use_controller_vm="true"/>
  </clients>

<servers>
  <server host="192.168.1.1" port="80" type="tcp"></server>
</servers>

This will start the load on the same host and on the same Erlang virtual machine as the controller.

The server is the entry point into the cluster (New in 1.2.0: if several servers are defined, a round robin algorithm is used to choose the server).

The next example is a more complex, and use several features for advanced distributed testing:
 
  <clients>
     <client host="louxor" weight="1" maxusers="800">
         <ip value="10.9.195.12"></ip>
         <ip value="10.9.195.13"></ip>
     </client>
     <client host="memphis" weight="3" maxusers="600" cpu="2">
         <ip value="10.9.195.14"></ip>
     </client>
  </clients>

<servers>
  <server host="10.9.195.1" port="8080" type="tcp"></server>
</servers>

Several virtual IP can be used to simulate more machines. This is very useful when a load-balancer use the client's IP to distribute the traffic among a cluster of servers. New in 1.1.1: IP is no longer mandatory. If not specified, the default IP will be used.

In this example, a second machine is used in the Tsung cluster, with a higher weight, and 2 cpus. Two Erlang virtual machines will be used to take advantage of the number of CPU.

By default, the load is distributed uniformly on all CPU (one cpu per client by default). The weight parameter (integer) can be used to take into account the speed of the client machine. For instance, if one real client has a weight of 1 and the other client has a weight of 2, the second one will start twice the number of users as the first (the proportions will be 1/3 and 2/3). In the earlier example where for the second client has 2 CPU and weight=3, the weight is equal to 1.5 for each CPU.

8.3  Monitoring

Scenarios can contain optional monitoring informations. For example, here is a cluster monitoring definition based on Erlang agents, for a cluster of 6 computers:
 
  <monitoring>
    <monitor host="geronimo" type="erlang"></monitor>
    <monitor host="bigfoot-1" type="erlang"></monitor>
    <monitor host="bigfoot-2" type="erlang"></monitor>
    <monitor host="f14-1" type="erlang"></monitor>
    <monitor host="f14-2" type="erlang"></monitor>
    <monitor host="db" type="erlang"></monitor>
  </monitoring>

The type keyword snmp can replace the erlang keyword, if SNMP monitoring is preferred. They can be mixed. erlang is the default value for monitoring.

Note: For Erlang monitoring, monitored computers need to be accessible through the network. SSH (or rsh) needs to be configured to allow connection without password on. You must use the same version of Erlang/OTP on all nodes otherwise it may not work properly !

8.4  Defining the load progression

The load progression is set-up by defining several arrival phases:
 
 <load>
  <arrivalphase phase="1" duration="10" unit="minute">
    <users interarrival="2" unit="second"></users>
  </arrivalphase>

  <arrivalphase phase="2" duration="10" unit="minute">
    <users interarrival="1" unit="second"></users>
  </arrivalphase>

  <arrivalphase phase="3" duration="10" unit="minute">
    <users interarrival="0.1" unit="second"></users>
  </arrivalphase>
 </load>

With this setup, during the first 10 minutes of the test, a new user will be created every 2 seconds, then during the next 10 minutes, a new user will be created every second, and for the last 10 minutes, 10 users will be generated every second. The test will finish when all users have ended their session.

The load generated in terms of HTTP requests / seconds will also depend on the mean number of requests within a session (if you have a mean value of 100 requests per session and 10 new users per seconds, the theoretical average throughput will be 1000 requests/ sec).

8.5  Setting options

Default values can be set-up globally: thinktime between requests in the scenario and ssl cipher algorithms. These values overrides those set in session configuration tags if override is true. 
  <option name="thinktime" value="3" random="false" override="true"/>
  <option name="ssl_ciphers" 
           value="EXP1024-RC4-SHA,EDH-RSA-DES-CBC3-SHA"/>

8.5.1  Jabber options

Default values for specific protocols can be defined. Here is an example of option values for Jabber:
 
  <option type="ts_jabber" name="global_number" value="5" />
  <option type="ts_jabber" name="userid_max" value="100" />
  <option type="ts_jabber" name="domain" value="jabber.org" />
  <option type="ts_jabber" name="username" value="myuser" />
  <option type="ts_jabber" name="passwd" value="mypasswd" />

Using these values, users will be myuserXXX where XXX is an integer in the interval [1:userid_max] and passwd mypasswdXXX

If not set in the configuration file, the values will be set to:

8.5.2  HTTP options

For HTTP, you can set the UserAgent values (available since Tsung 1.1.0), using a probability for each value (the sum of all probabilities must be equal to 100)
 
  <option type="ts_http" name="user_agent">
    <user_agent probability="80">
       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Galeon/1.3.21
    </user_agent>
    <user_agent probability="20">
      Mozilla/5.0 (Windows; U; Windows NT 5.2; fr-FR; rv:1.7.8) Gecko/20050511 Firefox/1.0.4
    </user_agent>
  </option>

8.6  Sessions

Sessions define the content of the scenario itself. They describe the requests to execute.

Each session has a given probability. This is used to decide which session a new user will execute. The sum of all session's probabilities must be 100.

A transaction is just a way to have customized statistics. Say if you want to know the response time of the login page of your website, you just have to put all the requests of this page (HTML + embedded pictures) within a transaction. In the example above, the transaction called index_request will gives you in the statistics/reports the mean response time to get index.en.html + header.gif. Be warn that If you have a thinktime inside the transaction, the thinktime will be part of the response time.

8.6.1  HTTP

This example shows several features of the HTTP protocol support in Tsung: GET and POST request, basic authentication, transaction for statistics definition, ...
 
<sessions>
  <session name="http-example" probability="70" type="ts_http">

    <request> <http url="/" method="GET" version="1.1">
                    </http> </request>
    <request> <http url="/images/logo.gif"
               method="GET" version="1.1" 
               if_modified_since="Fri, 14 Nov 2003 02:43:31 GMT">
              </http></request>

    <thinktime value="20" random="true"></thinktime>

    <transaction name="index_request">
     <request><http url="/index.en.html"
                          method="GET" version="1.1" >
              </http> </request>
     <request><http url="/images/header.gif"
                          method="GET" version="1.1">
              </http> </request>
    </transaction>

    <thinktime value="60" random="true"></thinktime>
    <request>
      <http url="/" method="POST" version="1.1"
               contents="bla=blu">
      </http> </request>
    <request>
       <http url="/bla" method="GET" version="1.1"
             contents="bla=blu&name=glop">
       <www_authenticate userid="Aladdin"
                         passwd="open sesame"/></http>
    </request>
  </session>

  <session name="backoffice" probability="30" ...>
  ... </session>
</sessions>

8.6.2  Jabber

Here is an example of a session definition for the Jabber protocol: 
<sessions>
 <session probability="70" name="jabber-example" type="ts_jabber">

    <request> <jabber type="connect" ack="no_ack" /> </request>

    <thinktime value="2"></thinktime>

    <transaction name="authenticate">
      <request> <jabber type="authenticate" ack="local"> </jabber> </request>
    </transaction>

    <request> <jabber type="presence:initial" ack="no_ack"/> </request>


    <thinktime value="30"></thinktime>

    <transaction name="online">
    <request> <jabber type="chat" ack="no_ack" size="16" destination="online"/></request>
    </transaction>
    <thinktime value="30"></thinktime>

    <transaction name="offline">
      <request> <jabber type="chat" ack="no_ack" size="56" destination="offline"/><request>
    </transaction>

    <thinktime value="30"></thinktime>

    <transaction name="close">
      <request> <jabber type="close" ack="local"> </jabber></request>
    </transaction>
  </session>
</sessions>

Roster
What you can do with rosters using Tsung:

You can
  1. Add a new contact to their roster - The new contact is added to the Tsung Group group, and their name matches their JID
  2. Send a subscribe presence notification to the new contact's JID - This results in a pending subscription
  3. Rename a roster contact This changes the previously added contact's name from the default JID, to Tsung Testuser
  4. Delete the previously added contact.
Note that when you add a new contact, the contact JID is stored and used for the operations that follow. It is recommended that for each session which is configured to perform these operations, only do so once. In other words, you would NOT want to ADD more than one new contact per session. If you want to alter the rate that these roster functions are used during your test, it is best to use the session 'probablity' factor to shape this.

The nice thing about this is that when you test run is complete, your roster tables should look the same as before you started the test. So, if you set it up properly, you can have pre-loaded roster entries before the test, and then use these methods to dynamically add, modify, and remove roster entries during the test as well.

Example roster modification setup:
 
<session probability="100" name="jabber-rostermod" type="ts_jabber">

    <!-- connect, authenticate, roster 'get', etc... -->

    <transaction name="rosteradd">
      <request> <jabber type="iq:roster:add" ack="no_ack" destination="online"></jabber> </request>
      <request> <jabber type="presence:subscribe" ack="no_ack"/> </request>
    </transaction>

    <!-- ... -->

    <transaction name="rosterrename">
      <request> <jabber type="iq:roster:rename" ack="no_ack"></jabber> </request>
    </transaction>

    <!-- ... -->

    <transaction name="rosterdelete">
      <request> <jabber type="iq:roster:remove" ack="no_ack"></jabber> </request>
    </transaction>

    <!-- remainder of session... -->

  </session>

Presence
For more info, see section 2.2 of RFC 3921.

If you omit the show or status attributes, they default to chat and Available respectively.

Example of broadcast presence (broadcast to members of your roster): 
    <request> <jabber type="presence:broadcast" show="away" status="Be right back..." ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:broadcast" show="chat" status="Available to chat" ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:broadcast" show="dnd" status="Don't bother me!" ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:broadcast" show="xa" status="I may never come back..." ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:broadcast" ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

Example of directed presence (sent to random online users): 
    <request> <jabber type="presence:directed" show="away" status="Be right back..." ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:directed" show="chat" status="Available to chat" ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:directed" show="dnd" status="Don't bother me!" ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:directed" show="xa" status="I may never come back..." ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

    <request> <jabber type="presence:directed" ack="no_ack"/> </request>
    <thinktime value="5"></thinktime>

8.6.3  PostgreSQL

For postgresql, 4 types of requests are available:
  1. connect (to a given database with a given username
  2. authenticate (with password or not)
  3. sql
  4. close
This example shows most of the features of a PostgreSQL session:
 
  <session probability="100" name="pgsql-example" type="ts_pgsql">
    <transaction name="connection">
      <request>
        <pgsql type="connect" database="bench" username="bench" />
      </request>
    </transaction>

    <request><pgsql type="authenticate" password="sesame"/></request>

    <thinktime value="12"/>

    <request><pgsql type="sql">SELECT * from accounts;</pgsql></request>

    <thinktime value="20"/>

    <request><pgsql type="sql">SELECT * from users;</pgsql></request>

    <request><pgsql type='sql'><![CDATA[SELECT n.nspname as "Schema",
  c.relname as "Name",
  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN '%_toto_% END as "Type",
  u.usename as "Owner"
FROM pg_catalog.pg_class c
     LEFT JOIN pg_catalog.pg_user u ON u.usesysid = c.relowner
     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind IN ('r','v','S','')
      AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
      AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 1,2;]]></pgsql></request>

    <request><pgsql type="close"></pgsql></request>

  </session>

8.6.4  Dynamic substitutions

Dynamic substitution are mark-up placed in element of the scenario. For HTTP, this mark-up can be placed in basic authentication (www_authenticate tag: userid and passwd attributes), URL (to change GET parameter) and POST content.

Those mark-up are of the form %%Module:Function%%. Substitutions are executed on a request-by-request basis, only if the request tag has the attribute subst="true".

When a substitution is requested, the substitution mark-up is replaced by the result of the call to the Erlang function: Module:Function(Pid, DynData) where Pid is the Erlang process id of the current virtual user and DynData the list of all Dynamic variables (Warn: before version 1.1.0, the argument was just the Pid !).

Here is an example of use of substitution in a Tsung scenario:
 
<session name="rec20040316-08:47" probability="100" type="ts_http">
 <request subst="true">
  <http url="/echo?symbol=%%symbol:new%%" method="GET">
  </http></request>
</session>

Here is the Erlang code of the module used for dynamic substitution:
 
-module(symbol).
-export([new/1]).

new({Pid, DynData}) ->
    case random:uniform(3) of
        1 -> "IBM";
        2 -> "MSFT";
        3 -> "RHAT"
    end.

(use erlc to compiled the code, and put the resulting .beam file in \$PREFIX/lib/erlang/lib/tsung-X.X.X/ebin/ on all client machines)

As you can see, writing scenario with dynamic substitution is trivial.

If you want to set unique id, you can use the built-in function ts_user_server:get_unique_id. 
<session name="rec20040316-08:47" probability="100" type="ts_http">
 <request subst="true">
  <http url="/echo?id=%%ts_user_server:get_unique_id%%" method="GET">
  </http></request>
</session>

8.6.5  Reading external file

New in 1.0.3: A new module ts_file_server is available. You can use it to read external files. For example, if you need to read user names and passwd from a CSV file, you can do it with it (currently, you can read only a single file). Note: Reading usernames and password from a CSV file is not possible for Jabber.

You have to add this in the xml configuration file: 
 <option name="file_server"  value="/tmp/userlist.csv"></option>

Now you can build you own function to use it, for example:
 
-module(readcsv).
-export([user/1]).

user(Pid)->
    {ok,Line} = ts_file_server:get_next_line(), 
    [Username, Passwd] = string:tokens(Line,";"),
    "username=" ++ Username ++"&amp;passwd=" ++ Passwd.

In your session, use something like: 
  <request subst="true">
    <http url='/login.cgi' version='1.0' contents='%%readcsv:user%%&amp;op=login'
    content_type='application/x-www-form-urlencoded' method='POST'>
    </http>
  </request>

Two functions are available: ts_file_server:get_next_line and ts_file_server:get_random_line. For the get_next_line function, when the end of file is reached, the first line of the file will be the next line.

8.6.6  Dynamic variables

In some cases, you may want to use a value given by the server in a response later in the session, and this value is dynamically generated by the server for each user. For this, you can use <dyn_variable> in the scenario

Let's take an example with HTTP. You can easily grab a value in a HTML form like: 
<form action="go.cgi" method="POST">
<hidden name="random_num" value="42"></form>
</form>

with: 
 <request>
   <http url="/testtsung.html" method="GET" version="1.0"></http>
   <dyn_variable name="random_num" ></dyn_variable>
 </request>

Now random_num will be set to 42 during the user's session. It's value will be replace in all mark-up of the form %%_random_num%% if and only if the request tag has the attribute subst="true", like:
 
    <request subst="true">
      <http url='/go.cgi' version='1.0' 
      contents='username=nic&amp;random_num=%%_random_num%%&amp;op=login' 
      content_type='application/x-www-form-urlencoded' method='POST'>
      </http>
    </request>
If the dynamic value is not a form variable, you can set a regexp by hand, for example to get the title of a HTML page: 
    <request>
      <http url="/testtsung.html" method="GET" version="1.0"></http>
      <dyn_variable name="mytitlevar" 
                    regexp="&lt;title&gt;\(.*\)&lt;/title&gt;"/>
    </request>

8.6.7  Checking the server's response

With the tag match in a request tag, you can check the server's response against a given string, and do some actions depending on the result. In any case, if it matches, this will increment the match counter, if it does not match, the nomatch counter will be incremented.

For example, let's say you want to test a login page. If the login is ok, the server will respond with Welcome ! in the HTML body, otherwise not. To check that: 
 <request>
      <match do="continue" when="match">Welcome !</match>
      <http url='/login.php' version='1.0' method='POST' 
       contents='username=nic&amp;user_password=sesame'
       content_type='application/x-www-form-urlencoded' >
 </request>

You can use a regexp instead of a simple string.

The list of available actions to do is: You can mixed several match tag in a single request: 
 <request>
      <match do="loop" sleep_loop="5" max_loop="10" when="match">Retry</match>
      <match do="abort" when="match">Error</match>
      <http url='/index.php' method=GET'>
 </request>

You can also do the action on "nomatch" instead of "match".

9  Statistics and reports

9.1  Available stats

The mean response time (for requests, page, etc.) is computed every 10 sec (and reset). That's why you have the highest mean and lowest mean values in the Stats report (the mean for the whole test is not computed, since it's not very meaningful if you have several phases with different input rates in your setup).

HTTP specific stats:

9.2  Design

A bit of explanation on the design and internals of the statistics engine:

Tsung was designed to handle thousands of requests/sec, for very long period of times (several hours) so it do not write all data to the disk (for performance reasons). Instead it computes on the fly an estimation of the mean and standard variation for each type of data, and writes these estimations every 10 seconds to the disk (and then starts a new estimation for the next 10 sec). These computations are done for two kinds of data: There are also two other types of useful data (no averaging is done for those) :

9.3  Generating the report

cd to the log directory of your test (say ~/.tsung/log/20040325-16:33/) and use the script tsung_stats.pl:
 
/usr/lib/tsung/bin/tsung_stats.pl

(use –help to view all available options)

9.4  tsung summary

images/tsung-report.png


Figure 1: Report


9.5  Graphical overview

images/tsung-graph.png


Figure 2: Graphical output


10  References

11  Acknowledgments

The first version of this document was based on a talk given by Mickael Rémond4 during an Object Web benchmarking workshop in April 2004 (more info at http://jmob.objectweb.org/).

A  Frequently Asked Questions

A.1  Tsung crashes when I start it

Does your Erlang system has ssl support enabled ?

to test it: 
  > erl
  Eshell V5.2  (abort with ^G)
  1> ssl:start().
  you should see 'ok'

A.2  Tsung still doesn't start ...

Most of the time, when a crash happened at startup without any traffic generated, the problem arise because the main Erlang controller node cannot create a "slave" Erlang virtual machine. The message looks like:
 
===============================================
=ERROR REPORT==== 4-May-2004::22:38:26 ===
** Generic server ts_config_server terminating
** Last message in was {'$gen_cast',{newbeam,myshortname,[]}}
** When Server state == {state,{config,
                                     undefined,
                                     5,
                                     full,
                                     undefined,
                                     [{client,
                                          "myshortname",
                                          2.00000,
                                          5,
                                          [{10,68,133,140}]}],
                                     {server,"foo.net",80,gen_tcp},
                                     [],
                                     [{arrivalphase,
                                          1,
                                          60,
                                          undefined,
                                          undefined,
                                          5.00000e-5,
                                          infinity}],
                                     undefined,
                                     [{session,
                                          1,
                                          100,
                                          ts_http,
                                          parse,
                                          true,
                                          undefined}],
                                     14,
                                     3,
                                     7,
                                     6,
                                     "negociate"},
  "/home/username/.tsung/log/20040204-18:32",
                               undefined,
                               0,
                               undefined,
                               2.00000}
** Reason for termination ==
** {{badmatch,{error,timeout}},
    [{ts_config_server,handle_cast,2},
     {gen_server,handle_msg,6},
     {proc_lib,init_p,5}]}

Tsung launches a new erl virtual machine to do the actual injection even when you have only one machine in the injection cluster. This is because it needs to by-pass some limit with the number of open socket from a single process (1024 most of the time). The idea is to have several system processes (Erl beam) that can handle only a small part of the network connection from the given computer. When the maxclient limit (simultaneous) is reach, a new Erlang beam is launched and the newest connection can be handled by the new beam).

The problem is that the Erlang slave module cannot start a local slave node. It tries to start it with the short node name "myshortname" (erl -sname myshortname). If this fails the injection process cannot start. Most of the time, adding the short name with the correct IP address in the /etc/hosts file is sufficient to make it work.

You can test this using these simple commands: 
>erl -rsh ssh -sname foo -setcookie mycookie

Eshell V5.4.3 (abort with ^G) 
(foo@myhostname)1>slave:start(remotehost,bar,"-setcookie mycookie").
{ok,bar@remotehost}

Note that you do not need to use the 127.0.0.1 address in the configuration file. It will not work if you use it as the injection interface. The shortname of your client machine should not refer to this address.

New in 1.1.0: If you don't use the distributed feature of Tsung and have trouble to start a remote beam on a local machine, you can set the 'use_controller_vm' attribute to true, for ex.:
 
  <client host="mymachine" use_controller_vm="true">

You may also have problems due to the sshd configuration:

For example, for SuSE 9.2 sshd is compiled with restricted set of paths (ie. when you shell into the account you get the users shell, when you execute a command via ssh you don't) and this makes it impossible to start an erlang node (if erlang is installed in /usr/local for example).

Run:
 
ssh myhostname erl

If the erlang shell doesn't start then check what paths sshd was compiled with (in SuSE see /etc/ssh/sshd_config) and symlink from one of the approved paths to the erlang executable (thanks to Gordon Guthrie for reporting this).

A.3  Tsung still crashes/fails when I start it !

First look at the log file ~/.tsung/log/XXX/tsung_controller@yourhostname' to see if there is a problem.

If the file is not created and a crashed dump file is present, maybe you are using a binary installation of Tsung not compatible with the version of erlang you used.

If you see nothing wrong, you can compile Tsung with full debugging: recompile with make debug , and don't forget to set the loglevel to "debug" in the XML file.

To start the debugger or see what happen, start tsung with the debug argument instead of start. You will have an erlang shell on the tsung_controller node. Use toolbar:start(). to launch the graphical tools provided by Erlang.

A.4  What is the format of the stats file tsung.log ?


 
# stats: dump at 1083694995
stats: users 11 11
stats: request 41 1.03289 0.125108 1.59802 0.901978
stats: connect 41 0.220170 6.67110e-2 0.494019 0.171997
stats: users_count 11 11
stats: page 24 6.80416 17.2794 80.4609 0.958984
stats: size 26818 26818
stats: 404 7 7
stats: 200 20 20
# stats: dump at 1083695005
stats: users 21 21
stats: request 113 1.03980 0.110650 1.59802 0.791016
stats: connect 118 0.197619 4.26037e-2 0.494019 0.163940
stats: users_count 10 21
stats: page 52 2.72266 1.74204 80.4609 0.791016
stats: size 78060 104878
stats: 404 15 22
stats: 200 51 71
 ...

the format is, for request, page, session:

# stats:'name' count(during the last 10sec), mean, stdvar, max, min

or for HTTP returns code, size ...

# stats:'name' count(during the last 10sec), totalcount(since the beginning)

A.5  How can I specify the number of concurrent users ?

You can't. But it's on purpose: the load generated by Tsung is dependent on the arrival time between new clients. Indeed, once a client has finished his session in tsung, it stops. So the number of concurrent users is a function of the arrival rate and the mean session duration.

For example, if your web site has 1000 visits/hour, the arrival rate is 1000/3600 = 0.2778 visits/second. If you want to simulate the same load, set the inter-arrival time is to 1/0.27778 = 3.6 sec (<users interarrival="3.6" unit="second"> in the arrivalphase node in the XML config file).

A.6  SNMP monitoring doesn't work ?!

It use SNMP v1 and the 'public' community. It has been tested with http://net-snmp.sourceforge.net/.

You can try with snmpwalk to see if your snmpd config is ok:
 
>snmpwalk -v 1 -c public IP-OF-YOUR-SERVER .1.3.6.1.4.1.2021.4.5.0
UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 1033436

SNMP doesn't work with erlang R10B and Tsung older than 1.2.0.

There is a small bug in the snmp\_mgr module in old Erlang release (R9C-0). You have to apply this patch to make it work. This is fixed in erlang R9C-1 and up.
 
--- lib/snmp-3.4/src/snmp_mgr.erl.orig  2004-03-22 15:21:59.000000000 +0100
+++ lib/snmp-3.4/src/snmp_mgr.erl       2004-03-22 15:23:46.000000000 +0100
@@ -296,6 +296,10 @@
     end;
 is_options_ok([{recbuf,Sz}|Opts]) when 0 < Sz, Sz =< 65535 ->
     is_options_ok(Opts);
+is_options_ok([{receive_type, msg}|Opts]) ->
+    is_options_ok(Opts);
+is_options_ok([{receive_type, pdu}|Opts]) ->
+    is_options_ok(Opts);
 is_options_ok([InvOpt|_]) ->
     {error,{invalid_option,InvOpt}};
 is_options_ok([]) -> true.

1
http://www.erlang-projects.org/Members/mremond/events/dossier_de_presentat/block_10766817551485/file
2
http://www.editions-eyrolles.com/php.accueil/Ouvrages/ouvrage.php3?ouv_ean13=9782212110791
3
http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf
4
mickael.remond@erlang-fr.org

Copyright 2004-2006 © Nicolas Niclausse

This document was translated from LATEX by HEVEA.