Yaws testsuites use the Common Test framework bundeled with the Erlang releases.
All testsuites are placed in the directory testsuite. A test suite is
implemented as an Erlang module named <suite_name>_SUITE.erl which contains a
number of test cases. A test case is an Erlang function that tests one or more
things.
To run it, you need to compile Yaws with the autotools. The command make check
should be used to run all tests.
Log files generated by tests execution are placed in the directory
testsuite/logs directory. Global information about the last run are placed in
the file testsuite/logs/yaws.log and a summary is printed on the standard
output.
To run all tests, you should run:
$> make checkThis command will execute all tests, which can be quite long. When you write a
new testsuite/testcase or for debugging purpose, it could be handy to execute
some testsuites, skipping others. This can be done by defining SUITES, GRPS
or CASES variables. For example:
$> make check SUITES="name1_SUITE name2_SUITE"
$> make check SUITES="name_SUITE" GRPS="group1 group2"
$> make check SUITES="name_SUITE" GRPS="group" CASES="testcase1 testcase2"Defining SUITES variable, you can choose to execute one or more
testsuites. You must use Erlang module name of the testsuites, without the
.erl extension. Testcases inside a testsuite can be grouped. This is not
mandatory, but when appropriate, you can be interested to not execute all
groups. To do so, you must define GRPS variable. In the same way, you can also
choose to not execute all testcases of a testsuite. This can be done by defining
CASES variable. To choose groups or testcases, the testsuite must be
defined. If testcases of a testsuite are grouped, the corresponding group must
be specified to choose testcases inside a testsuite. if SUITES variable is
undefined or contains more than one name, GRPS and CASES are ignored. And if
GRPS variable contains more than on name, CASES are ignored.
When the execution of the test suites proceeds, information are logged in three different way:
-
On the console, a summary about the execution is logged. You will find the status of each testcase (
OK,KOorSKIPPED), the execution result of each testsuite and finally the result of the run. -
in the file
testsuite/logs/yaws.log. You will find global information in this file. When a failure occurs, the first thing to do is to look inside. -
In the HTML report. All runs executed are listed in the file
testsuite/logs/all_runs.htmland direct links to all tests (the latest results) are written to the top-leveltestsuite/logs/index.html.
It is possible to enable code coverage analysis by setting USE_COVER variable
to yes:
$> make check USE_COVER=yesThe code coverage report is written in the file testsuite/logs/cover.html.
When you write a testsuite or when an error occurs, it could by handy to execute
tests step-by-step with the Erlang
Debugger. To do so, you
must set USE_DEBUGGER variable to yes. This enables the Debugger auto-attach
feature of the Common Test framework, which means that for every new interpreted
test case function that starts to execute, a new trace window automatically pops
up.
Another way to find a bug is to trace function calls. To do so, you must define
TRACES variable using the following format:
TRACES ::= "[PATTERN,...]"
PATTERN ::= Module | {Module, Function} | {Module, Function, Artiy}For example:
$> make check SUITES="auth_SUITE" TRACES="[{yaws_server, is_auth, 2}, {yaws_server, handle_auth}, yaws_outmod]"Traces are dumped in testsuites/testcases logs in the HTML reports. Tracing function calls can be expensive. So it important to reduce its scope as much as possible by selecting the rigth testsuite/testcase.
Here, we assume that you know Erlang and you are familiar with the Common Test framework.
Testsuites must be written in the directory testsuite and must be named using
the format <suite_name>_SUITE.erl. Data associated to this testsuite, if any,
must be placed in the directory testsuite/<suite_name>_SUITE_data. If you
need to use custom Erlang modules (appmod, runmod or anything else), it must be
placed directly in the testsuite date directory. These modules will be compiled
automatically. Template files (like the yaws configuration used by your
testsuite) that requires variables substitution (see
Template files and variables substitution)
must be placed in the directory
testsuite/<suite_name>_SUITE_data/templates. And finally, temporary files must
be placed in the directory testsuite/<suite_name>_SUITE_data/temp. Note this
directory will be automatically created and it will be removed with a make clean.
So here is a typical organisation for a testsuite:
testsuite
├── <suite_name>_SUITE.erl
└── <suite_name>_SUITE_data
├── temp # automatically created
│ └── yaws.conf # generated during the variables substitution
│ └── tmp_file1 # created by the testsuite
│ └── tmp_file2 # created by the testsuite
├── templates
│ └── yaws.conf
├── my_appmod.erl
├── other_module.erl
└── some_permanent_file1
└── some_permanent_file2
Here is a basic skeleton that can be used to write new testsuites. You can refine it, depending on you needs :
-module(suite_name_SUITE).
-include("testsuite.hrl").
-include_lib("kernel/include/file.hrl").
-compile(export_all).
%%====================================================================
%% Common test callback functions
%%====================================================================
%% For details about testcases and group declarations see Common Test
%% documentation.
%% (http://erlang.org/doc/apps/common_test/write_test_chapter.html)
%%--------------------------------------------------------------------
%% Function: all() -> GroupsAndTestCases | {skip,Reason}
%%
%% GroupsAndTestCases = [{group,GroupName} | TestCase]
%% GroupName = atom()
%% Name of a test case group.
%% TestCase = atom()
%% Name of a test case.
%% Reason = term()
%% The reason for skipping all groups and test cases.
%%
%% Description: Returns the list of groups and test cases that
%% are to be executed.
%%--------------------------------------------------------------------
all() ->
[my_test_case].
%%--------------------------------------------------------------------
%% Function: groups() -> [Group]
%%
%% Group = {GroupName,Properties,GroupsAndTestCases}
%% GroupName = atom()
%% The name of the group.
%% Properties = [Shuffle | {RepeatType,N}]
%% Group properties that may be combined.
%% GroupsAndTestCases = [Group | {group,GroupName} | TestCase]
%% TestCase = atom()
%% The name of a test case.
%% Shuffle = shuffle | {shuffle,Seed}
%% To get cases executed in random order.
%% Seed = {integer(),integer(),integer()}
%% RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail |
%% repeat_until_any_ok | repeat_until_any_fail
%% To get execution of cases repeated.
%% N = integer() | forever
%%
%% Description: Returns a list of test case group definitions.
%%--------------------------------------------------------------------
groups() ->
[].
%%--------------------------------------------------------------------
%% Function: init_per_suite(Config0) ->
%% Config1 | {skip,Reason} | {skip_and_save,Reason,Config1}
%%
%% Config0 = Config1 = [tuple()]
%% A list of key/value pairs, holding the test case configuration.
%% Reason = term()
%% The reason for skipping the suite.
%%
%% Description: Initialization before the suite.
%%
%% Note: This function is free to add any key/value pairs to the Config
%% variable, but should NOT alter/remove any existing entries.
%%--------------------------------------------------------------------
init_per_suite(Config) ->
Config.
%%--------------------------------------------------------------------
%% Function: end_per_suite(Config0) -> term() | {save_config,Config1}
%%
%% Config0 = Config1 = [tuple()]
%% A list of key/value pairs, holding the test case configuration.
%%
%% Description: Cleanup after the suite.
%%--------------------------------------------------------------------
end_per_suite(_Config) ->
ok.
%%--------------------------------------------------------------------
%% Function: init_per_group(GroupName, Config0) ->
%% Config1 | {skip,Reason} | {skip_and_save,Reason,Config1}
%%
%% GroupName = atom()
%% Name of the test case group that is about to run.
%% Config0 = Config1 = [tuple()]
%% A list of key/value pairs, holding configuration data for the group.
%% Reason = term()
%% The reason for skipping all test cases and subgroups in the group.
%%
%% Description: Initialization before each test case group.
%%--------------------------------------------------------------------
init_per_group(_Group, Config) ->
Config.
%%--------------------------------------------------------------------
%% Function: end_per_group(GroupName, Config0) ->
%% term() | {save_config,Config1}
%%
%% GroupName = atom()
%% Name of the test case group that is finished.
%% Config0 = Config1 = [tuple()]
%% A list of key/value pairs, holding configuration data for the group.
%%
%% Description: Cleanup after each test case group.
%%--------------------------------------------------------------------
end_per_group(_Group, _Config) ->
ok.
%%--------------------------------------------------------------------
%% Function: init_per_testcase(TestCase, Config0) ->
%% Config1 | {skip,Reason} | {skip_and_save,Reason,Config1}
%%
%% TestCase = atom()
%% Name of the test case that is about to run.
%% Config0 = Config1 = [tuple()]
%% A list of key/value pairs, holding the test case configuration.
%% Reason = term()
%% The reason for skipping the test case.
%%
%% Description: Initialization before each test case.
%%
%% Note: This function is free to add any key/value pairs to the Config
%% variable, but should NOT alter/remove any existing entries.
%%--------------------------------------------------------------------
init_per_testcase(_Test, Config) ->
Config.
%%--------------------------------------------------------------------
%% Function: end_per_testcase(TestCase, Config0) ->
%% term() | {save_config,Config1} | {fail,Reason}
%%
%% TestCase = atom()
%% Name of the test case that is finished.
%% Config0 = Config1 = [tuple()]
%% A list of key/value pairs, holding the test case configuration.
%% Reason = term()
%% The reason for failing the test case.
%%
%% Description: Cleanup after each test case.
%%--------------------------------------------------------------------
end_per_testcase(_Test, _Config) ->
ok.
%%====================================================================
%% Write your testcases here
%%====================================================================
%%--------------------------------------------------------------------
%% Function: TestCase() -> Info
%%
%% Info = [tuple()]
%% List of key/value pairs.
%%
%% Description: Test case info function - returns list of tuples to set
%% properties for the test case.
%%
%% Note: This function is only meant to be used to return a list of
%% values, not perform any other operations.
%%--------------------------------------------------------------------
my_test_case() ->
[].
%%--------------------------------------------------------------------
%% Function: TestCase(Config0) ->
%% ok | exit() | {skip,Reason} | {comment,Comment} |
%% {save_config,Config1} | {skip_and_save,Reason,Config1}
%%
%% Config0 = Config1 = [tuple()]
%% A list of key/value pairs, holding the test case configuration.
%% Reason = term()
%% The reason for skipping the test case.
%% Comment = term()
%% A comment about the test case that will be printed in the html log.
%%
%% Description: Test case function. (The name of it must be specified in
%% the all/0 list or in a test case group for the test case
%% to be executed).
%%--------------------------------------------------------------------
my_test_case(_Config) ->
ok.NOTE: For now, testcases cannot be run in parallel
When a testsuite starts, its temporary directory is created and template files
are parsed. When it ends, we stop and unload yaws application, if needed to
avoid any conflict with other testsuites.
Because many information depend on your configuration, like paths, it is
convenient to have templates where predefined variables will be substituted
during testsuite execution. This is especially true for yaws configurations. So
you can write template files in the directory
testsuite/<suite_name>_SUITE_data/templates. All files in this directory will
be parsed when the testsuite starts and all known variables will be
substituted. Generated files will be written in the directory
testsuite/<suite_name>_SUITE_data/temp, using the same filename.
Here is the list of substituted variables:
$top_srcdir$: The absolute path of the top-level source code directory.$top_builddir$: The absolute path of the top level of the current build tree.$srcdir$: The path$top_srcdir$/src.$ebindir$: The path$top_builddir$/ebin.$ts_srcdir$: The path$top_srcdir$/testsuite.$ts_builddir$: The path$top_builddir$/testsuite.$wwwdir$: The path$top_srcdir$/www.$ssldir$: The path$top_srcdir$/ssl.$sslkeyfile$: The path$top_srcdir$/ssl/yaws-key.pem.$sslcertfile$: The path$top_srcdir$/ssl/yaws-cert.pem.$data_srcdir$: The path$ts_srcdir$/<suite_name>_SUITE_data.$data_builddir$: The path$ts_builddir$/<suite_name>_SUITE_data.$templatedir$: The path$data_srcdir$/templates.$tempdir$: The path$data_builddir$/temp.$logdir$: The private direcotry of the testsuite. It is equivalent to?config(priv_dir, Config).$yaws_port[1-20]$: Reserved ports that can be used to configure Yaws.
All these variables are also available from your Erlang module using macros for
many of them. See testsuite/testsuite.hrl.in file for details. There are only
$logdir$ and $yaws_port[1-20]$ that require a function call:
- To get the log directory in your testsuite, you should use
?config(priv_dir, Config). - To get a reserved port number you should use the call
testsuite:get_yaws_config(N, Config).
Here is an example of a templatized Yaws configuration:
logdir = $logdir$
ebin_dir = $data_builddir$
trace = false
copy_error_log = true
log_wrap_size = 1000000
log_resolve_hostname = false
fail_on_bind_err = true
pick_first_virthost_on_nomatch = true
keepalive_timeout = 10000
<server localhost>
listen = 127.0.0.1
port = $yaws_port1$
deflate = true
auth_log = true
docroot = $data_srcdir$/www $tempdir$/www
</server>
<server localhost>
listen = 127.0.0.1
port = $yaws_port2$
docroot = $wwwdir$
<ssl>
keyfile = $sslkeyfile$
certfile = $sslcertfile$
depth = 0
</ssl>
</server>The module testsuite/testsuite.erl is used as a
Common Test Hook
module to instrument testsuite executions. But it also provides an API to help
you writing testcases.
create_dir(Dir) -> ok | {error, Reason} when
Dir :: file:name_all(),
Reason :: any().Ensures a directory exsit and creates it and all its parent, if necessary. If
the direcory exists or if it is successfully created, the function returns
ok. Otherwise it returns {error, Reason}.
get_yaws_port(N, Config) -> PortNum when
N :: pos_integer(), %% must be in the range [1, 20]
Config :: [tuple()], %% Provided by the Common Test framework
PortNum :: inet:port_number().Returns the reserved port corresponding to the number N.
add_yaws_server(Docroot, SL) -> {ok, SConf} when
Docroot :: string(),
SL :: [tuple()],
SConf :: #sconf{}.Adds a new server into the running instance of Yaws. This function returns the
computed configuration. This is equivalent to do yaws:add_server(Docroot, SL). To succeed, Yaws should have been previously started.
delete_yaws_server(SConf) -> ok when
SConf :: #sconf{}.Removes a server from the running instance of Yaws. The server configuration
must be used to find the right server. This is equivalent to do
yaws_config:delete_sconf(Sconf). To succeed, Yaws should have been previously
started.
reset_yaws_servers() -> ok | {error, Reason} when
Reason :: any().Removes all servers from the running instance of Yaws. This is equivalent to do
yaws_api:setconf(GConf, []), where the global configuration is retrieved
calling yaws_api:getconf/0. To succeed, Yaws should have been previously
started.
yaws_servers() -> [{IP, Port, VHost, SConf}] when
IP :: inet:ip_address(),
Port :: inet:port_number(),
VHost :: string(),
SConf :: #sconf{}.Retrieves all servers hosted by the running instance of Yaws. To succeed, Yaws should have been previously started.
make_url(Scheme, Host, Port, Path) -> Url when
Scheme :: http | https || string(),
Host :: string(),
Port :: inet:port_number(),
Path :: string(),
Url :: string().Builds an absolute URL from its parts.
receive_http_headers(Sock) -> {ok, Result} | {error, Reason} when
Sock :: inet:socket() | ssl:socket(),
Result :: {StatusLine, Headers},
StatusLine :: {Vsn, Code, ReasonPhrase},
Headers :: [{Name, Value}],
Vsn :: string(), %% "HTTP/X.Y",
Code :: integer(),
ReasonPhrase :: string(),
Name :: string(),
Value :: string(),
Reason :: any().Reads the status lust and all the headers of a HTTP response. To succeed, a request should have been previously sent.
receive_http_body(Sock, Length) -> {ok, Body} | {error, Reaosn} when
Sock :: inet:socket() | ssl:socket(),
Length :: undefined | pos_integer(),
Body :: binary(),
Reason :: any().Reads the all the body of the HTTP response. To succeed, a request should have
been previously sent. If Length is set to undefined, then the body is read
until the socket is closed. Else the Length value is supposed to be extracted
from the Content-Lengh header of the response. For chunked transfer encoding
body, use receive_http_chunked_body/1.
receive_chunked_http_body(Sock) -> {ok, Body} | {error, Reaosn} when
Sock :: inet:socket() | ssl:socket(),
Body :: Data | {Data, Trailers},
Data :: binary(),
Trailers :: [{Name, Value}],
Name :: string(),
Value :: string(),
Reason :: any().Reads the all chunks of the HTTP response body. The response is supposed to be chunked encoded. To succeed, a request should have been previously sent.
receive_http_response(Sock) -> receive_http_response(unknown, Sock, infinity).
receive_http_response(Meth, Sock) -> receive_http_response(Meth, Sock, infinity).
receive_http_response(Meth, Sock, Tout) -> {ok, Result} | {error, Reason} when
Sock :: inet:socket() | ssl:socket(),
Meth :: head | get | put | post | trace | options | delete | atom(),
Tout :: infinity | pos_integer(),
Result :: {StatusLine, Headers, Body},
StatusLine :: {Vsn, Code, ReasonPhrase},
Headers :: [{Name, Value}],
Body :: Data | {Data, Trailers},
Vsn :: string(), %% "HTTP/X.Y",
Code :: integer(),
ReasonPhrase :: string(),
Name :: string(),
Value :: string(),
Data :: binary(),
Trailers :: [{Name, Value}],
Reason :: any().Reads a full HTTP response. To succeed, a request should have been previously sent.
send_http_headers(Sock, Req, Headers) -> ok | {error, Reason} when
Sock :: inet:socket() | ssl:socket(),
Req :: {Meth, Path, Vsn} | string(),
Headers :: [{Name, Value}],
Meth :: head | get | put | post | trace | options | delete | atom(),
Path :: string(),
Vsn :: string(), %% "HTTP/X.Y"
Name :: string(),
Value :: string(),
Reason :: any().Sends the request line and all headers of a HTTP request.
send_http_body(Sock, Body) -> ok | {error, Reason} when
Sock :: inet:socket() | ssl:socket(),
Body :: iolist() | {chunks, iolist()} | {Process, Acc} | {chunkify, Process, Acc},
Process :: fun(Acc) -> eof | {ok, iolist(), Acc} | {error, Reason},
Acc :: any(),
Reason :: any().Sends the body of a HTTP request.
send_http_request(Sock, Req, Headers) -> send_http_request(Sock, Req, Headers, <<>>).
send_http_request(Sock, Req, Headers, Body) -> ok | {error, Reason} when
Sock :: inet:socket() | ssl:socket(),
Req :: {Meth, Path, Vsn} | string(),
Headers :: [{Name, Value}],
Body :: iolist() | {chunks, iolist()} | {Process, Acc} | {chunkify, Process, Acc},
Meth :: head | get | put | post | trace | options | delete | atom(),
Path :: string(),
Vsn :: string(), %% "HTTP/X.Y"
Name :: string(),
Value :: string(),
Process :: fun(Acc) -> eof | {ok, iolist(), Acc} | {error, Reason},
Acc :: any(),
Reason :: any().Sends a full HTTP request.
http_get(Url) -> http_get(Url, [], [], []).
http_get(Url, Headers) -> http_get(Url, Headers, [], []).
http_get(Url, Headers, HttpOpts) -> http_get(Url, Headers, HttpOpts, []).
http_get(Url, Headers, HttpOpts, SockOpts) -> {ok, Result} | {error, Reason} when
Url :: string(), %% "scheme://host:port/path?QS",
Headers :: [{Name, Value}],
HttpOpts :: [HttpOpt],
SockOpts :: gen_tcp:connect_option() | ssl:socket_option(),
Name :: string(),
Value :: string(),
HttpOpt :: {connect_timeout, pos_integer() | infinity}
| {timeout, pos_integer() | infinity}
| {proxy, {PHost, PPort}}
| {version, string()}, %% "HTTP/X.Y"
PHost :: string(),
PPort :: inet:port_number(),
Reason :: any().Sends a HTTP request using the GET Method, and returns the corrsponding HTTP
response.
http_post(Url, {CT, Body}) -> http_post(Url, [], {CT, Body}, [], []).
http_post(Url, Headers, {CT, Body}) -> http_post(Url, Headers, {CT, Body}, [], []).
http_post(Url, Headers, {CT, Body}, HttpOpts) -> http_post(Url, Headers, {CT, Body}, HttpOpts, []).
http_post(Url, Headers, {CT, Body}, HttpOpts, SockOpts) -> {ok, Result} | {error, Reason} when
Url :: string(), %% "scheme://host:port/path?QS",
Headers :: [{Name, Value}],
CT :: string(),
Body :: iolist() | {chunks, iolist()} | {Process, Acc} | {chunkify, Process, Acc},
HttpOpts :: [HttpOpt],
SockOpts :: gen_tcp:connect_option() | ssl:socket_option(),
Name :: string(),
Value :: string(),
Process :: fun(Acc) -> eof | {ok, iolist(), Acc} | {error, Reason},
Acc :: any(),
HttpOpt :: {connect_timeout, pos_integer() | infinity}
| {timeout, pos_integer() | infinity}
| {proxy, {PHost, PPort}}
| {version, string()}, %% "HTTP/X.Y"
PHost :: string(),
PPort :: inet:port_number(),
Reason :: any().Sends a HTTP request using the POST Method, and returns the corrsponding HTTP
response.
http_req(Meth, Url) -> http_req(Meth, Url, [], <<>>, [], []).
http_req(Meth, Url, Headers) -> http_req(Meth, Url, Headers, <<>>, [], []).
http_req(Meth, Url, Headers, Body) -> http_req(Meth, Url, Headers, Body, [], []).
http_req(Meth, Url, Headers, Body, HttpOpts) -> http_req(Meth, Url, Headers, Body, HttpOpts, []).
http_req(Meth, Url, Headers, Body, HttpOpts, SockOpts) -> {ok, Result} | {error, Reason} when
Meth :: head | get | put | post | trace | options | delete | atom(),
Url :: string(), %% "scheme://host:port/path?QS",
Headers :: [{Name, Value}],
Body :: iolist() | {CT, Data},
HttpOpts :: [HttpOpt],
SockOpts :: gen_tcp:connect_option() | ssl:socket_option(),
Name :: string(),
Value :: string(),
CT :: string(),
Data :: iolist() | {chunks, iolist()} | {Process, Acc} | {chunkify, Process, Acc},
Process :: fun(Acc) -> eof | {ok, iolist(), Acc} | {error, Reason},
Acc :: any(),
HttpOpt :: {connect_timeout, pos_integer() | infinity}
| {timeout, pos_integer() | infinity}
| {proxy, {PHost, PPort}}
| {version, string()}, %% "HTTP/X.Y"
PHost :: string(),
PPort :: inet:port_number(),
Reason :: any().
Sends a HTTP request and returns the corresponding HTTP response.
post_file(Acc) -> Result when
Acc :: {File, MaxSz},
Result :: eof | {ok, iolist(), Acc} | {error, Reason},
File :: file:name_all(),
MaxSz :: non_neg_integer(),
Reason :: any().A body processor function that could be used in previous functions to send a file as the request body.