runn
( means "Run N". is pronounced /rʌ́n én/. ) is a package/tool for running operations following a scenario.
Key features of runn
are:
- As a tool for scenario based testing.
- As a test helper package for the Go language.
- As a tool for workflow automation.
- Support HTTP request, gRPC request, DB query, Chrome DevTools Protocol, and SSH/Local command execution
- OpenAPI Document-like syntax for HTTP request testing.
- Single binary = CI-Friendly.
You can use the runn new
command to quickly start creating scenarios (runbooks).
🚀 Create and run scenario using curl
or grpcurl
commands:
Command details
$ curl https://httpbin.org/json -H "accept: application/json"
{
"slideshow": {
"author": "Yours Truly",
"date": "date of publication",
"slides": [
{
"title": "Wake up to WonderWidgets!",
"type": "all"
},
{
"items": [
"Why <em>WonderWidgets</em> are great",
"Who <em>buys</em> WonderWidgets"
],
"title": "Overview",
"type": "all"
}
],
"title": "Sample Slide Show"
}
}
$ runn new --and-run --desc 'httpbin.org GET' --out http.yml -- curl https://httpbin.org/json -H "accept: application/json"
$ grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
{
"reply": "hello alice"
}
$ runn new --and-run --desc 'grpcb.in Call' --out grpc.yml -- grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
$ runn list *.yml
Desc Path If
---------------------------------
grpcb.in Call grpc.yml
httpbin.org GET http.yml
$ runn run *.yml
..
2 scenarios, 0 skipped, 0 failures
🚀 Create scenario using access log:
Command details
$ cat access_log
183.87.255.54 - - [18/May/2019:05:37:09 +0200] "GET /?post=%3script%3ealert(1); HTTP/1.0" 200 42433
62.109.16.162 - - [18/May/2019:05:37:12 +0200] "GET /core/files/js/editor.js/?form=\xeb\x2a\x5e\x89\x76\x08\xc6\x46\x07\x00\xc7\x46\x0c\x00\x00\x00\x80\xe8\xdc\xff\xff\xff/bin/sh HTTP/1.0" 200 81956
87.251.81.179 - - [18/May/2019:05:37:13 +0200] "GET /login.php/?user=admin&amount=100000 HTTP/1.0" 400 4797
103.36.79.144 - - [18/May/2019:05:37:14 +0200] "GET /authorize.php/.well-known/assetlinks.json HTTP/1.0" 200 9436
$ cat access_log| runn new --out axslog.yml
$ cat axslog.yml| yq
desc: Generated by `runn new`
runners:
req: https://dummy.example.com
steps:
- req:
/?post=%3script%3ealert(1);:
get:
body: null
- req:
/core/files/js/editor.js/?form=xebx2ax5ex89x76x08xc6x46x07x00xc7x46x0cx00x00x00x80xe8xdcxffxffxff/bin/sh:
get:
body: null
- req:
/login.php/?user=admin&amount=100000:
get:
body: null
- req:
/authorize.php/.well-known/assetlinks.json:
get:
body: null
$
runn
can run a multi-step scenario following a runbook
written in YAML format.
runn
can run one or more runbooks as a CLI tool.
$ runn list path/to/**/*.yml
id: desc: if: steps: path
-------------------------------------------------------------------------
a1b7b02 Only if included included 2 p/t/only_if_included.yml
85ccd5f List projects. 4 p/t/p/list.yml
47d7ef7 List users. 3 p/t/u/list.yml
97f9884 Login 2 p/t/u/login.yml
2249d1b Logout 3 p/t/u/logout.yml
$ runn run path/to/**/*.yml
S....
5 scenarios, 1 skipped, 0 failures
runn
can also behave as a test helper for the Go language.
Run N runbooks using httptest.Server and sql.DB
func TestRouter(t *testing.T) {
ctx := context.Background()
dsn := "username:password@tcp(localhost:3306)/testdb"
db, err := sql.Open("mysql", dsn)
if err != nil {
log.Fatal(err)
}
dbr, err := sql.Open("mysql", dsn)
if err != nil {
log.Fatal(err)
}
ts := httptest.NewServer(NewRouter(db))
t.Cleanup(func() {
ts.Close()
db.Close()
dbr.Close()
})
opts := []runn.Option{
runn.T(t),
runn.Runner("req", ts.URL),
runn.DBRunner("db", dbr),
}
o, err := runn.Load("testdata/books/**/*.yml", opts...)
if err != nil {
t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
t.Fatal(err)
}
}
Run single runbook using httptest.Server and sql.DB
func TestRouter(t *testing.T) {
ctx := context.Background()
dsn := "username:password@tcp(localhost:3306)/testdb"
db, err := sql.Open("mysql", dsn)
if err != nil {
log.Fatal(err)
}
dbr, err := sql.Open("mysql", dsn)
if err != nil {
log.Fatal(err)
}
ts := httptest.NewServer(NewRouter(db))
t.Cleanup(func() {
ts.Close()
db.Close()
dbr.Close()
})
opts := []runn.Option{
runn.T(t),
runn.Book("testdata/books/login.yml"),
runn.Runner("req", ts.URL),
runn.DBRunner("db", dbr),
}
o, err := runn.New(opts...)
if err != nil {
t.Fatal(err)
}
if err := o.Run(ctx); err != nil {
t.Fatal(err)
}
}
Run N runbooks using grpc.Server
func TestServer(t *testing.T) {
addr := "127.0.0.1:8080"
l, err := net.Listen("tcp", addr)
if err != nil {
t.Fatal(err)
}
ts := grpc.NewServer()
myapppb.RegisterMyappServiceServer(s, NewMyappServer())
reflection.Register(s)
go func() {
ts.Serve(l)
}()
t.Cleanup(func() {
ts.GracefulStop()
})
opts := []runn.Option{
runn.T(t),
runn.Runner("greq", fmt.Sprintf("grpc://%s", addr),
}
o, err := runn.Load("testdata/books/**/*.yml", opts...)
if err != nil {
t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
t.Fatal(err)
}
}
Run N runbooks with http.Handler and sql.DB
func TestRouter(t *testing.T) {
ctx := context.Background()
dsn := "username:password@tcp(localhost:3306)/testdb"
db, err := sql.Open("mysql", dsn)
if err != nil {
log.Fatal(err)
}
dbr, err := sql.Open("mysql", dsn)
if err != nil {
log.Fatal(err)
}
t.Cleanup(func() {
db.Close()
dbr.Close()
})
opts := []runn.Option{
runn.T(t),
runn.HTTPRunnerWithHandler("req", NewRouter(db)),
runn.DBRunner("db", dbr),
}
o, err := runn.Load("testdata/books/**/*.yml", opts...)
if err != nil {
t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
t.Fatal(err)
}
}
See the details
The runbook file has the following format.
step:
section accepts list or ordered map.
List:
desc: Login and get projects.
runners:
req: https://example.com/api/v1
db: mysql://root:mypass@localhost:3306/testdb
vars:
username: alice
password: ${TEST_PASS}
steps:
-
db:
query: SELECT * FROM users WHERE name = '{{ vars.username }}'
-
req:
/login:
post:
body:
application/json:
email: "{{ steps[0].rows[0].email }}"
password: "{{ vars.password }}"
test: steps[1].res.status == 200
-
req:
/projects:
get:
headers:
Authorization: "token {{ steps[1].res.body.session_token }}"
body: null
test: steps[2].res.status == 200
-
test: len(steps[2].res.body.projects) > 0
Map:
desc: Login and get projects.
runners:
req: https://example.com/api/v1
db: mysql://root:mypass@localhost:3306/testdb
vars:
username: alice
password: ${TEST_PASS}
steps:
find_user:
db:
query: SELECT * FROM users WHERE name = '{{ vars.username }}'
login:
req:
/login:
post:
body:
application/json:
email: "{{ steps.find_user.rows[0].email }}"
password: "{{ vars.password }}"
test: steps.login.res.status == 200
list_projects:
req:
/projects:
get:
headers:
Authorization: "token {{ steps.login.res.body.session_token }}"
body: null
test: steps.list_projects.res.status == 200
count_projects:
test: len(steps.list_projects.res.body.projects) > 0
List:
Map:
Description of runbook.
desc: Login and get projects.
runners:
req: https://example.com/api/v1
vars:
username: alice
steps:
[...]
Labels of runbook.
desc: Login
runners:
req: https://example.com/api/v1
labels:
- users
- auth
steps:
[...]
Runbooks to be run can be filtered by labels.
$ runn run path/to/**/*.yml --label users --label projects
$ runn run path/to/**/*.yml --label 'users and auth'
Mapping of runners that run steps:
of runbook.
In the steps:
section, call the runner with the key specified in the runners:
section.
Built-in runners such as test runner do not need to be specified in this section.
runners:
ghapi: ${GITHUB_API_ENDPOINT}
idp: https://auth.example.com
db: my:dbuser:${DB_PASS}@hostname:3306/dbname
In the example, each runner can be called by ghapi:
, idp:
or db:
in steps:
.
Allows remapping any request hostname to another hostname, IP address in HTTP/gRPC/DB/CDP/SSH runners.
hostRules:
example.com: 127.0.0.1:8080
'*.example.test': 192.168.0.16
Mapping of variables available in the steps:
of runbook.
vars:
username: alice@example.com
token: ${SECRET_TOKEN}
In the example, each variable can be used in {{ vars.username }}
or {{ vars.token }}
in steps:
.
Enable debug output for runn.
debug: true
Interval between steps.
interval: 1
Conditions for skip all steps.
if: included # Run steps only if included
Skip all test:
sections
skipTest: true
Force all steps to run.
force: true
Add tokens for tracing to headers and queries by default.
Currently, HTTP runner, gRPC runner and DB runner are supported.
trace: true
Loop setting for runbook.
loop: 10
steps:
[...]
or
loop:
count: 10
steps:
[...]
It can be used as a retry mechanism by setting a condition in the until:
section.
If the condition of until:
is met, the loop is broken without waiting for the number of count:
to be run.
Also, if the run of the number of count:
completes but does not satisfy the condition of until:
, then the step is considered to be failed.
loop:
count: 10
until: 'outcome == "success"' # until the runbook outcome is successful.
minInterval: 0.5 # sec
maxInterval: 10 # sec
# jitter: 0.0
# interval: 5
# multiplier: 1.5
steps:
waitingroom:
req:
/cart/in:
post:
body:
[...]
outcome
... the result of a completed (success
,failure
,skipped
).
Runbooks with the same key are assured of a single run at the same time.
concurrency: use-shared-db
or
concurrency:
- use-shared-db
- use-shared-api
It is possible to identify runbooks that must be pre-run.
needs:
prebook: path/to/prebook.yml
prebook2: path/to/prebook2.yml
Values bound by the bind runner can be referenced by needs.<key>. *
.
Steps to run in runbook.
The steps are invoked in order from top to bottom.
Any return values are recorded for each step.
When steps:
is array, recorded values can be retrieved with {{ steps[*].* }}
.
steps:
-
db:
query: SELECT * FROM users WHERE name = '{{ vars.username }}'
-
req:
/users/{{ steps[0].rows[0].id }}:
get:
body: null
When steps:
is map, recorded values can be retrieved with {{ steps.<key>.* }}
.
steps:
find_user:
db:
query: SELECT * FROM users WHERE name = '{{ vars.username }}'
user_info:
req:
/users/{{ steps.find_user.rows[0].id }}:
get:
body: null
Description of step.
steps:
-
desc: Login
req:
/login:
post:
body:
[...]
Conditions for skip step.
steps:
login:
if: 'len(vars.token) == 0' # Run step only if var.token is not set
req:
/login:
post:
body:
[...]
Loop settings for steps.
steps:
multicartin:
loop: 10
req:
/cart/in:
post:
body:
application/json:
product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]
or
steps:
multicartin:
loop:
count: 10
req:
/cart/in:
post:
body:
application/json:
product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]
It can be used as a retry mechanism by setting a condition in the until:
section.
If the condition of until:
is met, the loop is broken without waiting for the number of count:
to be run.
Also, if the run of the number of count:
completes but does not satisfy the condition of until:
, then the step is considered to be failed.
steps:
waitingroom:
loop:
count: 10
until: 'steps.waitingroom.res.status == "201"' # Store values of latest loop
minInterval: 500ms
maxInterval: 10 # sec
# jitter: 0.0
# interval: 5
# multiplier: 1.5
req:
/cart/in:
post:
body:
[...]
runn can use variables and functions when running step.
Also, after step runs, HTTP responses, DB query results, etc. are automatically stored in variables.
The values are stored in predefined variables.
Variable name | Description |
---|---|
vars |
Values set in the vars: section |
steps |
Return values for each step |
i |
Loop index (only in loop: section) |
env |
Environment variables |
current |
Return values of current step |
previous |
Return values of previous step |
parent |
Variables of parent runbook (only included) |
Use https://
or http://
scheme to specify HTTP Runner.
When the step is invoked, it sends the specified HTTP Request and records the response.
runners:
req: https://example.com
steps:
-
desc: Post /users # description of step
req: # key to identify the runner. In this case, it is HTTP Runner.
/users: # path of http request
post: # method of http request
headers: # headers of http request
Authorization: 'Bearer xxxxx'
body: # body of http request
application/json: # Content-Type specification. In this case, it is "Content-Type: application/json"
username: alice
password: passw0rd
trace: false # add `X-Runn-Trace` header to HTTP request for tracing
test: | # test for current step
current.res.status == 201
See testdata/book/http.yml and testdata/book/http_multipart.yml.
The following response
HTTP/1.1 200 OK
Content-Length: 29
Content-Type: application/json
Date: Wed, 07 Sep 2022 06:28:20 GMT
Set-Cookie: cookie-name=cookie-value
{"data":{"username":"alice"}}
is recorded with the following structure.
[`step key` or `current` or `previous`]:
res:
status: 200 # current.res.status
headers:
Content-Length:
- '29' # current.res.headers["Content-Length"][0]
Content-Type:
- 'application/json' # current.res.headers["Content-Type"][0]
Date:
- 'Wed, 07 Sep 2022 06:28:20 GMT' # current.res.headers["Date"][0]
Set-Cookie:
- 'cookie-name=cookie-value' # current.res.headers["Set-Cookie"][0]
cookies:
cookie-name: *http.Cookie # current.res.cookies["cookie-name"].Value
body:
data:
username: 'alice' # current.res.body.data.username
rawBody: '{"data":{"username":"alice"}}' # current.res.rawBody
The HTTP Runner interprets HTTP responses and automatically redirects.
To disable this, set notFollowRedirect
to true.
runners:
req:
endpoint: https://example.com
notFollowRedirect: true
The HTTP Runner automatically saves cookies by interpreting HTTP responses.
To enable cookie sending during requests, set useCookie
to true.
runners:
req:
endpoint: https://example.com
useCookie: true
See testdata/book/cookie.yml and testdata/book/cookie_in_requests_automatically.yml.
HTTP requests sent by runn
and their HTTP responses can be validated.
OpenAPI v3:
runners:
myapi:
endpoint: https://api.example.com
openapi3: path/to/openapi.yaml
# skipValidateRequest: false
# skipValidateResponse: false
# skipCircularReferenceCheck: false # skip checking circular references in OpenAPIv3 document.
runners:
myapi:
endpoint: https://api.github.com
cacert: path/to/cacert.pem
cert: path/to/cert.pem
key: path/to/key.pem
# skipVerify: false
runners:
myapi:
endpoint: https://api.github.com
trace: true
Use grpc://
scheme to specify gRPC Runner.
When the step is invoked, it sends the specified gRPC Request and records the response.
runners:
greq: grpc://grpc.example.com:80
steps:
-
desc: Request using Unary RPC # description of step
greq: # key to identify the runner. In this case, it is gRPC Runner.
grpctest.GrpcTestService/Hello: # package.Service/Method of rpc
headers: # headers of rpc
authentication: tokenhello
message: # message of rpc
name: alice
num: 3
request_time: 2022-06-25T05:24:43.861872Z
trace: false # add `x-runn-trace` header to gRPC request for tracing
-
desc: Request using Server streaming RPC
greq:
grpctest.GrpcTestService/ListHello:
headers:
authentication: tokenlisthello
message:
name: bob
num: 4
request_time: 2022-06-25T05:24:43.861872Z
timeout: 3sec # timeout for rpc
test: |
steps.server_streaming.res.status == 0 && len(steps.server_streaming.res.messages) > 0
-
desc: Request using Client streaming RPC
greq:
grpctest.GrpcTestService/MultiHello:
headers:
authentication: tokenmultihello
messages: # messages of rpc
-
name: alice
num: 5
request_time: 2022-06-25T05:24:43.861872Z
-
name: bob
num: 6
request_time: 2022-06-25T05:24:43.861872Z
runners:
greq:
addr: grpc.example.com:8080
tls: true
cacert: path/to/cacert.pem
cert: path/to/cert.pem
key: path/to/key.pem
# skipVerify: false
# importPaths:
# - protobuf/proto
# protos:
# - general/health.proto
# - myapp/**/*.proto
The following response
message HelloResponse {
string message = 1;
int32 num = 2;
google.protobuf.Timestamp create_time = 3;
}
{"create_time":"2022-06-25T05:24:43.861872Z","message":"hello","num":32}
and headers
content-type: ["application/grpc"]
hello: ["this is header"]
and trailers
hello: ["this is trailer"]
are recorded with the following structure.
[`step key` or `current` or `previous`]:
res:
status: 0 # current.res.status
headers:
content-type:
- 'application/grpc' # current.res.headers[0].content-type
hello:
- 'this is header' # current.res.headers[0].hello
trailers:
hello:
- 'this is trailer' # current.res.trailers[0].hello
message:
create_time: '2022-06-25T05:24:43.861872Z' # current.res.message.create_time
message: 'hello' # current.res.message.message
num: 32 # current.res.message.num
messages:
-
create_time: '2022-06-25T05:24:43.861872Z' # current.res.messages[0].create_time
message: 'hello' # current.res.messages[0].message
num: 32 # current.res.messages[0].num
runners:
greq:
addr: grpc.example.com:8080
trace: true
gRPC Runner supports Buf ecosystem includes Buf Schema Registry.
It can use the buf modules ( and protos ) it depends on.
runners:
greq:
addr: grpc.example.com:8080
bufDirs:
- path/to # Set buf directories for registering buf modules and protos
runners:
greq:
addr: grpc.example.com:8080
bufLocks:
- path/to/buf.lock # Register buf modules using buf.lock
runners:
greq:
addr: grpc.example.com:8080
bufConfigs:
- path/to/buf.yaml # Register buf modules using buf.yaml
runners:
greq:
addr: grpc.example.com:8080
bufModules:
- buf.build/owner/repository
- buf.build/owner2/repository2
Use dsn (Data Source Name) to specify DB Runner.
When step is invoked, it executes the specified query the database.
runners:
db: postgres://dbuser:dbpass@hostname:5432/dbname
steps:
-
desc: Select users # description of step
db: # key to identify the runner. In this case, it is DB Runner.
query: SELECT * FROM users; # query to execute
trace: false # add comment with trace token to query for tracing
See testdata/book/db.yml.
If the query is a SELECT clause, it records the selected rows
,
[`step key` or `current` or `previous`]:
rows:
-
id: 1 # current.rows[0].id
username: 'alice' # current.rows[0].username
password: 'passw0rd' # current.rows[0].password
email: 'alice@example.com' # current.rows[0].email
created: '2017-12-05T00:00:00Z' # current.rows[0].created
-
id: 2 # current.rows[1].id
username: 'bob' # current.rows[1].username
password: 'passw0rd' # current.rows[1].password
email: 'bob@example.com' # current.rows[1].email
created: '2022-02-22T00:00:00Z' # current.rows[1].created
otherwise it records last_insert_id
and rows_affected
.
[`step key` or `current` or `previous`]:
last_insert_id: 3 # current.last_insert_id
rows_affected: 1 # current.rows_affected
runners:
db:
dsn: mysql://dbuser:dbpass@hostname:3306/dbname
trace: true
PostgreSQL:
runners:
mydb: postgres://dbuser:dbpass@hostname:5432/dbname
runners:
db: pg://dbuser:dbpass@hostname:5432/dbname
MySQL:
runners:
testdb: mysql://dbuser:dbpass@hostname:3306/dbname
runners:
db: my://dbuser:dbpass@hostname:3306/dbname
SQLite3:
runners:
db: sqlite:///path/to/dbname.db
runners:
local: sq://dbname.db
Cloud Spanner:
runners:
testdb: spanner://test-project/test-instance/test-database
runners:
db: sp://test-project/test-instance/test-database
Use cdp://
or chrome://
scheme to specify CDP Runner.
When the step is invoked, it controls browser via Chrome DevTools Protocol.
runners:
cc: chrome://new
steps:
-
desc: Navigate, click and get h1 using CDP # description of step
cc: # key to identify the runner. In this case, it is CDP Runner.
actions: # actions to control browser
- navigate: https://pkg.go.dev/time
- click: 'body > header > div.go-Header-inner > nav > div > ul > li:nth-child(2) > a'
- waitVisible: 'body > footer'
- text: 'h1'
-
test: |
previous.text == 'Install the latest version of Go'
attributes
(aliases: getAttributes
, attrs
, getAttrs
)
Get the element attributes for the first element node matching the selector (sel
).
actions:
- attributes:
sel: 'h1'
# record to current.attrs:
or
actions:
- attributes: 'h1'
click
Send a mouse click event to the first element node matching the selector (sel
).
actions:
- click:
sel: 'nav > div > a'
or
actions:
- click: 'nav > div > a'
doubleClick
Send a mouse double click event to the first element node matching the selector (sel
).
actions:
- doubleClick:
sel: 'nav > div > li'
or
actions:
- doubleClick: 'nav > div > li'
evaluate
(aliases: eval
)
Evaluate the Javascript expression (expr
).
actions:
- evaluate:
expr: 'document.querySelector("h1").textContent = "hello"'
or
actions:
- evaluate: 'document.querySelector("h1").textContent = "hello"'
fullHTML
(aliases: getFullHTML
, getHTML
, html
)
Get the full html of page.
actions:
- fullHTML
# record to current.html:
innerHTML
(aliases: getInnerHTML
)
Get the inner html of the first element node matching the selector (sel
).
actions:
- innerHTML:
sel: 'h1'
# record to current.html:
or
actions:
- innerHTML: 'h1'
latestTab
(aliases: latestTarget
)
Change current frame to latest tab.
actions:
- latestTab
localStorage
(aliases: getLocalStorage
)
Get localStorage items.
actions:
- localStorage:
origin: 'https://github.com'
# record to current.items:
or
actions:
- localStorage: 'https://github.com'
location
(aliases: getLocation
)
Get the document location.
actions:
- location
# record to current.url:
navigate
Navigate the current frame to url
page.
actions:
- navigate:
url: 'https://pkg.go.dev/time'
or
actions:
- navigate: 'https://pkg.go.dev/time'
outerHTML
(aliases: getOuterHTML
)
Get the outer html of the first element node matching the selector (sel
).
actions:
- outerHTML:
sel: 'h1'
# record to current.html:
or
actions:
- outerHTML: 'h1'
screenshot
(aliases: getScreenshot
)
Take a full screenshot of the entire browser viewport.
actions:
- screenshot
# record to current.png:
scroll
(aliases: scrollIntoView
)
Scroll the window to the first element node matching the selector (sel
).
actions:
- scroll:
sel: 'body > footer'
or
actions:
- scroll: 'body > footer'
sendKeys
Send keys (value
) to the first element node matching the selector (sel
).
actions:
- sendKeys:
sel: 'input[name=username]'
value: 'k1lowxb@gmail.com'
sessionStorage
(aliases: getSessionStorage
)
Get sessionStorage items.
actions:
- sessionStorage:
origin: 'https://github.com'
# record to current.items:
or
actions:
- sessionStorage: 'https://github.com'
setUploadFile
(aliases: setUpload
)
Set upload file (path
) to the first element node matching the selector (sel
).
actions:
- setUploadFile:
sel: 'input[name=avator]'
path: '/path/to/image.png'
setUserAgent
(aliases: setUA
, ua
, userAgent
)
Set the default User-Agent
actions:
- setUserAgent:
userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'
or
actions:
- setUserAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'
submit
Submit the parent form of the first element node matching the selector (sel
).
actions:
- submit:
sel: 'form.login'
or
actions:
- submit: 'form.login'
text
(aliases: getText
)
Get the visible text of the first element node matching the selector (sel
).
actions:
- text:
sel: 'h1'
# record to current.text:
or
actions:
- text: 'h1'
textContent
(aliases: getTextContent
)
Get the text content of the first element node matching the selector (sel
).
actions:
- textContent:
sel: 'h1'
# record to current.text:
or
actions:
- textContent: 'h1'
title
(aliases: getTitle
)
Get the document title
.
actions:
- title
# record to current.title:
value
(aliases: getValue
)
Get the Javascript value field of the first element node matching the selector (sel
).
actions:
- value:
sel: 'input[name=address]'
# record to current.value:
or
actions:
- value: 'input[name=address]'
wait
(aliases: sleep
)
Wait for the specified time
.
actions:
- wait:
time: '10sec'
or
actions:
- wait: '10sec'
waitReady
Wait until the element matching the selector (sel
) is ready.
actions:
- waitReady:
sel: 'body > footer'
or
actions:
- waitReady: 'body > footer'
waitVisible
Wait until the element matching the selector (sel
) is visible.
actions:
- waitVisible:
sel: 'body > footer'
or
actions:
- waitVisible: 'body > footer'
Use ssh://
scheme to specify SSH Runner.
When step is invoked, it executes commands on a remote server connected via SSH.
runners:
sc: ssh://username@hostname:port
steps:
-
desc: 'execute `hostname`' # description of step
sc:
command: hostname
runners:
sc:
hostname: hostname
user: username
port: 22
# host: myserver
# sshConfig: path/to/ssh_config
# keepSession: false
# localForward: '33306:127.0.0.1:3306'
# keyboardInteractive:
# - match: Username
# answer: k1low
# - match: OTP
# answer: ${MY_OTP}
The response to the run command is always stdout
and stderr
.
[`step key` or `current` or `previous`]:
stdout: 'hello world' # current.stdout
stderr: '' # current.stderr
Note Exec runner requires
run:exec
scope to run.
The exec
runner is a built-in runner, so there is no need to specify it in the runners:
section.
It execute command using command:
, stdin:
, shell:
, background:
and liveOutput:
.
-
exec:
command: grep hello
stdin: '{{ steps[3].res.rawBody }}'
-
exec:
command: echo $0
shell: bash
background:
set to true
to run the command in the background.
-
exec:
command: |
echo "Start heavy commands"
sleep 5
echo "Heavy command finished"
liveOutput: true
liveOutput:
set to true
to output the command output live.
The response to the run command is always stdout
, stderr
and exit_code
.
[`step key` or `current` or `previous`]:
stdout: 'hello world' # current.stdout
stderr: '' # current.stderr
exit_code: 0 # current.exit_code
The test
runner is a built-in runner, so there is no need to specify it in the runners:
section.
It evaluates the conditional expression using the recorded values.
-
test: steps[3].res.status == 200
The test
runner can run in the same steps as the other runners.
The dump
runner is a built-in runner, so there is no need to specify it in the runners:
section.
It dumps the specified recorded values.
-
dump: steps[4].rows
or
-
dump:
expr: steps[4].rows
out: path/to/dump.out
disableTrailingNewline: true
The dump
runner can run in the same steps as the other runners.
The include
runner is a built-in runner, so there is no need to specify it in the runners:
section.
Include runner reads and runs the runbook in the specified path.
Recorded values are nested.
-
include: path/to/get_token.yml
It is also possible to override vars:
of included runbook.
-
include:
path: path/to/login.yml
vars:
username: alice
password: alicepass
-
include:
path: path/to/login.yml
vars:
username: bob
password: bobpass
It is also possible to skip all test:
sections in the included runbook.
-
include:
path: path/to/signup.yml
skipTest: true
It is also possible to force all steps in the included runbook to run.
-
include:
path: path/to/signup.yml
force: true
The bind
runner is a built-in runner, so there is no need to specify it in the runners:
section.
It bind runner binds any values with another key.
-
req:
/users/k1low:
get:
body: null
-
bind:
user_id: steps[0].res.body.data.id
-
dump: user_id
The bind
runner can run in the same steps as the other runners.
The runner
runner is a built-in runner, so there is no need to specify it in the runners:
section.
It defines a runner in the middle of steps.
-
runner:
sc: ssh://username@hostname:port
-
sc:
command: hostname
The runner
runner can not run in the same steps as the other runners.
runn has embedded expr-lang/expr as the evaluation engine for the expression.
See Language Definition.
urlencode
... url.QueryEscapebool
... cast.ToBoolcompare
... Compare two values (func(x, y any, ignorePaths ...string) bool
). The optionalignorePaths
argument is a list of jq syntax path expressions to ignore when comparing two values.diff
... Difference between two values (func(x, y any, ignorePaths ...string) string
). The optionalignorePaths
argument is a list of jq syntax path expressions to ignore when comparing two values.pick
... Returns same map type filtered by given keys left lo.PickByKeys.omit
... Returns same map type filtered by given keys excluded lo.OmitByKeys.merge
... Merges multiple maps from left to right lo.Assign.input
... prompter.Promptintersect
... Find the intersection of two iterable values (func(x, y any) any
).secret
... prompter.Passwordselect
... Select from candidates.func(message string, candidates []string, default string) string
basename
... filepath.Basetime
... Converts the given string or number totime.Time{}
.faker.*
... Generate fake data using Faker ).
See https://pkg.go.dev/github.com/k1LoW/runn#Option
https://pkg.go.dev/github.com/k1LoW/runn#T
o, err := runn.Load("testdata/**/*.yml", runn.T(t))
if err != nil {
t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
t.Fatal(err)
}
https://pkg.go.dev/github.com/k1LoW/runn#Func
desc: Test using GitHub
runners:
req:
endpoint: https://github.com
steps:
-
req:
/search?l={{ urlencode('C++') }}&q=runn&type=Repositories:
get:
body:
application/json:
null
test: 'steps[0].res.status == 200'
o, err := runn.Load("testdata/**/*.yml", runn.Func("urlencode", url.QueryEscape))
if err != nil {
t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
t.Fatal(err)
}
runn requires explicit specification of scope for some features.
runn has the following scopes.
Scope | Description | Default |
---|---|---|
read:parent |
Required for reading files above the working directory. | false |
read:remote |
Required for reading remote files. | false |
run:exec |
Required for running Exec runner. | false |
To specify scopes, using the --scopes
option or the environment variable RUNN_SCOPES
.
$ runn run path/to/**/*.yml --scopes read:parent,read:remote
$ env RUNN_SCOPES=read:parent,read:remote runn run path/to/**/*.yml
Also, runn.Scopes can be used in the code
o, err := runn.Load("path/to/**/*.yml", runn.Scopes(runn.ScopeAllowReadParent, runn.ScopeAllowReadRemote))
if err != nil {
t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
t.Fatal(err)
}
To disable scope, can use !read:*
instead of read:*
$ runn run path/to/**/*.yml --scopes '!read:parent'
Run only runbooks matching the filename "login".
$ env RUNN_RUN=login go test ./... -run TestRouter
opts := []runn.Option{
runn.T(t),
runn.Book("testdata/books/login.yml"),
runn.Profile(true),
}
o, err := runn.New(opts...)
if err != nil {
t.Fatal(err)
}
if err := o.Run(ctx); err != nil {
t.Fatal(err)
}
f, err := os.Open("profile.json")
if err != nil {
t.Fatal(err)
}
if err := o.DumpProfile(f); err != nil {
t.Fatal(err)
}
or
$ runn run testdata/books/login.yml --profile
The runbook run profile can be read with runn rprof
command.
$ runn rprof runn.prof
runbook[login site](t/b/login.yml) 2995.72ms
steps[0].req 747.67ms
steps[1].req 185.69ms
steps[2].req 192.65ms
steps[3].req 188.23ms
steps[4].req 569.53ms
steps[5].req 299.88ms
steps[6].test 0.14ms
steps[7].include 620.88ms
runbook[include](t/b/login_include.yml) 605.56ms
steps[0].req 605.54ms
steps[8].req 190.92ms
[total] 2995.84ms
opts := []runn.Option{
runn.T(t),
runn.Capture(capture.Runbook("path/to/dir")),
}
o, err := runn.Load("testdata/books/**/*.yml", opts...)
if err != nil {
t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
t.Fatal(err)
}
or
$ runn run path/to/**/*.yml --capture path/to/dir
You can use the runn loadt
command for load testing using runbooks.
$ runn loadt --load-concurrent 2 --max-rps 0 path/to/*.yml
Number of runbooks per RunN....: 15
Warm up time (--warm-up).......: 5s
Duration (--duration)..........: 10s
Concurrent (--load-concurrent).: 2
Max RunN per second (--max-rps): 0
Total..........................: 12
Succeeded......................: 12
Failed.........................: 0
Error rate.....................: 0%
RunN per seconds...............: 1.2
Latency .......................: max=1,835.1ms min=1,451.3ms avg=1,627.8ms med=1,619.8ms p(90)=1,741.5ms p(99)=1,788.4ms
It also checks the results of the load test with the --threshold
option. If the condition is not met, it returns exit status 1.
$ runn loadt --load-concurrent 2 --max-rps 0 --threshold 'error_rate < 10' path/to/*.yml
Number of runbooks per RunN...: 15
Warm up time (--warm-up)......: 5s
Duration (--duration).........: 10s
Concurrent (--load-concurrent): 2
Total.........................: 13
Succeeded.....................: 12
Failed........................: 1
Error rate....................: 7.6%
RunN per seconds..............: 1.3
Latency ......................: max=1,790.2ms min=95.0ms avg=1,541.4ms med=1,640.4ms p(90)=1,749.7ms p(99)=1,786.5ms
Error: (error_rate < 10) is not true
error_rate < 10
├── error_rate => 14.285714285714285
└── 10 => 10
Variable name | Type | Description |
---|---|---|
total |
int |
Total |
succeeded |
int |
Succeeded |
failed |
int |
Failed |
error_rate |
float |
Error rate |
rps |
float |
RunN per seconds |
max |
float |
Latency max (ms) |
mid |
float |
Latency mid (ms) |
min |
float |
Latency min (ms) |
p90 |
float |
Latency p(90) (ms) |
p99 |
float |
Latency p(99) (ms) |
avg |
float |
Latency avg (ms) |
deb:
$ export RUNN_VERSION=X.X.X
$ curl -o runn.deb -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.deb
$ dpkg -i runn.deb
RPM:
$ export RUNN_VERSION=X.X.X
$ yum install https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.rpm
apk:
$ export RUNN_VERSION=X.X.X
$ curl -o runn.apk -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.apk
$ apk add runn.apk
homebrew tap:
$ brew install k1LoW/tap/runn
aqua:
$ aqua g -i k1LoW/runn
manually:
Download binary from releases page
docker:
$ docker container run -it --rm --name runn -v $PWD:/books ghcr.io/k1low/runn:latest list /books/*.yml
go install:
$ go install github.com/k1LoW/runn/cmd/runn@latest
$ go get github.com/k1LoW/runn
- zoncoen/scenarigo: An end-to-end scenario testing tool for HTTP/gRPC server.
- zoncoen/scenarigo: An end-to-end scenario testing tool for HTTP/gRPC server.
- fullstorydev/grpcurl: Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers
- ktr0731/evans: Evans: more expressive universal gRPC client
- MIT License
- Include logo as well as source code.
- Only logo license can be selected CC BY 4.0.
- Also, if there is no alteration to the logo and it is used for technical information about runn, I would not say anything if the copyright notice is omitted.