This document summarizes the most important changes in the current Bro

release. For an exhaustive list of changes, see the ``CHANGES`` file

(note that submodules, such as BroControl and Broccoli, come with

their own ``CHANGES``.)

Bro 2.4

=======

New Functionality

-----------------

- Bro now has support for external plugins that can extend its core

functionality, like protocol/file analysis, via shared libraries.

Plugins can be developed and distributed externally, and will be

pulled in dynamically at startup (the environment variables

BRO_PLUGIN_PATH and BRO_PLUGIN_ACTIVATE can be used to specify the

locations and names of plugins to activate). Currently, a plugin

can provide custom protocol analyzers, file analyzers, log writers,

input readers, packet sources and dumpers, and new built-in functions.

A plugin can furthermore hook into Bro's processing at a number of

places to add custom logic.

See https://www.bro.org/sphinx-git/devel/plugins.html for more

information on writing plugins.

- Bro now has support for the MySQL wire protocol. Activity gets

logged into mysql.log.

- Bro now parses DTLS traffic. Activity gets logged into ssl.log.

- Bro now has support for the Kerberos KRB5 protocol over TCP and

UDP. Activity gets logged into kerberos.log.

- Bro now has an RDP analyzer. Activity gets logged into rdp.log.

- Bro now has a file analyzer for Portable Executables. Activity gets

logged into pe.log.

- Bro now has support for the SIP protocol over UDP. Activity gets

logged into sip.log.

- Bro now features a completely rewritten, enhanced SSH analyzer. The

new analyzer is able to determine if logins failed or succeeded in

most circumstances, logs a lot more more information about SSH

sessions, supports v1, and introduces the intelligence type

``Intel::PUBKEY_HASH`` and location ``SSH::IN_SERVER_HOST_KEY``. The

analayzer also generates a set of additional events

(``ssh_auth_successful``, ``ssh_auth_failed``, ``ssh_capabilities``,

``ssh2_server_host_key``, ``ssh1_server_host_key``,

``ssh_encrypted_packet``, ``ssh2_dh_server_params``,

``ssh2_gss_error``, ``ssh2_ecc_key``). See next section for

incompatible SSH changes.

- Bro's file analysis now supports reassembly of files that are not

transferred/seen sequentially. The default file reassembly buffer

size is set with the ``Files::reassembly_buffer_size`` variable.

- Bro's file type identification has been greatly improved (new file types,

bug fixes, and performance improvements).

- Bro's scripting language now has a ``while`` statement::

while ( i < 5 )

print ++i;

``next`` and ``break`` can be used inside the loop's body just like

with ``for`` loops.

- Bro now integrates Broker, a new communication library. See

aux/broker/README for more information on Broker, and

doc/frameworks/broker.rst for the corresponding Bro script API.

With Broker, Bro has the similar capabilities of exchanging events and

logs with remote peers (either another Bro process or some other

application that uses Broker). It also includes a key-value store

API that can be used to share state between peers and optionally

allow data to persist on disk for longer-term storage.

Broker support is by default off for now; it can be enabled at

configure time with --enable-broker. It requires CAF version 0.13+

(https://github.com/actor-framework/actor-framework) as well as a

C++11 compiler (e.g. GCC 4.8+ or Clang 3.3+).

Broker will become a mandatory dependency in future Bro versions and

replace the current communication and serialization system.

- Add --enable-c++11 configure flag to compile Bro's source code in

C++11 mode with a corresponding compiler. Note that 2.4 will be the

last version of Bro that compiles without C++11 support.

- The SSL analysis now alerts when encountering SSL connections with

old protocol versions or unsafe cipher suites. It also gained

extended reporting of weak keys, caching of already validated

certificates, and full support for TLS record defragmentation. SSL generally

became much more robust and added several fields to ssl.log (while

removing some others).

- A new icmp_sent_payload event provides access to ICMP payload.

- The input framework's raw reader now supports seeking by adding an

option "offset" to the config map. Positive offsets are interpreted

to be from the beginning of the file, negative from the end of the

file (-1 is end of file).

- One can now raise events when a connection crosses a given size

threshold in terms of packets or bytes. The primary API for that

functionality is in base/protocols/conn/thresholds.bro.

- There is a new command-line option -Q/--time that prints Bro's execution

time and memory usage to stderr.

- BroControl now has a new command "deploy" which is equivalent to running

the "check", "install", "stop", and "start" commands (in that order).

- BroControl now has a new option "StatusCmdShowAll" that controls whether

or not the broctl "status" command gathers all of the status information.

This option can be used to make the "status" command run significantly

faster (in this case, the "Peers" column will not be shown in the output).

- BroControl now has a new option "StatsLogEnable" that controls whether

or not broctl will record information to the "stats.log" file. This option

can be used to make the "broctl cron" command run slightly faster (in this

case, "broctl cron" will also no longer send email about not seeing any

packets on the monitoring interfaces).

- BroControl now has a new option "MailHostUpDown" which controls whether or

not the "broctl cron" command will send email when it notices that a host

in the cluster is up or down.

- BroControl now has a new option "CommandTimeout" which specifies the number

of seconds to wait for a command that broctl ran to return results.

Changed Functionality

---------------------

- bro-cut has been rewritten in C, and is hence much faster.

- File analysis

* Removed ``fa_file`` record's ``mime_type`` and ``mime_types``

fields. The event ``file_sniff`` has been added which provides

the same information. The ``mime_type`` field of ``Files::Info``

also still has this info.

* The earliest point that new mime type information is available is

in the ``file_sniff`` event which comes after the ``file_new`` and

``file_over_new_connection`` events. Scripts which inspected mime

type info within those events will need to be adapted. (Note: for

users that worked w/ versions of Bro from git, for a while there was

also an event called ``file_mime_type`` which is now replaced with

the ``file_sniff`` event).

* Removed ``Files::add_analyzers_for_mime_type`` function.

* Removed ``offset`` parameter of the ``file_extraction_limit``

event. Since file extraction now internally depends on file

reassembly for non-sequential files, "offset" can be obtained

with other information already available -- adding together

``seen_bytes`` and ``missed_bytes`` fields of the ``fa_file``

record gives how many bytes have been written so far (i.e.

the "offset").

- The SSH changes come with a few incompatibilities. The following

events have been renamed:

* ``SSH::heuristic_failed_login`` to ``SSH::ssh_auth_failed``

* ``SSH::heuristic_successful_login`` to ``SSH::ssh_auth_successful``

The ``SSH::Info`` status field has been removed and replaced with

the ``auth_success`` field. This field has been changed from a

string that was previously ``success``, ``failure`` or

``undetermined`` to a boolean. a boolean that is ``T``, ``F``, or

unset.

- The has_valid_octets function now uses a string_vec parameter instead of

string_array.

- conn.log gained a new field local_resp that works like local_orig,

just for the responder address of the connection.

- GRE tunnels are now identified as ``Tunnel::GRE`` instead of

``Tunnel::IP``.

- The default name for extracted files changed from extract-protocol-id

to extract-timestamp-protocol-id.

- The weird named "unmatched_HTTP_reply" has been removed since it can

be detected at the script-layer and is handled correctly by the

default HTTP scripts.

- When adding a logging filter to a stream, the filter can now inherit

a default ``path`` field from the associated ``Log::Stream`` record.

- When adding a logging filter to a stream, the

``Log::default_path_func`` is now only automatically added to the

filter if it has neither a ``path`` nor a ``path_func`` already

explicitly set. Before, the default path function would always be set

for all filters which didn't specify their own ``path_func``.

- BroControl now establishes only one ssh connection from the manager to

each remote host in a cluster configuration (previously, there would be

one ssh connection per remote Bro process).

- BroControl now uses SQLite to record state information instead of a

plain text file (the file "spool/broctl.dat" is no longer used).

On FreeBSD, this means that there is a new dependency on the package

"py27-sqlite3".

- BroControl now records the expected running state of each Bro node right

before each start or stop. The "broctl cron" command uses this info to

either start or stop Bro nodes as needed so that the actual state matches

the expected state (previously, "broctl cron" could only start nodes in

the "crashed" state, and could never stop a node).

- BroControl now sends all normal command output (i.e., not error messages)

to stdout. Error messages are still sent to stderr, however.

- The capability of processing NetFlow input has been removed for the

time being. Therefore, the -y/--flowfile and -Y/--netflow command-line

options have been removed, and the netflow_v5_header and netflow_v5_record

events have been removed.

- The -D/--dfa-size command-line option has been removed.

- The -L/--rule-benchmark command-line option has been removed.

- The -O/--optimize command-line option has been removed.

- The deprecated fields "hot" and "addl" have been removed from the

connection record. Likewise, the functions append_addl() and

append_addl_marker() have been removed.

- Log files now escape non-printable characters consistently as "\xXX'.

Furthermore, backslashes are escaped as "\\", making the

representation fully reversible.

Deprecated Functionality

------------------------

- The split* family of functions are to be replaced with alternate

versions that return a vector of strings rather than a table of

strings. This also allows deprecation for some related string

concatenation/extraction functions. Note that the new functions use

0-based indexing, rather than 1-based.

The full list of now deprecated functions is:

* split: use split_string instead.

* split1: use split_string1 instead.

* split_all: use split_string_all instead.

* split_n: use split_string_n instead.

* cat_string_array: see join_string_vec instead.

* cat_string_array_n: see join_string_vec instead.

* join_string_array: see join_string_vec instead.

* sort_string_array: use sort instead.

* find_ip_addresses: use extract_ip_addresses instead.

Bro 2.3

=======

Dependencies

------------

- Libmagic is no longer a dependency.

New Functionality

-----------------

- Support for GRE tunnel decapsulation, including enhanced GRE

headers. GRE tunnels are treated just like IP-in-IP tunnels by

parsing past the GRE header in between the delivery and payload IP

packets.

- The DNS analyzer now actually generates the dns_SRV_reply() event.

It had been documented before, yet was never raised.

- Bro now uses "file magic signatures" to identify file types. These

are defined via two new constructs in the signature rule parsing

grammar: "file-magic" gives a regular expression to match against,

and "file-mime" gives the MIME type string of content that matches

the magic and an optional strength value for the match. (See also

"Changed Functionality" below for changes due to switching from

using libmagic to such signatures.)

- A new built-in function, "file_magic", can be used to get all file

magic matches and their corresponding strength against a given chunk

of data.

- The SSL analyzer now supports heartbeats as well as a few

extensions, including server_name, alpn, and ec-curves.

- The SSL analyzer comes with Heartbleed detector script in

protocols/ssl/heartbleed.bro. Note that loading this script changes

the default value of "SSL::disable_analyzer_after_detection" from true

to false to prevent encrypted heartbeats from being ignored.

- StartTLS is now supported for SMTP and POP3.

- The X509 analyzer can now perform OSCP validation.

- Bro now has analyzers for SNMP and Radius, which produce corresponding

snmp.log and radius.log output (as well as various events of course).

- BroControl has a new option "BroPort" which allows a user to specify

the starting port number for Bro.

- BroControl has a new option "StatsLogExpireInterval" which allows a

user to specify when entries in the stats.log file expire.

- BroControl has a new option "PFRINGClusterType" which allows a user

to specify a PF_RING cluster type.

- BroControl now supports PF_RING+DNA. There is also a new option

"PFRINGFirstAppInstance" that allows a user to specify the starting

application instance number for processes running on a DNA cluster.

See the BroControl documentation for more details.

- BroControl now warns a user to run "broctl install" if Bro has

been upgraded or if the broctl or node configuration has changed

since the most recent install.

Changed Functionality

---------------------

- string slices now exclude the end index (e.g., "123"[1:2] returns

"2"). Generally, Bro's string slices now behave similar to Python.

- ssl_client_hello() now receives a vector of ciphers, instead of a

set, to preserve their order.

- Notice::end_suppression() has been removed.

- Bro now parses X.509 extensions headers and, as a result, the

corresponding event got a new signature:

event x509_extension(c: connection, is_orig: bool, cert: X509, ext: X509_extension_info);

- In addition, there are several new, more specialized events for a

number of x509 extensions.

- Generally, all x509 events and handling functions have changed their

signatures.

- X509 certificate verification now returns the complete certificate

chain that was used for verification.

- Bro no longer special-cases SYN/FIN/RST-filtered traces by not

reporting missing data. Instead, if Bro never sees any data segments

for analyzed TCP connections, the new

base/misc/find-filtered-trace.bro script will log a warning in

reporter.log and to stderr. The old behavior can be reverted by

redef'ing "detect_filtered_trace".

- We have removed the packet sorter component.

- Bro no longer uses libmagic to identify file types but instead now

comes with its own signature library (which initially is still

derived from libmagic's database). This leads to a number of further

changes with regards to MIME types:

* The second parameter of the "identify_data" built-in function

can no longer be used to get verbose file type descriptions,

though it can still be used to get the strongest matching file

magic signature.

* The "file_transferred" event's "descr" parameter no longer

contains verbose file type descriptions.

* The BROMAGIC environment variable no longer changes any behavior

in Bro as magic databases are no longer used/installed.

* Removed "binary" and "octet-stream" mime type detections. They

don't provide any more information than an uninitialized

mime_type field.

* The "fa_file" record now contains a "mime_types" field that

contains all magic signatures that matched the file content

(where the "mime_type" field is just a shortcut for the

strongest match).

- dns_TXT_reply() now supports more than one string entry by receiving

a vector of strings.

- BroControl now runs the "exec" and "df" broctl commands only once

per host, instead of once per Bro node. The output of these

commands has been changed slightly to include both the host and

node names.

- Several performance improvements were made. Particular emphasis

was put on the File Analysis system, which generally will now emit

far fewer file handle request events due to protocol analyzers now

caching that information internally.

Bro 2.2

=======

New Functionality

-----------------

- A completely overhauled intelligence framework for consuming

external intelligence data. It provides an abstracted mechanism

for feeding data into the framework to be matched against the

data available. It also provides a function named ``Intel::match``

which makes any hits on intelligence data available to the

scripting language.

Using input framework, the intel framework can load data from

text files. It can also update and add data if changes are

made to the file being monitored. Files to monitor for

intelligence can be provided by redef-ing the

``Intel::read_files`` variable.

The intel framework is cluster-ready. On a cluster, the

manager is the only node that needs to load in data from disk,

the cluster support will distribute the data across a cluster

automatically.

Scripts are provided at ``policy/frameworks/intel/seen`` that

provide a broad set of sources of data to feed into the intel

framwork to be matched.

- A new file analysis framework moves most of the processing of file

content from script-land into the core, where it belongs. See

``doc/file-analysis.rst``, or the online documentation, for more

information.

Much of this is an internal change, but the framework also comes

with the following user-visible functionality (some of that was

already available before but is done differently, and more

efficiently, now):

- HTTP:

* Identify MIME type of messages.

* Extract messages to disk.

* Compute MD5 for messages.

- SMTP:

* Identify MIME type of messages.

* Extract messages to disk.

* Compute MD5 for messages.

* Provide access to start of entity data.

- FTP data transfers:

* Identify MIME types of data.

* Record to disk.

- IRC DCC transfers: Record to disk.

- Support for analyzing data transferred via HTTP range requests.

- A binary input reader interfaces the input framework with the

file analysis, allowing to inject files on disk into Bro's

content processing.

- A new framework for computing a wide array of summary statistics,

such as counters and thresholds checks, standard deviation and mean,

set cardinality, top K, and more. The framework operates in

real-time, independent of the underlying data, and can aggregate

information from many independent monitoring points (including

clusters). It provides a transparent, easy-to-use user interface,

and can optionally deploy a set of probabilistic data structures for

memory-efficient operation. The framework is located in

``scripts/base/frameworks/sumstats``.

A number of new applications now ship with Bro that are built on top

of the summary statistics framework:

* Scan detection: Detectors for port and address scans. See

``policy/misc/scan.bro`` (these scan detectors used to exist in

Bro versions <2.0; it's now back, but quite different).

* Tracerouter detector: ``policy/misc/detect-traceroute.bro``

* Web application detection/measurement:

``policy/misc/app-stats/*``

* FTP and SSH brute-forcing detector:

``policy/protocols/ftp/detect-bruteforcing.bro``,

``policy/protocols/ssh/detect-bruteforcing.bro``

* HTTP-based SQL injection detector:

``policy/protocols/http/detect-sqli.bro`` (existed before, but

now ported to the new framework)

- GridFTP support. This is an extension to the standard FTP analyzer

and includes:

- An analyzer for the GSI mechanism of GSSAPI FTP AUTH method.

GSI authentication involves an encoded TLS/SSL handshake over

the FTP control session. For FTP sessions that attempt GSI

authentication, the ``service`` field of the connection log

will include ``gridftp`` (as well as also ``ftp`` and

``ssl``).

- An example of a GridFTP data channel detection script. It

relies on the heuristics of GridFTP data channels commonly

default to SSL mutual authentication with a NULL bulk cipher

and that they usually transfer large datasets (default

threshold of script is 1 GB). For identified GridFTP data

channels, the ``services`` fields of the connection log will

include ``gridftp-data``.

- Modbus and DNP3 support. Script-level support is only basic at this

point but see ``src/analyzer/protocol/{modbus,dnp3}/events.bif``, or

the online documentation, for the events Bro generates. For Modbus,

there are also some example policies in

``policy/protocols/modbus/*``.

- The documentation now includes a new introduction to writing Bro

scripts. See ``doc/scripting/index.rst`` or, much better, the online

version. There's also the beginning of a chapter on "Using Bro" in

``doc/using/index.rst``.

- GPRS Tunnelling Protocol (GTPv1) decapsulation.

- The scripting language now provide "hooks", a new flavor of

functions that share characteristics of both standard functions and

events. They are like events in that multiple bodies can be defined

for the same hook identifier. They are more like functions in the

way they are invoked/called, because, unlike events, their execution

is immediate and they do not get scheduled through an event queue.

Also, a unique feature of a hook is that a given hook handler body

can short-circuit the execution of remaining hook handlers simply by

exiting from the body as a result of a ``break`` statement (as

opposed to a ``return`` or just reaching the end of the body). See

``doc/scripts/builtins.rst``, or the online documentation, for more

informatin.

- Bro's language now has a working ``switch`` statement that generally

behaves like C-style switches (except that case labels can be

comprised of multiple literal constants delimited by commas). Only

atomic types are allowed for now. Case label bodies that don't

execute a ``return`` or ``break`` statement will fall through to

subsequent cases. A ``default`` case label is supported.

- Bro's language now has a new set of types ``opaque of X``. Opaque

values can be passed around like other values but they can only be

manipulated with BiF functions, not with other operators. Currently,

the following opaque types are supported::

opaque of md5

opaque of sha1

opaque of sha256

opaque of cardinality

opaque of topk

opaque of bloomfilter

These go along with the corrsponding BiF functions ``md5_*``,

``sha1_*``, ``sha256_*``, ``entropy_*``, etc. . Note that where

these functions existed before, they have changed their signatures

to work with opaques types rather than global state.

- The scripting language now supports constructing sets, tables,

vectors, and records by name::

type MyRecordType: record {

c: count;

s: string &optional;

};

global r: MyRecordType = record($c = 7);

type MySet: set[MyRec];

global s = MySet([$c=1], [$c=2]);

- Strings now support the subscript operator to extract individual

characters and substrings (e.g., ``s[4]``, ``s[1:5]``). The index

expression can take up to two indices for the start and end index of

the substring to return (e.g. ``mystring[1:3]``).

- Functions now support default parameters, e.g.::

global foo: function(s: string, t: string &default="abc", u: count &default=0);

- Scripts can now use two new "magic constants" ``@DIR`` and

``@FILENAME`` that expand to the directory path of the current

script and just the script file name without path, respectively.

- ``ssl.log`` now also records the subject client and issuer

certificates.

- The ASCII writer can now output CSV files on a per filter basis.

- New SQLite reader and writer plugins for the logging framework allow

to read/write persistent data from on disk SQLite databases.

- A new packet filter framework supports BPF-based load-balancing,

shunting, and sampling; plus plugin support to customize filters

dynamically.

- Bro now provides Bloom filters of two kinds: basic Bloom filters

supporting membership tests, and counting Bloom filters that track

the frequency of elements. The corresponding functions are::

bloomfilter_basic_init(fp: double, capacity: count, name: string &default=""): opaque of bloomfilter

bloomfilter_basic_init2(k: count, cells: count, name: string &default=""): opaque of bloomfilter

bloomfilter_counting_init(k: count, cells: count, max: count, name: string &default=""): opaque of bloomfilter

bloomfilter_add(bf: opaque of bloomfilter, x: any)

bloomfilter_lookup(bf: opaque of bloomfilter, x: any): count

bloomfilter_merge(bf1: opaque of bloomfilter, bf2: opaque of bloomfilter): opaque of bloomfilter

bloomfilter_clear(bf: opaque of bloomfilter)

See ``src/probabilistic/bloom-filter.bif``, or the online

documentation, for full documentation.

- Bro now provides a probabilistic data structure for computing

"top k" elements. The corresponding functions are::

topk_init(size: count): opaque of topk

topk_add(handle: opaque of topk, value: any)

topk_get_top(handle: opaque of topk, k: count)

topk_count(handle: opaque of topk, value: any): count

topk_epsilon(handle: opaque of topk, value: any): count

topk_size(handle: opaque of topk): count

topk_sum(handle: opaque of topk): count

topk_merge(handle1: opaque of topk, handle2: opaque of topk)

topk_merge_prune(handle1: opaque of topk, handle2: opaque of topk)

See ``src/probabilistic/top-k.bif``, or the online documentation,

for full documentation.

- Bro now provides a probabilistic data structure for computing set

cardinality, using the HyperLogLog algorithm. The corresponding

functions are::

hll_cardinality_init(err: double, confidence: double): opaque of cardinality

hll_cardinality_add(handle: opaque of cardinality, elem: any): bool

hll_cardinality_merge_into(handle1: opaque of cardinality, handle2: opaque of cardinality): bool

hll_cardinality_estimate(handle: opaque of cardinality): double

hll_cardinality_copy(handle: opaque of cardinality): opaque of cardinality

See ``src/probabilistic/cardinality-counter.bif``, or the online

documentation, for full documentation.

- ``base/utils/exec.bro`` provides a module to start external

processes asynchronously and retrieve their output on termination.

``base/utils/dir.bro`` uses it to monitor a directory for changes,

and ``base/utils/active-http.bro`` for providing an interface for

querying remote web servers.

- BroControl can now pin Bro processes to CPUs on supported platforms:

To use CPU pinning, a new per-node option ``pin_cpus`` can be

specified in node.cfg if the OS is either Linux or FreeBSD.

- BroControl now returns useful exit codes. Most BroControl commands

return 0 if everything was OK, and 1 otherwise. However, there are

a few exceptions. The "status" and "top" commands return 0 if all Bro

nodes are running, and 1 if not all nodes are running. The "cron"

command always returns 0 (but it still sends email if there were any

problems). Any command provided by a plugin always returns 0.

- BroControl now has an option "env_vars" to set Bro environment variables.

The value of this option is a comma-separated list of environment variable

assignments (e.g., "VAR1=value, VAR2=another"). The "env_vars" option

can apply to all Bro nodes (by setting it in broctl.cfg), or can be

node-specific (by setting it in node.cfg). Environment variables in

node.cfg have priority over any specified in broctl.cfg.

- BroControl now supports load balancing with PF_RING while sniffing

multiple interfaces. Rather than assigning the same PF_RING cluster ID

to all workers on a host, cluster ID assignment is now based on which

interface a worker is sniffing (i.e., all workers on a host that sniff

the same interface will share a cluster ID). This is handled by

BroControl automatically.

- BroControl has several new options: MailConnectionSummary (for

disabling the sending of connection summary report emails),

MailAlarmsInterval (for specifying a different interval to send alarm

summary emails), CompressCmd (if archived log files will be compressed,

this specifies the command that will be used to compress them),

CompressExtension (if archived log files will be compressed, this

specifies the file extension to use).

- BroControl comes with its own test-suite now. ``make test`` in

``aux/broctl`` will run it.

In addition to these, Bro 2.2 comes with a large set of smaller

extensions, tweaks, and fixes across the whole code base, including

most submodules.

Changed Functionality

---------------------

- Previous versions of ``$prefix/share/bro/site/local.bro`` (where

"$prefix" indicates the installation prefix of Bro), aren't compatible

with Bro 2.2. This file won't be overwritten when installing over a

previous Bro installation to prevent clobbering users' modifications,

but an example of the new version is located in

``$prefix/share/bro/site/local.bro.example``. So if no modification

has been done to the previous local.bro, just copy the new example

version over it, else merge in the differences. For reference,

a common error message when attempting to use an outdated local.bro

looks like::

fatal error in /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro, line 41: BroType::AsRecordType (table/record) (set[record { min:record { major:count; minor:count; minor2:count; minor3:count; addl:string; }; max:record { major:count; minor:count; minor2:count; minor3:count; addl:string; }; }])

- The type of ``Software::vulnerable_versions`` changed to allow

more flexibility and range specifications. An example usage:

.. code:: bro

const java_1_6_vuln = Software::VulnerableVersionRange(

$max = Software::Version($major = 1, $minor = 6, $minor2 = 0, $minor3 = 44)

);

const java_1_7_vuln = Software::VulnerableVersionRange(

$min = Software::Version($major = 1, $minor = 7),

$max = Software::Version($major = 1, $minor = 7, $minor2 = 0, $minor3 = 20)

);

redef Software::vulnerable_versions += {

["Java"] = set(java_1_6_vuln, java_1_7_vuln)

};

- The interface to extracting content from application-layer protocols

(including HTTP, SMTP, FTP) has changed significantly due to the

introduction of the new file analysis framework (see above).

- Removed the following, already deprecated, functionality:

* Scripting language:

- ``&disable_print_hook attribute``.

* BiF functions:

- ``parse_dotted_addr()``, ``dump_config()``,

``make_connection_persistent()``, ``generate_idmef()``,

``split_complete()``

- ``md5_*``, ``sha1_*``, ``sha256_*``, and ``entropy_*`` have

all changed their signatures to work with opaque types (see

above).

- Removed a now unused argument from ``do_split`` helper function.

- ``this`` is no longer a reserved keyword.

- The Input Framework's ``update_finished`` event has been renamed to

``end_of_data``. It will now not only fire after table-reads have

been completed, but also after the last event of a whole-file-read

(or whole-db-read, etc.).

- Renamed the option defining the frequency of alarm summary mails to

``Logging::default_alarm_mail_interval``. When using BroControl, the

value can now be set with the new broctl.cfg option

``MailAlarmsInterval``.

- We have completely rewritten the ``notice_policy`` mechanism. It now

no longer uses a record of policy items but a ``hook``, a new

language element that's roughly equivalent to a function with

multiple bodies (see above). For existing code, the two main changes

are:

- What used to be a ``redef`` of ``Notice::policy`` now becomes a

hook implementation. Example:

Old::

redef Notice::policy += {

[$pred(n: Notice::Info) = {

return n$note == SSH::Login && n$id$resp_h == 10.0.0.1;

},

$action = Notice::ACTION_EMAIL]

};

New::

hook Notice::policy(n: Notice::Info)

{

if ( n$note == SSH::Login && n$id$resp_h == 10.0.0.1 )

add n$actions[Notice::ACTION_EMAIL];

}

- notice() is now likewise a hook, no longer an event. If you

have handlers for that event, you'll likely just need to change

the type accordingly. Example:

Old::

event notice(n: Notice::Info) { ... }

New::

hook notice(n: Notice::Info) { ... }

- The ``notice_policy.log`` is gone. That's a result of the new notice

policy setup.

- Removed the ``byte_len()`` and ``length()`` bif functions. Use the

``|...|`` operator instead.

- The ``SSH::Login`` notice has been superseded by an corresponding

intelligence framework observation (``SSH::SUCCESSFUL_LOGIN``).

- ``PacketFilter::all_packets`` has been replaced with

``PacketFilter::enable_auto_protocol_capture_filters``.

- We removed the BitTorrent DPD signatures pending further updates to

that analyzer.

- In previous versions of BroControl, running "broctl cron" would create

a file ``$prefix/logs/stats/www`` (where "$prefix" indicates the

installation prefix of Bro). Now, it is created as a directory.

Therefore, if you perform an upgrade install and you're using BroControl,

then you may see an email (generated by "broctl cron") containing an

error message: "error running update-stats". To fix this problem,

either remove that file (it is not needed) or rename it.

- Due to lack of maintenance the Ruby bindings for Broccoli are now

deprecated, and the build process no longer includes them by

default. For the time being, they can still be enabled by

configuring with ``--enable-ruby``, however we plan to remove

Broccoli's Ruby support with the next Bro release.

Bro 2.1

=======

New Functionality

-----------------

- Bro now comes with extensive IPv6 support. Past versions offered

only basic IPv6 functionality that was rarely used in practice as it

had to be enabled explicitly. IPv6 support is now fully integrated

into all parts of Bro including protocol analysis and the scripting

language. It's on by default and no longer requires any special

configuration.

Some of the most significant enhancements include support for IPv6

fragment reassembly, support for following IPv6 extension header

chains, and support for tunnel decapsulation (6to4 and Teredo). The

DNS analyzer now handles AAAA records properly, and DNS lookups that

Bro itself performs now include AAAA queries, so that, for example,

the result returned by script-level lookups is a set that can

contain both IPv4 and IPv6 addresses. Support for the most common

ICMPv6 message types has been added. Also, the FTP EPSV and EPRT

commands are now handled properly. Internally, the way IP addresses

are stored has been improved, so Bro can handle both IPv4

and IPv6 by default without any special configuration.

In addition to Bro itself, the other Bro components have also been

made IPv6-aware by default. In particular, significant changes were

made to trace-summary, PySubnetTree, and Broccoli to support IPv6.

- Bro now decapsulates tunnels via its new tunnel framework located in

scripts/base/frameworks/tunnels. It currently supports Teredo,

AYIYA, IP-in-IP (both IPv4 and IPv6), and SOCKS. For all these, it

logs the outer tunnel connections in both conn.log and tunnel.log,

and then proceeds to analyze the inner payload as if it were not

tunneled, including also logging that session in conn.log. For

SOCKS, it generates a new socks.log in addition with more

information.

- Bro now features a flexible input framework that allows users to

integrate external information in real-time into Bro while it's

processing network traffic. The most direct use-case at the moment

is reading data from ASCII files into Bro tables, with updates

picked up automatically when the file changes during runtime. See

doc/input.rst for more information.

Internally, the input framework is structured around the notion of

"reader plugins" that make it easy to interface to different data

sources. We will add more in the future.

- BroControl now has built-in support for host-based load-balancing

when using either PF_RING, Myricom cards, or individual interfaces.

Instead of adding a separate worker entry in node.cfg for each Bro

worker process on each worker host, it is now possible to just

specify the number of worker processes on each host and BroControl

configures everything correctly (including any neccessary enviroment

variables for the balancers).

This change adds three new keywords to the node.cfg file (to be used

with worker entries): lb_procs (specifies number of workers on a

host), lb_method (specifies what type of load balancing to use:

pf_ring, myricom, or interfaces), and lb_interfaces (used only with

"lb_method=interfaces" to specify which interfaces to load-balance

on).

- Bro's default ASCII log format is not exactly the most efficient way

for storing and searching large volumes of data. An alternatives,

Bro now comes with experimental support for two alternative output

formats:

* DataSeries: an efficient binary format for recording structured

bulk data. DataSeries is developed and maintained at HP Labs.

See doc/logging-dataseries for more information.

* ElasticSearch: a distributed RESTful, storage engine and search

engine built on top of Apache Lucene. It scales very well, both

for distributed indexing and distributed searching. See

doc/logging-elasticsearch.rst for more information.

Note that at this point, we consider Bro's support for these two

formats as prototypes for collecting experience with alternative

outputs. We do not yet recommend them for production (but welcome

feedback!)

Changed Functionality

---------------------

The following summarizes the most important differences in existing

functionality. Note that this list is not complete, see CHANGES for

the full set.

- Changes in dependencies:

* Bro now requires CMake >= 2.6.3.

* On Linux, Bro now links in tcmalloc (part of Google perftools)

if found at configure time. Doing so can significantly improve

memory and CPU use.

On the other platforms, the new configure option

--enable-perftools can be used to enable linking to tcmalloc.

(Note that perftools's support for non-Linux platforms may be

less reliable).

- The configure switch --enable-brov6 is gone.

- DNS name lookups performed by Bro now also query AAAA records. The

results of the A and AAAA queries for a given hostname are combined

such that at the scripting layer, the name resolution can yield a

set with both IPv4 and IPv6 addresses.

- The connection compressor was already deprecated in 2.0 and has now

been removed from the code base.

- We removed the "match" statement, which was no longer used by any of

the default scripts, nor was it likely to be used by anybody anytime

soon. With that, "match" and "using" are no longer reserved keywords.

- The syntax for IPv6 literals changed from "2607:f8b0:4009:802::1012"

to "[2607:f8b0:4009:802::1012]". When an IP address variable or IP

address literal is enclosed in pipes (for example,

``|[fe80::db15]|``) the result is now the size of the address in

bits (32 for IPv4 and 128 for IPv6).

- Bro now spawns threads for doing its logging. From a user's

perspective not much should change, except that the OS may now show

a bunch of Bro threads.

- We renamed the configure option --enable-perftools to

--enable-perftools-debug to indicate that the switch is only relevant

for debugging the heap.

- Bro's ICMP analyzer now handles both IPv4 and IPv6 messages with a

joint set of events. The `icmp_conn` record got a new boolean field

'v6' that indicates whether the ICMP message is v4 or v6.

- Log postprocessor scripts get an additional argument indicating the

type of the log writer in use (e.g., "ascii").

- BroControl's make-archive-name script also receives the writer

type, but as its 2nd(!) argument. If you're using a custom version

of that script, you need to adapt it. See the shipped version for

details.

- Signature files can now be loaded via the new "@load-sigs"

directive. In contrast to the existing (and still supported)

signature_files constant, this can be used to load signatures

relative to the current script (e.g., "@load-sigs ./foo.sig").

- The options "tunnel_port" and "parse_udp_tunnels" have been removed.

Bro now supports decapsulating tunnels directly for protocols it

understands.

- ASCII logs now record the time when they were opened/closed at the

beginning and end of the file, respectively (wall clock). The

options LogAscii::header_prefix and LogAscii::include_header have

been renamed to LogAscii::meta_prefix and LogAscii::include_meta,

respectively.

- The ASCII writers "header_*" options have been renamed to "meta_*"

(because there's now also a footer).

- Some built-in functions have been removed: "addr_to_count" (use

"addr_to_counts" instead), "bro_has_ipv6" (this is no longer

relevant because Bro now always supports IPv6), "active_connection"

(use "connection_exists" instead), and "connection_record" (use

"lookup_connection" instead).

- The "NFS3::mode2string" built-in function has been renamed to

"file_mode".

- Some built-in functions have been changed: "exit" (now takes the

exit code as a parameter), "to_port" (now takes a string as

parameter instead of a count and transport protocol, but

"count_to_port" is still available), "connect" (now takes an