This document summarizes the most important changes in the current Zeek

release. For an exhaustive list of changes, see the ``CHANGES`` file

(note that submodules, such as Broker, come with their own ``CHANGES``.)

Zeek 7.1.0

==========

Breaking Changes

----------------

* The ``OpaqueVal::DoSerialize`` and ``OpaqueVal::DoUnserialize`` methods were

marked as deprecated in v7.0 and have now been removed as per the Zeek

deprecation policy. Plugins that were overriding these methods and were not

updated will fail to compile. Those plugins should be updated to override the

new ``OpaqueVal::DoSerializeData`` and ``OpaqueVal::DoUnserializeData``

methods.

* Certain internal methods on the broker and logging classes have been changed to

accept std::vector<threading::Value> parameters instead of threading::Value**

to leverage automatic memory management, reduce the number of allocations

and use move semantics to express ownership.

The DoWrite() and HookLogWrite() methods which can be provided by plugins

are not affected by this change, so we keep backwards compatibility with

existing log writers.

* ``Func::Name()`` was deprecated, use ``Func::GetName()`` instead.

New Functionality

-----------------

- Zeek now includes a PostgreSQL protocol analyzer. This analyzer is enabled

by default. The analyzer's events and its ``postgresql.log`` should be

considered preliminary and experimental until the arrival of Zeek's next

long-term-stable release (8.0).

If you observe unusually high CPU consumption or other issues due to this

analyzer being enabled by default, the easiest way to disable it is via the

``Analyzer::disabled_analyzers`` const as follows:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_POSTGRESQL,

};

If you observe PostgreSQL traffic in your environment, please provide feedback

about the analyzer and structure of the new log.

* The LDAP analyzer now supports handling of non-sealed GSS-API WRAP tokens.

* StartTLS support was added to the LDAP analyzer. The SSL analyzer is enabled

for connections where client and server negotiate to TLS through the extended

request/response mechanism.

* The ``unknown_protocols()`` event now includes the name of all packet

analyzer used for processing the packet when the event is raised. The

``unknown_protocol.log`` file was extended to include this information.

* The MySQL analyzer now generates a ``mysql_user_change()`` event when

the user changes mid-session via the ``COM_USER_CHANGE`` command.

* The DNS analyzer was extended to support TKEY RRs (RFC 2390). A corresponding

``dns_TKEY`` event was added.

* The ``signature_match()`` and custom signature events now receive the end of

match offset within the ``data`` parameter as an optional parameter named

``end_of_match``.

event signature_match(state: signature_state, msg: string, data: string, end_of_match: count);

Changed Functionality

---------------------

* Heuristics for parsing SASL encrypted and signed LDAP traffic have been

made more strict and predictable. Please provide input if this results in

less visibility in your environment.

* The MySQL analyzer has been improved to better support plugin authentication

mechanisms, like caching_sha2_password, as well as recognizing MySQL query

attributes.

* The ``mysql.log`` for user change commands will contain *just* the username

instead of the remaining parts of the command, including auth plugin data.

* The POP3 parser has been hardened to avoid unbounded state growth in the

face of one-sided traffic capture or when enabled for non-POP3 traffic.

Concretely, the Redis protocol's AUTH mechanism enables the POP3 analyzer

for such connections through DPD.

* Batching and flushing for local log writers can now be controlled via the

options ``Log::flush_interval`` and ``Log::write_buffer_size``. Previously

the ``Threading::heartbeat_interval`` was used for flushing and the buffer

size fixed at 1000.

* Logging of the FTP PASS command in ``ftp.log`` now honors ``FTP::default_capture_password``

and the password is blanked with "<hidden>". Previously, the argument for the PASS

command would be logged in clear.

* The ASCII input reader now suppresses warnings for consecutive invalid lines,

producing a summary of total suppressions once a valid line is encountered.

* The `Telemetry::sync()` hook is now invoked on demand. Either when the metrics

of a node are scraped via the Prometheus HTTP endpoint, or one of the collect

methods is invoked from Zeek script.

Removed Functionality

---------------------

Deprecated Functionality

------------------------

Zeek 7.0.0

==========

Breaking Changes

----------------

- The Telemetry framework has had a major rework, and includes a number of

breaking changes. The biggest change is a move towards a Prometheus-first

model. This removes the internal aggregation of metrics from nodes onto the

manager node, replacing it with a Prometheus service discovery endpoint. The

usage of this endpoint is described in the updated documentation for the

Telemetry framework. Using this endpoint also requires adding a new

``metrics_port`` field to each node in the cluster configuration, denoting

what port to connect to for each node.

All of the metrics-related script-level options, type, and methods have been

moved to the Telemetry framework:

* Option ``Broker::metrics_port` is now ``Telemetry::metrics_port``

* Option ``Broker::metrics_export_endpoint_name`` is now ``Telemetry::metrics_endpoint_name``

The following options have been removed:

* ``Broker::metrics_export_interval``

* ``Broker::metrics_export_topic``

* ``Broker::metrics_import_topics``

* ``Broker::metrics_export_prefixes``

The ``unit`` field has been removed from the telemetry log.

All of the ``BROKER_METRICS_*`` environment variables have been removed. Use

the equivalent script-level variables for setting configuration settings. For

cluster settings, a new ``metrics_port`` option was added to the ``node.cfg``

options and can be used to set the port number. The cluster layout record

defined in ``cluster-layout.zeek`` has the same option, and it can be

automatically populated by ``zeekctl`` during the ``install`` or ``deploy``

commands.

The instruments that previously supported ``count`` in scripts and ``int64_t``

in C++ were removed in favor of only providing ``double`` versions. Prometheus

only handles ``double`` underneath the Zeek code, so it makes sense to only

support it directly in Zeek as well. This also simplifies the code

significantly.

The ``is_sum`` argument has been removed from the constructors/creation

methods for all of the instruments. This again follows how Prometheus works,

where ``counter`` instruments are always considered sums and ``gauge``

instruments are not. ``Histogram`` instruments don't have the concept of

summing.

- Zeekctl now sets `FileExtract::prefix` to `spool/extract_files/<node>` to avoid

deletion of extracted files when stopping worker nodes. To revert to the

previous behavior, set `FileExtractDir` to an empty string in `zeekctl.cfg`.

If you never enabled Zeek's file extraction functionality, there's no impact.

New Functionality

-----------------

- Support ``delete`` on tables, sets and vectors to clear their contents.

global v = vector(1, 2, 3);

delete v;

assert |v| == 0;

- A new helper function ``can_load()`` backed by a new bif ``find_in_zeekpath()``

was added to determine if a non-relative ``@load`` directive might work. This

can be used to guard ``@load`` directives when script packages may or may not

be installed.

@if ( can_load("my-package") )

@load my-package

@endif

- Zeek packagers can now include a "local" addition into Zeek's version string.

Configure your Zeek build with ``--localversion=XXX`` to add the provided

string, dash-separated, at the end of Zeek's regular version string. The build

setup will refuse strings that contain dashes, to avoid confusing the version

components. For debug builds, the ``-debug`` suffix remains at the end. For

example, the version string on a debug-enabled build with local addition "XXX"

might be "7.0.0-dev.114-XXX-debug". For Docker builds, the ``LOCALVERSION``

environment variable configures the addition.

- SMB2 packets containing multiple PDUs now correctly parse all of the headers,

instead of just the first one and ignoring the rest. This may cause increased

CPU load on SMB2-heavy networks.

- The new built-in function ``lookup_connection_analyzer_id()`` retrieves the

numeric identifier of an analyzer associated with a connection. This enables

the use of the ``disable_analyzer()`` BiF outside of the analyzer

confirmation/violation events that have so far been the only providers of

those identifiers. For example, this allows the suppression of an analyzer

from the outset for specific connections:

event connection_established(c: connection):

{

if ( no_http_for_this_conn_wanted(c) )

{

local aid = lookup_connection_analyzer_id(c$id, Analyzer::ANALYZER_HTTP);

if ( aid > 0 )

disable_analyzer(c$id, aid, T, T);

}

}

Use ``Analyzer::get_tag()`` if you need to obtain an analyzer's tag from its

name (such as "HTTP").

- The ``from_json()`` function now supports ingesting JSON representations of

tables as produced by the ``to_json()`` function. It now also supports reading

the object-based representation of ports that ``to_json()`` generates for that

Zeek type.

- The ``analyzer.log`` now optionally supports logging of disabled analyzers

through the new option ``Analyzer::logging::include_disabling``.

Changed Functionality

---------------------

- The ``ftp.log`` fuid field is now cleared after handling a command with a fuid

associated with it. Previously, fuid was sticky and any subsequent FTP command

would reproduce the same fuid, even if the command itself did not result in

a file transfer over a data connection (e.g., CWD, DEL, PASV, SIZE).

- The type_name field populated by ``global_ids()`` now aligns with the value

returned by ``type_name()`` for each identifier. E.g, ``Site::local_nets``

has a type_name of ``set[subnet]`` rather than ``table``.

- The ISO 9660 file signature has been moved into the policy directory. The

signature has previously been non-functional due to implicit anchoring. Further,

this signature requires users to significantly increase their

``default_file_bof_buffer_size``. Users can now enable this signature by loading

``frameworks/signatures/iso-9660`` which also increases the BOF buffer sufficiently.

Note, doing so may increase memory and CPU usage significantly.

- The ``val_footprint()`` BiF now factors in the size of strings when reporting

footprints, roughly equating a string's size with the number of elements

comparable to that length. As before, footprints are not meant to be precise

but mainly for providing comparisons, which is why this is not a breaking

change.

- The tuning/defaults policy has been deprecated and will be removed in

v7.1. This policy was already being loaded by default via local.zeek. The

settings contained within have become the overall defaults for Zeek now,

instead of having to load the policy. The two changes here are that fragments

now timeout after 5 minutes by default instead of no timeout, and extracted

files now have a default size limit of 100MB instead of unlimited.

- If a Spicy protocol analyzers feeds data into file analysis, it now

needs to call Zeek's `Files::register_protocol()` and provide a

callback for computing file handles. If that's missing, Zeek will

issue a warning. While this was not necessary in previous versions,

it aligns with the same requirement for traditional analyzers and

enables customizing file handles for protocol-specific semantics.

- The Supervisor's API now returns NodeConfig records with a cluster table whose

ClusterEndpoints have a port value of 0/unknown, rather than 0/tcp, to

indicate that the node in question has no listening port.

Removed Functionality

---------------------

Deprecated Functionality

------------------------

- The ``--disable-archiver`` configure flag no longer does anything and will be

removed in 7.1. zeek-archiver has moved into the zeek-aux repository.

- The policy/frameworks/telemetry/prometheus.zeek script has been deprecated

and will be removed with Zeek 7.1. Setting the ``metrics_port`` field on a

``Cluster::Node`` implies listening on that port and exposing telemetry

in Prometheus format.

Zeek 6.2.0

==========

We would like to thank Anthony Verez (netantho), Bijoy Das (mute019), Jan

Grashöfer (J-Gras), Matti Bispham (mbispham), Phil Rzewski (philrz), and

xb-anssi for their contributions to this release.

Breaking Changes

----------------

- The methods ``Dispatcher::Lookup()`` and ``Analyzer::Lookup()`` in the packet_analysis

namespace were changed to return a reference to a std::shared_ptr instead of a copy

for performance reasons.

- Zeek's ``OPENSSL_INCLUDE_DIR`` is not automatically added to an external plugin's

include path anymore. A plugin using OpenSSL functionality directly can use the

following explicit entry to re-use Zeek's ``OPENSSL_INCLUDE_DIR``:

zeek_add_plugin(

Namespace Name

INCLUDE_DIRS "${OPENSSL_INCLUDE_DIR}"

SOURCES ...

)

- The "segment_profiling" functionality and ``load_sample`` event have been removed

without deprecation. This functionality was unmaintained and not known to be used.

- Certain ``ldap.log`` and ``ldap_search.log`` fields have been renamed from

plural to singular and their types changed to scalars. This maps better onto

the expected request-response protocol used between client and server. Additionally,

it removes the burden of working with non-scalar columns from downstream systems.

Specifically, for ``ldap.log``:

* ``arguments: vector of string`` is now ``argument: string``

* ``diagnostic_messages: vector of string`` is now ``diagnostic_message: string``

* ``objects: vector of string`` is now ``object: string``

* ``opcodes: set[string]`` is now ``opcode: string``

* ``results: set[string]`` is now ``result: string``

For ``ldap_search.log``, the following fields were changed:

* ``base_objects: vector of string`` is now ``base_object: string``

* ``derefs: set[string]`` is now ``deref_aliases: string``

* ``diagnostic_messages: vector of string`` is now ``diagnostic_message: string``

* ``results: set[string]`` is now ``result: string``

* ``scopes: set[string]`` is now ``scope: string``

In the unlikely scenario that a request-response pair with the same message

identifier is observed, containing different values for certain fields, new

weirds are raised and will appear in ``weird.log``, including the old and new

values as well as the LDAP message identifier. The value within the LDAP logs

will be the most recently observed one.

- BIF methods now return a ``ValPtr`` directly instead of a ``BifReturnVal`` object

which was just a thin wrapper around ``ValPtr``. This may cause compilation errors

in C++ code that was calling BIF methods directly.

New Functionality

-----------------

- The table type was extended to allow parallel regular expression matching

when a table's index is a pattern. Indexing such tables yields a vector

containing all values of matching patterns for keys of type string.

As an example, the following snippet outputs ``[a, a or b], [a or b]``.

global tbl: table[pattern] of string;

tbl[/a/] = "a";

tbl[/a|b/] = "a or b";

tbl[/c/] = "c";

print tbl["a"], tbl["b"];

Depending on the patterns and input used for matching, memory growth may

be observed over time as the underlying DFA is constructed lazily. Users are

advised to test with realistic and adversarial input data with focus on

memory growth. The DFA's state can be reset by removal/addition of a single

pattern. For observability, a new bif ``table_pattern_matcher_stats()``

can be used to gather ``MatcherStats``.

- Support for delaying log writes.

The logging framework offers two new functions ``Log::delay()`` and ``Log::delay_finish()``

to delay a ``Log::write()`` operation. This new functionality allows delaying of

a specific log record within the logging pipeline for a variable but bounded

amount of time. This can be used, for example, to query and wait for additional

information to attach to the pending record, or even change its final verdict.

Conceptually, delaying a log record happens after the execution of the global

``Log::log_stream_policy`` hook for a given ``Log::write()`` and before the

execution of filter policy hooks. Any mutation of the log record within the

delay period will be visible to filter policy hooks. Calling ``Log::delay()``

is currently only allowed within the context of the ``Log::log_stream_policy`` hook

for the active ``Log::write()` operation (or during the execution of post delay callbacks).

While this may appear restrictive, it makes it explicit which ``Log::write()``

operation is subject to the delay.

Interactions, semantics and conflicts of this feature when writing the same

log record multiple times to the same or different log streams need to be taken

into consideration by script writers.

Given this is the first iteration of this feature, feedback around usability and

use-cases that aren't covered are more than welcome.

- A WebSocket analyzer has been added together with a new ``websocket.log``.

The WebSocket analyzer is instantiated when a WebSocket handshake over HTTP is

recognized. By default, the payload of WebSocket messages is fed into Zeek's dynamic

protocol detection framework, possibly discovering and analyzing tunneled protocols.

The format of the log and the event semantics should be considered preliminary until

the arrival of the next long-term-stable release (7.0).

To disable the analyzer in case of fatal errors or unexpected resource usage,

use the ``Analyzer::disabled_analyzers`` pattern:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_WEBSOCKET,

};

- The SMTP analyzer was extended to recognize and properly handle the BDAT command

from RFC 3030. This improves visibility into the SMTP protocol when mail agents

and servers support and use this extension.

- The event keyword in signatures was extended to support choosing a custom event

to raise instead of ``signature_match()``. This can be more efficient in certain

scenarios compared to funneling every match through a single event.

The new syntax is to put the name of the event before the string used for the

``msg`` argument. As an extension, it is possible to only provide an event name,

skipping ``msg``. In this case, the framework expects the event's parameters to

consist of only state and data as follows:

signature only-event {

payload /.*root/

event found_root

}

event found_root(state: signature_state, data: string) { }

Using the ``msg`` parameter with a custom event looks as follows. The custom

event's parameters need to align with those for ``signature_match()` event:

signature event-with-msg {

payload /.*root/

event found_root_with_msg "the-message"

}

event found_root_with_msg(state: signature_state, msg: string, data: string) { }

Note, the message argument can currently still be specified as a Zeek identifier

referring to a script-level string value. If used, this is disambiguated behind

the scenes for the first variant. Specifying ``msg`` as a Zeek identifier has

been deprecated with the new event support and will be removed in the future.

Note that matches for signatures with custom events will not be recorded in

``signatures.log``. This log is based on the generation of ``signature_match()``

events.

- The QUIC analyzer has been extended to support analyzing QUIC Version 2

INITIAL packets (RFC 9369). Additionally, prior draft and some of

Facebook's mvfst versions are supported. Unknown QUIC versions will now be

reported in ``quic.log`` as an entry with a ``U`` history field.

- Conditional directives (``@if``, ``@ifdef``, ``@ifndef``, ``@else`` and

``@endif``) can now be placed within a record's definition to conditionally

define or extend a record type's fields.

type r: record {

c: count;

@if ( cond )

d: double;

@else

d: count;

@endif

};

Note that generally you should prefer record extension in conditionally loaded

scripts rather than using conditional directives in the original record definition.

- The 'X' code can now appear in a connection's history. It is meant to indicate

situations where Zeek stopped analyzing traffic due to exceeding certain limits or

when encountering unknown/unsupported protocols. Its first use is to indicate

``Tunnel::max_depth`` being exceeded.

- A new ``Intel::seen_policy`` hook has been introduced to allow intercepting

and changing ``Intel::seen` behavior:

hook Intel::seen_policy(s: Intel::Seen, found: bool)

- A new ``NetControl::rule_added_policy`` hook has been introduced to allow modification

of NetControl rules after they have been added.

- The IP geolocation / ASN lookup features in the script layer provide better

configurability. The file names of MaxMind databases are now configurable via

the new ``mmdb_city_db``, ``mmdb_country_db``, and ``mmdb_asn_db`` constants,

and the previously hardwired fallback search path when not using an

``mmdb_dir`` value is now adjustable via the ``mmdb_dir_fallbacks``

vector. Databases opened explicitly via the ``mmdb_open_location_db`` and

``mmdb_open_asn_db`` functions now behave more predictably when updated or

removed. For details, see:

https://docs.zeek.org/en/master/customizations.html#address-geolocation-and-as-lookups

- The ``zeek-config`` script now provides a set of ``--have-XXX`` checks for

features optionally compiled in. Each check reports "yes"/"no" to stdout and

exits with 0/1, respectively.

Changed Functionality

---------------------

- The ``split_string`` family of functions now respect the beginning-of-line ^ and

end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere

in the input string.

- The ``sub()`` and ``gsub()` functions now respect the beginning-of-line ^ and

end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere

in the input string.

- Ed25519 and Ed448 DNSKEY and RRSIG entries do not cause weirds anymore.

- The OpenSSL references in ``digest.h`` and ``OpaqueVal.h`` headers have been

hidden to avoid unneeded dependencies on OpenSSL headers. Plugins using the

detail API from ``digest.h`` to compute hashes likely need to accommodate for

this change.

- The ``Tunnel::max_depth`` default was changed from 2 to 4 allowing for more than

two encapsulation layers. Two layers are already easily reached in AWS GLB

environments.

- Nested MIME message analysis is now capped at a maximum depth of 100 to prevent

unbounded MIME message nesting. This limit is configurable with ``MIME::max_depth``.

A new weird named ``exceeded_mime_max_depth`` is reported when reached.

- The ``netcontrol_catch_release.log`` now contains a plugin column that shows which

plugin took an action. The logs also contain information when errors or existing

rules are encountered.

- The ``Cluster::PoolSpec`` record no longer provides default values for its

``topic`` and ``node_type`` fields, since defaults don't fit their intended

use and looked confusing in generated documentation.

Removed Functionality

---------------------

- Zeek no longer automatically subscribes to topics prefixed with "bro/" whenever

subscribing to topics prefixed with "zeek/". This was a leftover backward-

compatibility step in the Broker code that should have been removed long ago.

Deprecated Functionality

------------------------

- The virtual functions ``DoSerialize``and ``DoUnserialize`` of the ``OpaqueVal``

(and ``BloomFilter``) class will be removed with Zeek 7.1. Unfortunately, code

implementing the deprecated methods does not produce compiler warnings.

Plugin authors implementing an ``OpaqueVal`` subclass need to convert to

``DoSerializeData`` and ``DoUnserializeData``:

* ``std::optional<BrokerData> OpaqueVal::DoSerializeData() const``

* ``bool OpaqueVal::DoUnserializeData(BrokerDataView data)``

When overriding ``DoSerializeData()``, return ``std::nullopt`` (or a

default-constructed ``optional``) for values that cannot be serialized.

Otherwise, the canonical way to create a ``BrokerData`` for serialization is

by using a ``BrokerListBuilder``. For example, creating a ``BrokerData`` that

contains ``true`` and the count ``42`` could be implemented as follows:

BrokerListBuilder builder;

builder.Add(true);

builder.AddCount(42u);

return std::move(builder).Build();

Please refer to the respective class documentation for a full list of member

functions on ``BrokerListBuilder`` and ``BrokerDataView``.

For plugins that are using the macro ``DECLARE_OPAQUE_VALUE`` to generate the

function prototypes for the serialization functions: please use

``DECLARE_OPAQUE_VALUE_DATA`` instead to generate prototypes for the new API.

Plugin authors that need to support multiple Zeek versions can use the

``ZEEK_VERSION_NUMBER`` macro to conditionally implement the new and old

methods. Provide the new versions with Zeek 6.2 (60200) or later, otherwise

keep the old signature. The default implementations for the new functions

as used by Zeek will call the old signatures and convert the results.

- The ``Cluster::Node$interface`` field has been deprecated. It's essentially

unneeded, unused and not a reliable way to gather the actual interface used

by a worker. In Zeekctl deployments the field will be populated until its

removal. The ``packet_source()`` bif should be used on worker processes to

gather information about the interface.

- The ``policy/misc/load-balancing`` script has been deprecated in favor of

AF_PACKET PF_RING, Netmap or other NIC specific load balancing approaches.

- Time machine related enums, options and fields have been marked for removal.

- The ``check_for_unused_event_handlers`` options the related ``UsedHandlers()``,

``UnusedHandlers()`` and their related ``SetUsed()`` and ``Used()`` methods

have been marked for removal. The feature of finding unused event handlers is

provided by default via the ``UsageAnalyzer`` component.

- Using a Zeek identifier for the ``msg`` argument within a signatures's ``event``

keyword has been deprecated.

Zeek 6.1.0

==========

Breaking Changes

----------------

- ``assert`` is now a reserved keyword for the new ``assert`` statement.

- The ``__bro_plugin__`` file that gets generated as part of plugin builds was

renamed to ``__zeek_plugin__``. This will affect the ability for older

versions of ``zkg`` to use the ``zkg unload`` and ``zkg load`` commands. This

should only cause breakage for people using a version of ``zkg` that doesn't

come bundled with Zeek (which we generally don't recommend doing).

- Zeek does not traverse into dot directories to find plugins or hlto files

anymore. Any dot directories found below the directories specified in

ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are now skipped. Dot directories

explicitly listed in ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are not

skipped.

- External plugins will fail to configure if their minimum required CMake

version is below 3.15. This was a warning with Zeek 6.0, but has caused user

confusion due to unhelpful error messages around the IN_LIST operator policy.

- The FindBISON, FindOpenSSL, FindPackageHandleStandardArgs, FindPackageMessage,

and SelectLibraryConfigurations cmake files were removed from our cmake

repository in favor of the versions that come with CMake. This should not

cause any breakage, but it is possible in the case that someone was using

these in a plugin.

New Functionality

-----------------

- Zeek now includes the LDAP protocol analyzer from the zeek/spicy-ldap project

(https://github.com/zeek/spicy-ldap). This analyzer is enabled by default. The

analyzer's events and its ``ldap.log`` and ``ldap_search.log`` should be

considered preliminary and experimental until the arrival of Zeek's next

long-term-stable release (7.0).

If you observe unusually high CPU consumption or other issues due to this

analyzer being enabled by default, the easiest way to disable it is via the

``Analyzer::disabled_analyzers`` const as follows:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_LDAP_UDP,

Analyzer::ANALYZER_LDAP_TCP,

};

Please do report issues to us including diagnostic information in case this is

necessary in your environment. We're also open to general feedback about the

structure of the new logs.

- Zeek now includes the QUIC protocol analyzer from the zeek/spicy-quic project

(https://github.com/zeek/spicy-quic). This project is a fork of Fox-IT's

initial implementation (https://github.com/fox-ds/spicy-quic).

As for the LDAP analyzer, the analyzer's events and the new ``quic.log``

should be considered preliminary and experimental until the arrival of Zeek's

next long-term-stable release (7.0). As above, any feedback and contributions

to this analyzer and the new log are welcome.

The analyzer's functionality is limited to decryption of the INITIAL packets

of QUIC version 1. If decryption of these packets is successful, the handshake

data is forwarded to Zeek's SSL analyzer. An ``ssl.log`` entry will appear in

``ssl.log`` for QUIC connections. The entry in the ``conn.log`` will contain

``quic`` and ``ssl`` in the service field.

To disable the analyzer in case of issues, use the following snippet:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_QUIC,

};

- Added a new ``assert`` statement for assertion based testing and asserting

runtime state.

assert <expr: bool>[, <message: string>];

This statement comes with two hooks. First, ``assertion_failure()`` that is

invoked for every failing assert statement. Second, ``assertion_result()``

which is invoked for every assert statement and its outcome. The latter allows

to construct a summary of failing and passing assert statements. Both hooks

receive the location and call stack for the ``assert`` statement via a

``Backtrace`` vector.

A failing assert will abort execution of the current event handler similar to

scripting errors. By default, a reporter error message is logged. Using the

break statement within ``assertion_failure()`` or ``assertion_result()``

allows to suppress the default message.

- Add a new ``&default_insert`` attribute for tables. This behaves as

``&default`` with the addition that the default value is inserted into the

table upon a failed lookup. Particularly for tables with nested container

values, the ``&default`` behavior of not inserting the value can be of little

use.

- The ``from_json()`` function now takes an optional key_func argument to

normalize JSON object key names. This can be useful if the keys in a JSON

object are not valid Zeek identifiers or reserved keywords.

- Module names are now included in ``global_ids()``. Their key in the returned

table is prefixed with "module " and their value will have the ``type_name``

field set to "module".

- Identifiers in the global scope can now be referenced and defined from within

modules by prefixing their names with ``::``. Previously, these required an

explicit ``GLOBAL::`` prefix to be used. Using ``GLOBAL::`` has been

deprecated.

- The ``as`` keyword now supports casting between ``set`` and ``vector`` values

with the same element type. Converting ``set`` values with multiple index

values is not supported. We plan to extend the use of the ``as`` keyword to

support more type conversions in the future.

- Added new packet analyzer to handle PCAP files DLT_PPP link type.

- Fixed appending of ``any`` to ``vector of any``.

- The ModBus analyzer's function support was expanded, with new handling of the

Encapsulation Interface Transport (function 28) And Diagnostics (function 8)

functions. This adds new ``modbus_encap_interface_transport_{request,response}``

and ``modbus_diagnostics_{request,response}`` events.

- The ModBus file record read and write events now provide the full data from

the request and response messages as part of the event data.

- The full PDU length was added to the ``ModBusHeader`` record type passed with

all of the ModBus events.

Changed Functionality

---------------------

- A connection's value is now updated in-place when its directionality is

flipped due to Zeek's heuristics (for example, SYN/SYN-ACK reversal or

protocol specific approaches). Previously, a connection's value was discarded

when flipped, including any values set in a ``new_connection()`` handler. A

new ``connection_flipped()`` event is added to allow updating custom state in

script-land.

- Loading ``policy/frameworks/notice/community-id.zeek`` now also automatically

community ID logging. In the past, loading the script had no effect unless

``policy/protocols/conn/community-id-logging.zeek`` was loaded before. This

was fairly unusual and hard to debug behavior.

- Connections to broadcast addresses are not flipped based on

``likely_server_ports`` anymore. Previously, broadcast packets originating

from a likely server port resulted in 255.255.255.255 being the originator in

``conn.log``.

- When too many HTTP requests are pending, Zeek will now log them at once and

reset request/response correlation instead of running into unbounded state

growth. This behavior is configurable via a new option

``HTTP::max_pending_requests``. The default is ``100``.

- Fix deferred initialization of nested records containing non-const &default

attributes.

- Parameter lists for functions, events and hooks now use commas instead of

semicolons in error messages or when printing such functions.

- The IO buffer size used for PCAP file reading is now always 128kb. This new

default can be changed via ``Pcap::bufsize_offline_bytes``.

- The input framework now provides better information in error messages when

encountering missing non-optional field while loading data.

- The SSL analyzer will now parse a configurable maximum of 10 SSL Alerts per

SSL message. For TLS 1.3, the maximum is implicitly 1 as defined by RFC 8446.

If there are more alerts, a new weird "SSL_excessive_alerts_in_record" is raised.

For non-TLS 1.3, the maximum can be redefined via ``SSL::max_alerts_per_record``.

- The ``ssl_history`` field in the ssl.log is now capped at a configurable

limit of 100 characters prevent unbounded growth. The limit can be changed

via the option ``SSL::max_ssl_history_length``. When reached, a new weird

named "SSL_max_ssl_history_length_reached" is raised.

Deprecated Functionality

------------------------

- Accessing globals with ``GLOBAL::name`` has been deprecated and will be

removed with Zeek 7.1. Use ``::name`` instead.

- The original ``trigger::Trigger`` constructor has been deprecated and will be

removed with Zeek 7.1. Use the new alternative constructor (per

``src/Trigger.h``) instead, including replacing any use of ``new ...`` with

``make_intrusive<...>``. The new constructor differs only in the placement of

the ``timeout`` parameter, and in that - unlike the original - it always

returns a valid pointer, which must be Unref()'d after construction, either

explicitly (if using ``new``) or implicitly (if using

``make_intrusive<...>``).

Zeek 6.0.0

==========

Breaking Changes

----------------

- Zeek now treats private address space (i.e., non-routable IP address ranges)

as local by default, matching the intuition of many users that e.g. a

192.168/16 IP address should show up as local in the logs. To do this, Zeek

automatically adds ``Site::private_address_space`` to ``Site::local_nets`` at

startup. Subsequent runtime updates to ``Site::private_address_space``

propagate to ``Site::local_nets``, while updates to the latter don't affect

the former.

You're free to define ``Site::local_nets`` as before and do not need to update

your configurations. If you added standard private address space to

``Site::local_nets`` in the past, you no longer need to do so. This also

applies to zeekctl's ``networks.cfg`` file.

The new global Boolean ``Site::private_address_space_is_local``, true by

default, controls the behavior. A redef to false brings back Zeek's prior

behavior of considering private address space an unrelated concept, which will

come in handy for example when working with tests that compare results against

log baselines that have not yet been updated.

- Telemetry centralization and Prometheus exposition is not enabled by default

anymore. Previously, the manager node would open port 9911/tcp by default and

import all metrics from other nodes. For large clusters, the current implementation

introduces significant processing overhead on the manager even if the Prometheus

functionality is not used. While inconvenient, disable this functionality

(assumed to be used by few as of now) by default to preserve resources.

The script to enable centralization and the Prometheus endpoint is now

located in the ``policy/`` folder. Re-enable the old functionality with:

@load frameworks/telemetry/prometheus

You may experiment with increasing ``Broker::metrics_export_interval``

(default 1s) to reduce the extra overhead and communication at the expense

of stale metrics.

- Custom source tarballs require a ``repo-info.json`` file.

Note, should you be using official Zeek release tarballs only, or build

Zeek solely from git checkouts, this does not affect you.

However, if you're building your own Zeek source tarballs, it is now required

that a ``repo-info.json`` file exists at the top-level. The ``dist`` target was

extended to add this file and official Zeek release source tarballs will

contain it going forward.

The following command can be used to produce ``repo-info.json``:

python3 ./ci/collect-repo-info.py --only-git > ../path/to/tarballdir/repo-info.json

This is required to support the new ``-V`` / ``--build-info`` option that

provides information about git submodules and included plugins used during

the build. The ``ci/collect-repo-info.py`` tool runs at ``./configure`` time

and either collects the required information from a git clone (when git is

installed), or otherwise uses the content of a file named ``repo-info.json``.

If you see opportunities to extend ``repo-info.json`` with further information,

please get in touch.

- Plugin authors should raise the minimum required CMake version to 3.15 to

ensure compatibility with new CMake scaffolding included in this

release. Older versions will trigger a warning at configuration time and,

depending on the functionality included in the plugin, may trigger subsequent

errors during configuration or build.

- Zeek container images are not pushed to the zeekurity organization anymore.

Please switch to using the ``zeek/zeek`` image on DockerHub, or the images

published to ``public.ecr.aws/zeek/zeek``.

- The IRC_Data analyzer declaration has been moved to protocols/irc/IRC.h.

- The error message returned when using ``bro_init``, ``bro_done``, and

``bro_script_loaded`` events is now removed. removed. Usage of these events

has returned that error during script parsing for a few years, and time has

come to finally remove it.

New Functionality

-----------------

- Zeek now features experimental JavaScript support:

/* hello.js */

zeek.on('zeek_init', () => {

console.log('Hello, Zeek!');

});

$ zeek ./hello.js

Hello, Zeek!

When building Zeek on a system that features a recent (16.13+) version of the

libnode package with development headers, Zeek automatically includes the

externally-maintained ZeekJS plugin (https://github.com/corelight/zeekjs) as a

builtin plugin. This allows Zeek to load and execute JavaScript code located

in ``.js`` or ``.cjs`` files. When no such files are passed to Zeek, the

JavaScript engine and Node.js environment aren't initialized and there is no

runtime impact.

The Linux distributions Fedora 37 & 38, Ubuntu 22.10, and the upcoming Debian

12 release provide suitable packages. On other platforms, Node.js can be built

from source with the ``--shared`` option.

To disable this functionality, pass ``--disable-javascript`` to configure.

- Zeek now comes with Spicy support built in, meaning it can now

leverage any analyzers written in Spicy out of the box. While the

interface layer connecting Zeek and Spicy used to be implemented

through an external Zeek plugin, that code has now moved into the

Zeek code base itself. We also added infrastructure to Zeek that

enables its built-in standard analyzers to use Spicy instead of

Binpac. As initial (simple) examples, Zeek's Syslog and Finger

analyzers are now implemented in Spicy. While their legacy versions

remain available as fallbacks for now in case Spicy gets explicitly

disabled at build time, their use is deprecated and their code won't

be maintained any further. (Some of these Spicy updates were part of

Zeek 5.2 already, but hadn't been included in its NEWS section.)

- Zeek events now hold network timestamps. For scheduled events, the timestamp

represents the network time for which the event was scheduled for, otherwise

it is the network time at event creation. A new bif ``current_event_time()``

allows to retrieve the current event's network timestamp within the script-layer.

When Zeek sends events via Broker to other nodes in a cluster, an event's network

timestamp is attached to the Broker messages. On a receiving Zeek node executing a

handler for a remote event, ``current_event_time()`` returns the network time of

the sending node at the time the event was created.

The Broker level implementation allows to exchange arbitrary event metadata, but

Zeek's script and C++ APIs currently only expose network timestamp functionality.

- A new bif ``from_json()`` can be used to parse JSON strings into records.

type A: record { a: addr; };

local p = from_json({\"a\": \"192.168.0.1\"}", A);

if ( p$valid )

print (p$v as A)

Implicit conversion from JSON to Zeek types is implemented for bool, int, count,

real, interval (number as seconds) and time (number as unix timestamp), port

(strings in "80/tcp" notation), patterns, addr, subnet, enum, sets, vectors

and records similar to the rules of the input framework. Optional or default

record fields are allowed to be missing or null in the input.

- Zeek now provides native "Community ID" support with a new bif called

``community_id_v1()``. Two policy scripts ``protocols/conn/community-id-logging``

and ``frameworks/notice/community-id`` extend the respective logs with a

``community_id`` field the same way as the external zeek-community-id plugin

provides. A main difference to the external ``hash_conn()`` bif is that the

``community_id_v1()`` takes a ``conn_id`` record instead of a ``connection``.

Loading the new policy scripts and using the external zeek-community-id

plugin at the same time is unsupported.

- ZeekControl is now multi-logger aware. When multiple logger nodes are configured

in ZeekControl's node.cfg, by default the log archival logic adds a logger's name

as suffix to the rotated file name:

stats.11:18:57-11:19:00-logger-1.log.gz

stats.11:18:57-11:19:00-logger-2.log.gz

Previously, in a multi-logger setup, individual logger processes would overwrite

each other's log files during rotation, causing data loss.

For setups with a single logger, there's no change in behavior. The naming

of the final logs can be customized by providing an alternative

``make-archive-name`` script and using the new ``ZEEK_ARG_LOG_SUFFIX``

environment variable.

- A supervisor controlled Zeek cluster is now multi-logger aware. This avoids

loggers overwriting each other's log files within a single log-queue directory.

By default, a logger's name is appended to the rotated logs by zeek-archiver.

- Introduce a new command-line option ``-V`` / ``--build-info``. It produces

verbose output in JSON format about the repository state and any included

plugins.

- The X.509 certificate parser now exposes the signature type that is given inside

the signed portion of the certificate.

- The SSL parser now parses the CertificateRequest handshake message. There is a new

``ssl_certificate_request`` event and a new ``parse_distinguished_name`` function.

We also added the ``protocols/ssl/certificate-request-info`` policy script, that

adds some additional information to ``ssl.log``.

- Add logging metrics for streams (``zeek-log-stream-writes``) and writers

(``zeek-log-writer-writes-total``).

- Add networking metrics via the telemetry framework. These are enabled

when the ``misc/stats`` script is loaded.

zeek-net-dropped-packets

zeek-net-link-packets

zeek-net-received-bytes

zeek-net-packet-lag-seconds

zeek-net-received-packets-total

Except for lag, metrics originate from the ``get_net_stats()`` bif and are

updated through the ``Telemetry::sync()`` hook every 15 seconds by default.

- The DNS analyzer now parses RFC 2535's AD ("authentic data") and CD ("checking

disabled") flags from DNS requests and responses, making them available in

the ``dns_msg`` record provided by many of the ``dns_*`` events. The existing

``Z`` field remains unchanged and continues to subsume the two flags, for

backward compatibility.

- The supervisor framework can now start worker nodes that read from a trace file.

- Zeek can be prevented from updating ``network_time()`` to the current time

by setting ``allow_network_time_forward=F``. Together with ``set_network_time()``

or a custom plugin, this allows control of ``network_time()`` without Zeek

interfering.

- The setting ``Pcap::non_fd_timeout`` can be used to configure the timeout

used by non-selectable packet sources in the idle case (default 20usec).

This value has previously been hard-coded, but increasing it can significantly

reduce idle CPU usage in low packet rate deployments.

- Zeek now supports a new ``@pragma`` directive. It currently allows suppressing

deprecation warnings in Zeek scripts by opening with

``@pragma push ignore-deprecations`` and closing with ``@pragma pop``.

This particularly helps in situations where use of the Zeek base scripts, for

example to populate a deprecated field for API compatibility, would otherwise