devel

-----

* Updated OpenSSL to 1.1.1j and OpenLDAP to 2.4.57.

* added logging of elapsed time of ArangoSearch commit/consolidation/cleanup jobs

* Prevent arangosh from trying to connect after every executed command.

This fixes the case when arangosh is started with default options, but no

server is running on localhost:8529. In this particular case, arangosh will

try to connect on startup and after every executed shell command. The

connect attempts all fail and time out after 300ms.

In this case we now don't try to reconnect after every command.

* Added 'custom-query' testcase to arangobench to allow execution of custom

queries.

This also adds the options `--custom-query` and `--custom-query-file` for

arangobench.

* On Windows create a minidump in case of an unhandled SEH exception for

post-mortem debugging.

* Add JWT secret support for arangodump and arangorestore, i.e. they now also

provide the command-line options `--server.ask-jwt-secret` and

`--server.jwt-secret-keyfile` with the same meanings as in arangosh.

* Add optional hyperlink to program option sections for information purposes,

and add optional sub-headlines to program options for better grouping.

These changes will be visible only when using `--help`.

* For Windows builds, remove the defines `_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS`

and `_ENABLE_ATOMIC_ALIGNMENT_FIX` that were needed to build Boost components

with MSVC in older versions of Boost and MSVC.

Both of these defines are obsolete nowadays.

* Database initial sync considers document count on leader for

estimating timeouts when over 1 million docs on leader.

* EE only bugfix: On DisjointSmartGraphs that are used in anonymous way,

there was a chance that the query could fail, if non-disjoint collections

were part of the query. Named DisjointSmartGraphs have been save to this bug.

Example:

DisjointSmartGraph (graph) on vertices -edges-> vertices

Query:

WITH vertices, unrelated

FOR out IN 1 OUTBOUND "v/1:1" edges

FOR u IN unrelated

RETURN [out, u]

The "unrelated" collection was pulled into the DisjointSmartGraph, causing

the AQL setup to create erroneous state.

This is now fixed and the above query works.

This query:

WITH vertices, unrelated

FOR out IN 1 OUTBOUND "v/1:1" GRAPH "graph"

FOR u IN unrelated

RETURN [out, u]

was not affected by this bug.

* Fixed issue BTS-268: fix a flaky Foxx self-heal procedure.

* Fixed issue DEVSUP-720: Within an AQL query, the "COLLECT WITH COUNT INTO"

statement could lead to a wrong count output when used in combination with

an index which has been created with an array index attribute.

* Fixed ES-784 regression related to encryption cipher propagation to

ArangoSearch data.

* Improved the wording for sharding options displayed in the web interface.

Instead of offering `flexible` and `single`, now use the more intuitive

`Sharded` and `OneShard` options, and update the help text for them.

* Make all AQL cursors return compact result arrays.

* Fix profiling of AQL queries with the `silent` and `stream` options sets in

combination. Using the `silent` option makes a query execute, but discard all

its results instantly. This led to some confusion in streaming queries, which

can return the first query results once they are available, but don't

necessarily execute the full query.

Now, `silent` correctly discards all results even in streaming queries, but

this has the effect that a streaming query will likely be executed completely

when the `silent` option is set. This is not the default however, and the

`silent` option is normally not set. There is no change for streaming queries

if the `silent` option is not set.

As a side-effect of this change, this makes profiling (i.e. using

`db._profileQuery(...)` work for streaming queries as well. Previously,

profiling a streaming query could have led to some internal errors, and even

query results being returned, even though profiling a query should not return

any query results.

* Make dropping of indexes in cluster retry in case of precondition failed.

When dropping an indexes of a collection in the cluster, the operation could

fail with a "precondition failed" error in case there were simultaneous

index creation or drop actions running for the same collection. The error

was returned properly internally, but got lost at the point when

`<collection>.dropIndex()` simply converted any error to just `false`.

We can't make `dropIndex()` throw an exception for any error, because that

would affect downwards-compatibility. But in case there is a simultaneous

change to the collection indexes, we can just retry our own operation and

check if it succeeds then. This is what `dropIndex()` will do now.

* Try to raise file descriptors limit in local start scripts (in `scripts/`

directory - used for development only).

* Fix error reporting in the reloadTLS route.

* Fix potential undefined behavior when iterating over connected nodes in an

execution plan and calling callbacks for each of the nodes: if the callbacks

modified the list of connected nodes of the current that they were called

from, this could lead to potentially undefined behavior due to iterator

invalidation. The issue occurred when using a debug STL via `_GLIBCXX_DEBUG`.

* Fixed replication bug in MerkleTree sync protocol, which could lead to

data corruption. The visible effect was that shards could no longer get

in sync since the counts would not match after sync, even after a recount.

This corruption only happened if there were large amounts of differences

(at least 65537) and the destination side had newer revisions for some

keys than the source side.

* Fixed a RocksDB bug which could lead to an assertion failure when compiling

with STL debug mode -D_GLIBCXX_DEBUG.

* Fixed a rare internal buffer overflow around ridBuffers.

* Issue #13141: The `move-filters-into-enumerate` optimization, when applied to

an EnumerateCollectionNode (i.e. full collection scan), did not do regular

checks for the query being killed during the filtering of documents, resulting

in the maxRuntime option and manual kill of a query not working timely.

* Simplify the DistributeExecutor and avoid implicit modification of its input

variable. Previously the DistributeExecutor could update the input variable

in-place, leading to unexpected results (see #13509).

The modification logic has now been moved into three new _internal_ AQL

functions (MAKE_DISTRIBUTE_INPUT, MAKE_DISTRIBUTE_INPUT_WITH_KEY_CREATION,

and MAKE_DISTRIBUTE_GRAPH_INPUT) and an additional calculation node with an

according function call will be introduced if we need to prepare the input

data for the distribute node.

* Added new REST APIs for retrieving the sharding distribution:

- GET `/_api/database/shardDistribution` will return the number of

collections, shards, leaders and followers for the database it is run

inside. The request can optionally be restricted to include data from

only a single DB server, by passing the `DBserver` URL parameter.

This API can only be used on coordinators.

- GET `/_admin/cluster/shardDistribution` will return global statistics

on the current shard distribution, showing the total number of databases,

collections, shards, leaders and followers for the entire cluster.

The results can optionally be restricted to include data from only a

single DB server, by passing the `DBserver` URL parameter.

By setting the `details` URL parameter, the response will not contain

aggregates, but instead one entry per available database will be returned.

This API can only be used in the `_system` database of coordinators, and

requires admin user privileges.

* Decrease the size of serialized index estimates, by introducing a

compressed serialization format. The compressed format uses the previous

uncompressed format internally, compresses it, and stores the compressed

data instead. This makes serialized index estimates a lot smaller, which

in turn decreases the size of I/O operations for index maintenance.

* Do not create index estimator objects for proxy collection objects on

coordinators and DB servers. Proxy objects are created on coordinators and

DB servers for all shards, and they also make index objects available. In

order to reduce the memory usage by these objects, we don't create any

index estimator objects for indexes in those proxy objects. Index estimators

usually take several KB of memory each, so not creating them will pay out

for higher numbers of collections/shards.

* More improvements for logging:

* Added new REST API endpoint GET `/_admin/log/entries` to return log entries

in a more intuitve format, putting each log entry with all its properties

into an object. The API response is an array with all log message objects

that match the search criteria.

This is an extension to the already existing API endpoint GET `/_admin/log`,

which returned log messages fragmented into 5 separate arrays.

The already existing API endpoint GET `/_admin/log` for retrieving log

messages is now deprecated, altough it will stay available for some time.

* Truncation of log messages now takes JSON format into account, so that

the truncation of oversized JSON log messages still keeps a valid JSON

structure even after the truncation.

* The maximum size of in-memory log messages was doubled from 256 to 512

chars, so that longer parts of each log message can be preserved now.

* Improvements for logging. This adds the following startup options to arangod:

- `--log.max-entry-length`: controls the maximum line length for individual

log messages that are written into normal logfiles by arangod (note: this

does not include audit log messages).

Any log messages longer than the specified value will be truncated and the

suffix '...' will be added to them. The purpose of this parameter is to

shorten long log messages in case there is not a lot of space for logfiles,

and to keep rogue log messages from overusing resources.

The default value is 128 MB, which is very high and should effectively

mean downwards-compatiblity with previous arangod versions, which did not

restrict the maximum size of log messages.

- `--audit.max-entry-length`: controls the maximum line length for individual

audit log messages that are written into audit logs by arangod. Any audit

log messages longer than the specified value will be truncated and the

suffix '...' will be added to them.

The default value is 128 MB, which is very high and should effectively

mean downwards-compatiblity with previous arangod versions, which did not

restrict the maximum size of log messages.

- `--log.in-memory-level`: controls which log messages are preserved in

memory (in case `--log.in-memory` is set to `true`). The default value is

`info`, meaning all log messages of types `info`, `warning`, `error` and

`fatal` will be stored by an instance in memory (this was also the behavior

in previous versions of ArangoDB). By setting this option to `warning`,

only `warning`, `error` and `fatal` log messages will be preserved in memory,

and by setting the option to `error` only error and fatal messages will be kept.

This option is useful because the number of in-memory log messages is

limited to the latest 2048 messages, and these slots are by default shared

between informational, warning and error messages.

* Honor the value of startup option `--log.api-enabled` when set to `false`.

The desired behavior in this case is to turn off the REST API for logging,

but was not implemented. The default value for the option is `true`, so the

REST API is enabled. This behavior did not change, and neither did the

behavior when setting the option to a value of `jwt` (meaning the REST API

for logging is only available for superusers with a valid JWT token).

* Split the update operations for the _fishbowl system collection with Foxx

apps into separate insert/replace and remove operations. This makes the

overall update not atomic, but as removes are unlikely here, we can now get

away with a simple multi-document insert-replace operation instead of a

truncate and an exclusive transaction, which was used before.

* Fix `/_admin/cluster/removeServer` API.

This often returned HTTP 500 with an error message "Need open Array" due to

an internal error when setting up agency preconditions.

* Remove logging startup options `--log.api-enabled` and `--log.keep-logrotate`

for all client tools (arangosh, arangodump, arangorestore etc.), as these

options are only meaningful for arangod.

* Fixed BTS-284: upgrading from 3.6 to 3.7 in cluster enviroment.

Moved upgrade ArangoSearch links task to later step as it needs cluster

connection. Removed misleading error log records for failed ArangoSearch index

creation during upgrade phase.

* Extend the "move-calculations-up" optimizer rule so that it can move

calculations out of subqueries into the outer query.

* Don't allocate ahead-of-time memory for striped PRNG array in arangod,

but instead use thread-local PRNG instances. Not only does this save a

few megabytes or memory, but it also avoids potential (but unlikely)

sharing of the same PRNG instance by multiple threads.

* Remove undocumented CMake variable `USE_BACKTRACE`, and remove define

`ARANGODB_ENABLE_BACKTRACE`. Both were turned off by default before, and

when turned on allow to produce backtraces from within the executable in

case debug symbols were available, working and the build was also compiled

with `USE_MAINTAINER_MODE=On`. Some code in this context was obviously

unreachable, so now it has all been removed.

To log a backtrace from within arangod, it is now possible to call

`CrashHandler::logBacktrace()`, which will log a backtrace of the calling

thread to the arangod log. This is restricted to Linux builds only.

* Fix warnings about suggest-override which can break builds when warnings

are treated as errors.

* Turn off option `--server.export-read-write-metrics` for now, until there

is certainty about the runtime overhead it introduces.

* Fixed issue #12543: Unused Foxx service config can not be discarded.

* Fixed issue #12363: Foxx HTTP API upgrade/replace always enables

development mode.

* Remove unsafe query option `inspectSimplePlans`. This option previously

defaulted to `true`, and turning it off could make particular queries fail.

The option was ignored in the cluster previously, and turning it off only

had an effect in single server, there making very simple queries (queries

not containing any FOR loops) not going through the optimizer's complete

pipeline as a performance optimization. However, the optimization was only

possible for a very small number of queries and even had adverse effects,

so it is now removed entirely.

* On Linux and MacOS, require at least 65535 usable file descriptors at startup.

If less file descriptors are available to the arangod process, then the

startup is automatically aborted.

Even the chosen minimum value of 65535 will often not be high enough to

store considerable amounts of data. However, no higher value was chosen

in order to not make too many existing small installations fail at startup

after upgrading.

The required number of file descriptors can be configured using the startup

option `--server.descriptors-minimum`. It defaults to 65535, but it can be

increased to ensure that arangod can make use of a sufficiently high number

of files. Setting `--server.descriptors-minimum` to a value of `0` will

make the startup require only an absolute minimum limit of 1024 file

descriptors, effectively disabling the change.

Such low values should only be used to bypass the file descriptors check

in case of an emergency, but this is not recommended for production.

* Added metric `arangodb_transactions_expired` to track the total number

of expired and then garbage-collected transactions.

* Allow toggling the document read/write counters and histograms via the

new startup option `--server.export-read-write-metrics false`. This

option defaults to `true`, so these metrics will be exposed by default.

* Upgraded bundled version of libunwind to v1.5.

* Added startup option `--javascript.tasks` to allow turning off JavaScript

tasks if not needed. The default value for this option is `true`, meaning

JavaScript tasks are available as before.

However, with this option they can be turned off by admins to limit the

amount of JavaScript user code that is executed.

* Only instantiate a striped PRNG instance for the arangod server, but not

for any of the client tools (e.g. arangosh, arangodump, arangorestore).

The client tools do not use the striped PRNG, so we can save a few MBs of

memory for allocating the striped PRNG instance there, plus some CPU time

for initializing it.

* Improve shard synchronization protocol by only transferring the required

parts of the inventory from leader to follower. Previously, for each shard

the entire inventory was exchanged, which included all shards of the

respective database with all their details.

In addition, save 3 cluster-internal requests per shard in the initial shard

synchronization protocol by reusing already existing information in the

different steps of the replication process.

* Added metric `arangodb_scheduler_low_prio_queue_last_dequeue_time` that

provides the time (in millseconds) it took for the most recent low priority

scheduler queue item to bubble up to the queue's head. This metric can be

used to estimate the queuing time for incoming requests.

The metric will be updated probabilistically when a request is pulled from

the scheduler queue, and may remain at its previous value for a while if

only few requests are coming in or remain permanently at its previous value

if no further requests are incoming at all.

* Allow {USER} placeholder string also in `--ldap.search-filter`.

* Fix agency restart with mismatching compation and log indexes.

* Added metrics for document read and write operations:

- `arangodb_document_writes: Total number of document write operations

(successful and failed) not performed by synchronous replication.

- `arangodb_document_writes_replication`: Total number of document write

operations (successful and failed) by cluster synchronous replication.

- `arangodb_collection_truncates`: Total number of collection truncate

operations (successful and failed) not performed by cluster synchronous

replication.

- `arangodb_collection_truncates_replication`: Total number of collection

truncate operations (successful and failed) by synchronous replication.

- `arangodb_document_read_time`: Execution time histogram of all document

primary key read operations (successful and failed) [s]. Note: this

does not include secondary index lookups, range scans and full collection

scans.

- `arangodb_document_insert_time`: Execution time histogram of all

document insert operations (successful and failed) [s].

- `arangodb_document_replace_time`: Execution time histogram of all

document replace operations (successful and failed) [s].

- `arangodb_document_remove_time`: Execution time histogram of all

document remove operations (successful and failed) [s].

- `arangodb_document_update_time`: Execution time histogram of all

document update operations (successful and failed) [s].

- `arangodb_collection_truncate_time`: Execution time histogram of all

collection truncate operations (successful and failed) [s].

The timer metrics are turned off by default, and can be enabled by setting

the startup option `--server.export-read-write-metrics true`.

* Fixed some wrong behavior in single document updates. If the option

ignoreRevs=false was given and the precondition _rev was given in the body

but the _key was given in the URL path, then the rev was wrongly taken

as 0, rather than using the one from the document body.

* Improved logging for error 1489 ("a shard leader refuses to perform a

replication operation"). The log message will now provide the database and

shard name plus the differing information about the shard leader.

* Make `padded` and `autoincrement` key generators export their `lastValue`

values, so that they are available in dumps and can be restored elsewhere

from a dump.

* Fix decoding of values in `padded` key generator when restoring from a dump.

* Fixed error reporting for hotbackup restore from dbservers back to

coordinators. This could for example swallow out of disk errors during

hotbackup restore.

* Fixed rare objectId conflict for indexes.

* Fix for OASIS-409. Fixed indexing _id attribute at recovery.

* Add shard-parallelism to arangodump when dumping collections with multiple

shards.

Previously, arangodump could execute a dump concurrently on different

collections, but it did not parallelize the dump for multiple shards of the

same collection.

This change should speed up dumping of collections with multiple shards.

When dumping multiple shards of the same collection concurrently, parallelism

is still limited by all these threads needing to serialize their chunks into

the same (shared) output file.

* Add option `--envelope` for arangodump, to control if each dumped document

should be wrapped into a small JSON envelope (e.g.

`{"type":2300,"data":{...}}`). This JSON envelope is not necessary anymore

since ArangoDB 3.8, so omitting it can produce smaller (and slightly faster)

dumps.

Restoring a dump without these JSON envelopers is handled automatically by

ArangoDB 3.8 and higher. Restoring a dump without these JSON envelopes into

previous versions (pre 3.8) however is not supported. Thus the option should

only be used if the client tools (arangodump, arangorestore) and the arangod

server are all using v3.8 or higher, and the dumps will never be stored into

earlier versions.

The default value for this option is `true`, meaning the JSON wrappers will

be stored as part of the dump. This is compatible with all previous verions.

* Fix some issues with key generators not properly taking into account the

`allowUserKeys` attribute when in a cluster.

* Make AQL optimizer rule "splice-subqueries" mandatory, in the sense that it

cannot be disabled anymore. As a side effect of this change, there will no

query execution plans created by 3.8 that contain execution nodes of type

`SubqueryNode`. `SubqueryNode`s will only be used during query planning and

optimization, but at the end of the query optimization phase will all have

been replaced with nodes of types `SubqueryStartNode` and `SubqueryEndNode`.

The code to execute non-spliced subqueries remains in place so that 3.8 can

still execute queries planned on a 3.7 instance with the "splice-subqueries"

optimizer rule intentionally turned off. The code for executing non-spliced

subqueries can be removed in 3.9.

* Normalize user-provided input/output directory names in arangoimport,

arangoexport, arangodump and arangorestore before splitting them into path

components, in the sense that now both forward and backward slashes can be

used on Windows, even interchangingly.

* Added the following bit handling functions to AQL:

- BIT_AND(array): and-combined result

- BIT_OR(array): or-combined result

- BIT_XOR(array): xor-combined result

- BIT_NEGATE(value, bits): bitwise negation of `value`, with a mask of

`bits` length

- BIT_TEST(value, index): test if bit at position `index` is set in `value`

(indexes are 0-based)

- BIT_POPCOUNT(value): return number of bits set in `value`

- BIT_SHIFT_LEFT(value, shift, bits): bitwise shift-left of `value` by

`shift` bits, with a mask of `bits` length

- BIT_SHIFT_RIGHT(value, shift, bits): bitwise shift-right of `value` by

`shift` bits, with a mask of `bits` length

- BIT_CONSTRUCT(array): construct a number with bits set at the positions

given in the array

- BIT_DECONSTRUCT(value): deconstruct a number into an array of its individual

set bits

- BIT_TO_STRING(value): create a bitstring representation from numeric `value`

- BIT_FROM_STRING(value): parse a bitstring representation into a number

`BIT_AND`, `BIT_OR` and `BIT_XOR` are also available as aggregate functions

for usage inside COLLECT AGGREGATE.

All above bit operations support unsigned integer values with up to 32 bits.

Using values outside the supported range will make any of these bit functions

return `null` and register a warning.

* Add binary (base 2) and hexadecimal (base 16) integer literals to AQL.

These literals can be used where regular (base 10) integer literal can used.

The prefix for binary integer literals is `0b`, e.g. `0b10101110`.

The prefix for hexadecimal integer literals i `0x`, e.g. `0xabcdef02`.

Binary and hexadecimal integer literals can only be used for unsigned integers.

The maximum supported value is `(2 ^ 32) - 1`, i.e. `0xffffffff` (hexadecimal)

or `0b11111111111111111111111111111111` (binary).

* AQL query execution plan register usage optimization.

This is a performance optimization that may positively affect some AQL

queries that use a lot of variables that are only needed in certain

parts of the query.

The positive effect will come from saving registers, which directly

translates to saving columns in AqlItemBlocks.

Previously, the number of registers that were planned for each depth

level of the query never decreased when going from one level to the

next. Even though unused registers were recycled since 3.7, this did

not lead to unused registers being completely dismantled.

Now there is an extra step at the end of the register planning that

keeps track of the actually used registers on each depth, and that

will shrink the number of registers for the depth to the id of the

maximum register. This is done for each depth separately.

Unneeded registers on the right hand side of the maximum used register

are now discarded. Unused registers on the left hand side of the maximum

used register id are not discarded, because we still need to guarantee

that registers from depths above stay in the same slot when starting

a new depth.

* Added metric `arangodb_aql_current_query` to track the number of currently

executing AQL queries.

* Updated arangosync to 1.2.2.

* Fix a bug in the agency Supervision which could lead to removeFollower

jobs constantly being created and immediately stopped again.

* Limit additional replicas in failover cases to +2.

* Print a version mismatch (major/minor version difference) between the

arangosh version and the remote arangod version at arangosh startup.

* Internal refactoring of K_PATH feature, with the goal to have all graph

algorithms on the same framework. This change should not have any visible

effect on users.

* Fixed an endless busy loop which could happen if a coordinator tries to

roll back a database creation, but the database has already been dropped

by other means.

* Removed server-side JavaScript object `ArangoClusterComm`, so it cannot be

used from inside JavaScript operations or Foxx.

The `ArangoClusterComm` object was previously used inside a few internal

JavaScript operations, but was not part of the public APIs.

* Restrict access to functions inside JavaScript objects `ArangoAgency` and

`ArangoAgent` to JavaScript code that is running in privileged mode, i.e.

via the server's emergency console, the `/_admin/execute` API (if turned on)

or internal bootstrap scripts.

* Added startup option `--javascript.transactions` to allow turning off JavaScript

transactions if not needed. The default value for this option is `true`, meaning

JavaScript transactions are available as before.

However, with this option they can be turned off by admins to limit the amount

of JavaScript user code that is executed.

* Introduce a default memory limit for AQL queries, to prevent rogue queries from

consuming the entire memory available to an arangod instance.

The limit is introduced via changing the default value of the option `--query.memory-limit`

from previously `0` (meaning: no limit) to a dynamically calculated value.

The per-query memory limits defaults are now:

Available memory: 0 (0MiB) Limit: 0 unlimited, %mem: n/a

Available memory: 134217728 (128MiB) Limit: 33554432 (32MiB), %mem: 25.0

Available memory: 268435456 (256MiB) Limit: 67108864 (64MiB), %mem: 25.0

Available memory: 536870912 (512MiB) Limit: 201326592 (192MiB), %mem: 37.5

Available memory: 805306368 (768MiB) Limit: 402653184 (384MiB), %mem: 50.0

Available memory: 1073741824 (1024MiB) Limit: 603979776 (576MiB), %mem: 56.2

Available memory: 2147483648 (2048MiB) Limit: 1288490189 (1228MiB), %mem: 60.0

Available memory: 4294967296 (4096MiB) Limit: 2576980377 (2457MiB), %mem: 60.0

Available memory: 8589934592 (8192MiB) Limit: 5153960755 (4915MiB), %mem: 60.0

Available memory: 17179869184 (16384MiB) Limit: 10307921511 (9830MiB), %mem: 60.0

Available memory: 25769803776 (24576MiB) Limit: 15461882265 (14745MiB), %mem: 60.0

Available memory: 34359738368 (32768MiB) Limit: 20615843021 (19660MiB), %mem: 60.0

Available memory: 42949672960 (40960MiB) Limit: 25769803776 (24576MiB), %mem: 60.0

Available memory: 68719476736 (65536MiB) Limit: 41231686041 (39321MiB), %mem: 60.0

Available memory: 103079215104 (98304MiB) Limit: 61847529063 (58982MiB), %mem: 60.0

Available memory: 137438953472 (131072MiB) Limit: 82463372083 (78643MiB), %mem: 60.0

Available memory: 274877906944 (262144MiB) Limit: 164926744167 (157286MiB), %mem: 60.0

Available memory: 549755813888 (524288MiB) Limit: 329853488333 (314572MiB), %mem: 60.0

As previously, a memory limit value of `0` means no limitation.

The limit values are per AQL query, so they may still be too high in case queries

run in parallel. The defaults are intentionally high in order to not stop any valid,

previously working queries from succedding.

* Added startup option `--audit.queue` to control audit logging queuing

behavior (Enterprise Edition only):

The option controls whether audit log messages are submitted to a queue

and written to disk in batches or if they should be written to disk directly

without being queued.

Queueing audit log entries may be beneficial for latency, but can lead to

unqueued messages being lost in case of a power loss or crash. Setting

this option to `false` mimics the behavior from 3.7 and before, where

audit log messages were not queued but written in a blocking fashion.

* Fixed some situations of

[...]

SUBQUERY

FILTER

LIMIT

[...]

in AQL queries, yielding incorrect responses. A distributed

state within the subquery was not reset correctly.

This could also lead into "shrink" errors of AQL item blocks,

or much higher query runtimes.

Fixes:

- BTS-252

- ES-687

- github issue: #13099

- github issue: #13124

- github issue: #13147

- github issue: #13305

- DEVSUP-665

* Added metric `arangodb_server_statistics_cpu_cores` to provide the number of

CPU cores visibile to the arangod process. This is the number of CPU cores

reported by the operating system to the process.

If the environment variable `ARANGODB_OVERRIDE_DETECTED_NUMBER_OF_CORES` is

set to a positive value at instance startup, this value will be returned

instead.

* `COLLECT WITH COUNT INTO x` and `COLLECT var = expr WITH COUNT INTO x` are now

internally transformed into `COLLECT AGGREGATE x = LENGTH()` and

`COLLECT var = expr AGGREGATE x = LENGTH()` respectively. In addition, any

argument passed to the `COUNT`/`LENGTH` aggregator functions are now optimized

away. This not only simplified the code, but also allows more query optimizations:

- If the variable in `COLLECT WITH COUNT INTO var` is not used, the implicit

aggregator is now removed.

- All queries of the form `COLLECT AGGREGATE x = LENGTH()` are now executed

using the count executor, which can result in significantly improved

performance.

* Minor and rare AQL performance improvement, in nested subqueries:

LET sq1 ([..] FILTER false == true LET sq2 = (<X>) [..])

where sq1 produces no data (e.g. by the above filter) for sq2,

the part <X> have been asked two times (second returns empty result),

instead of one, if and only if the main query executes sq1 exactly one time.

Now we get away with one call only.

In the case sq1 has data, or sq1 is executed more often, only one call was needed

(assuming the data fits in one batch).

* Updated OpenSSL to 1.1.1i and OpenLDAP to 2.4.56.

* Make internal ClusterInfo::getPlan() wait for initial plan load from agency.

* Added AQL timezone functions `DATE_TIMEZONE` and `DATE_TIMEZONES`.

* Make DB servers report storage engine health to the agency, via a new "health"

attribute in requests sent to Sync/ServerStates/<id>.

The supervision can in the future check this attribute if it is posted,

and mark servers as BAD or FAILED in case an unhealthy status is reported.

DB server health is currently determined by whether or not the storage engine

(RocksDB) has reported a background error, and by whether or not the free disk

space has reached a critical low amount. The current threshold for free disk

space is set at 1% of the disk capacity (only the disk is considered that

contains the RocksDB database directory).

The minimum required free disk space percentage can be configured using the new

startup option `--rocksdb.minimum-disk-free-percent`, which needs to be between

0 and 1 (including). A value of 0 disables the check.

The minimum required free disk space can also be configured in bytes using the

new startup option `--rocksdb.minimum-disk-free-bytes`. A value of 0 disables

this check, too.

* Failed servers are now reported consistently in the web interface, at

approximately the same time in the navigation bar and in the nodes view.

Previously these two places had their own, independent poll mechanism for the

nodes' health, and they were updated independently, which could cause an

inconsistent view of the nodes' availability.

Using only one poll mechanism instead also saves some period background requests

for the second availability check.

* Updated arangosync to 1.2.1.

* Clean up callback bin and empty promises in single-host-agency.

* Stabilize a Foxx cleanup test.

* Drop a pair of braces {} in /_admin/metrics in case of empty labels, which

makes the API adhere better to the official Prometheus syntax.

* Add some more metrics to the ConnectionPool.

* Remove HTTP "Connection" header when forwarding requests in the cluster

from one coordinator to another, and let the internal network layer

handle closing of connections and keep-alive.

* Added new metric: "arangodb_collection_lock_sequential_mode" this will count

how many times we need to do a sequential locking of collections. If this

metric increases this indicates lock contention in transaction setup.

Most likely this is caused by exlcusive locks used on collections with

more than one shard.

* Fix for BTS-213

Changed the transaction locking mechanism in the cluster case.

For all installations that do not use "exclusive" collection locks this change

will not be noticable. In case of "exclusive" locks, and collections with more

than one shard, it is now less likely to get a LOCK_TIMEOUT (ErrorNum 18).

It is still possible to get into the LOCK_TIMEOUT case, especially if

the "exclusive" operation(s) are long-running.

* Reduce overhead of audit logging functionality if audit logging is turned

off.

* Add several more attributes to audit-logged queries, namely query execution

time and exit code (0 = no error, other values correspond to general ArangoDB

error codes).

* Added "startupTime", "computationTime" and "storageTime" to Pregel result

statistics.

* Fixed a bug in maintainer mode sorting followerinfo lists the wrong way.

* Limit value of `--rocksdb.block-cache-size` to 1 GB for agent instances to

reduce agency RAM usage, unless configured otherwise. In addition, limit the

value of `--rocksdb.total-write-buffer-size` to 512 MB on agent instances for

the same reason.

* Fixed issue #13238 Thread naming API on Windows are now used only if

available in KERNEL32.DLL

* When querying the list of currently running or slow AQL queries, ignore not-yet

created databases on other coordinators.

* Added support for fetching the list of currently running and slow AQL queries

from all databases at once, by adding an `all` parameter to the following

query APIs:

* `require("@arangodb/aql/queries").current({ all: true })`: will return the

currently running queries from all databases, not just the currently

selected database.

* HTTP GET `/_api/query/current?all=true`: same, but for the HTTP REST API.

* `require("@arangodb/aql/queries").slow({ all: true })`: will return the

slow query history from all databases, not just the currently selected

database.

* HTTP GET `/_api/query/slow?all=true`: same, but for the HTTP REST API.

* `require("@arangodb/aql/queries").clearSlow({ all: true })`: will clear

the slow query history for all databases, not just the currently selected

database.

* HTTP DELETE `/_api/query/slow?all=true`: same, but for the HTTP REST API.

Using the `all` parameter is only allowed when making the call inside the

`_system` database and with superuser privileges.

* Improve performance and memory efficiency of agency restart from persisted

database directory.

* Added the following agency-related metrics:

- `arangodb_agency_client_lookup_table_size`: current number of entries

in agency client id lookup table. This gauge is available only on

agent instances.

- `arangodb_agency_cache_callback_count`: current number of entries in

agency cache callbacks table. This gauge will be effective on coordinators

and DB servers.

- `arangodb_agency_callback_count`: current number of agency callbacks

registered. This gauge will be effective on coordinators and DB servers.

* Clean up agency change log, cluster info caches.

* Added SNI support for arangosh.

* Added new `rocksdb_write_stalls` and `rocksdb_write_stops` counter metrics,

which should be more accurate than existing metrics related to the underlying

conditions.

* Increased the default value of `--rocksdb.min-write-buffer-number-to-merge` in

some cases when we have allocated a sufficient amount of memory to the write

buffers for this to make sense. The increased value should help prevent

compaction-induced write stalls/stops, and should only be enabled when under

conditions such that it shouldn't greatly increase the chance of flush-induced

write stalls/stops.

* Changed the default values for `--rocksdb.cache-index-and-filter-blocks` and

`--rocksdb.cache-index-and-filter-blocks-with-high-priority` to true to

improve control over memory usage.

* Lowered the minimum allowed value for `--rocksdb.max-write-buffer-number` from

9 to 4 to allow more fine-grained memory usage control.

* Fix for issue #772: Optimized document counting for ArangoSearch views.

Added new ArangoSearch view option 'countApproximate' for customizing

view count strategy.

* Fix AR-113. Disallow non-values in the AQL geo-index-optimizer rule.

* Views on SmartGraph Edge collections do not contain some documents

twice.

* Fixed agency redirect in poll api.

* Fixed issue #12248: Web UI - Added missing HTML escaping in the setup script

section of a foxx app.

* The scheduler will now run a minimum of 4 threads at all times, and the

default and minimal value for `--server.maximal-threads` has been lowered from

64 to the greater of 32 and twice the number of detected cores.

* Throttle work coming from low priority queue, according to a constant

and to an estimate taking into account fanout for multi-shard operations.

* Move to 4 priority levels "low", "medium", "high" and "maintenance" in

scheduler to ensure that maintenance work and diagnostics is always

possible, even in the case of RocksDB throttles. Do not allow any

RocksDB work on "maintenance".

* Commit replications on high priority queue.

* Essentially get rid of timeout in replication to drop followers. This

is now entirely handled via reboot and failure tracking. The timeout

has now a default minimum of 15 minutes but can still be configured via

options.

* Additional metrics for all queue lengths and low prio ongoing work.

* New metric for number and total time of replication operations.

* New metrics for number of internal requests in flight, internal request

duration, and internal request timeouts

* Fix `Gauge` assignment operators.

* Fixed and extended LDAP log messages.

* Added LDAP_OFF if referrals and restart are false.

* If LDAP search fails, also retry (update to given number of retries).

* Add cluster support for collection.checksum() method to calculate CRC

checksums for collections.

* Fix cluster-internal replication of documents with special keys (percent

character, which has a special meaning when used inside URLs).

* Fix AQL cost estimate of spliced subqueries which could lead to overly large

numbers in the explain output of such queries.

* Make all Pregel HTTP and JavaScript APIs also accept stringified execution

number values, in addition to numeric ones.

This allows passing larger execution numbers as strings, so that any data

loss due to numeric data type conversion (uint32 => double) can be avoided.

The change also makes the Pregel HTTP and JavaScript APIs for starting a

run return a stringified execution number, e.g. "12345" instead of 12345.

* Updated ArangoDB Starter to 0.14.15-1.

* Fix Windows directory creation error handling.

* Add an AQL query kill check during early pruning. Fixes issue #13141.

* Remove a case in which followers were dropped unnecessarily in streaming

transactions that replicated to the same follower.

* Fixed ES-662 by introducing refactored thread pool to make more efficient

consolidation and commit routines for links of ArangoSearch views.

Added new command line options for fine-grained ArangoSearch maintenance

control:

- `--arangosearch.commit-threads` - max number of ArangoSearch commit

threads

- `--arangosearch.commit-threads-idle` - min number of ArangoSearch

commit threads

- `--arangosearch.consolidation-threads` - max number of ArangoSearch

consolidation threads

- `--arangosearch.consolidation-threads-idle` - min number of ArangoSearch

consolidation threads

Deprecated the following command line options:

- `--arangosearch.threads` - will be used in --arangosearch.commit-threads`

and `--arangosearch.consolidation-threads` as provided value divided by 2

if set to non-autodetect value > 0

- `--arangosearch.threads-limit`

* Turn off `StatisticsWorker` thread on DB servers.

This thread was previously only running queries on the local RocksDB

instance, but using the cluster-wide collection names. So effectively it

did nothing except use a bit of background CPU. In this case it is better

to turn off the background thread entirely on the DB servers.

* Avoid the usage of std::regex when constructing date/time string values

for log messages. This is a performance optimization only.

* Reimplement coordshort request handler. The new implementation only runs

two DB queries without any additional requests to other coordinators,

resulting in reduced load on the cluster. Previously this involved

requests to all coordinators, where each of them ran two DB queries.

* Fixed initial population of local AgencyCache values after a server restart.

Previously the local cache was populated from the agency using a commit index

value of 1, whereas it should have been 0 to get the full agency snapshot.

* Better tracking of memory used in AQL graph traversals, COLLECT and SORT

operations. From this version onwards, certain AQL queries can report a

higher memory usage than in previous versions of ArangoDB. This is not

because the queries use more memory than before, but because the memory

usage tracking has been improved.

A side effect of this change is that queries with a memory limit set may

now be aborted whereas in previous versions they ran through successfully

(but actually violated the limit). In this case it may be necessary to

adjust (i.e. raise) query memory limits accordingly.

* Increase background garbage-collection interval for cluster transactions

from 1 second to 2 seconds. This change should reduce the amount of

background task activity a tiny bit (though hardly measurable on an

otherwise idle server).

* Make the audit log honor the configured logging date/time output format

(i.e. `--log.time-format` option). Previously the audit logging always

created a time value in the server's local time, and logged it in

format YYYY-MM-DDTHH:MM:SS.

From 3.8 onwards, the audit logger will honor the date/time format

specified via the `--log.time-format` option, which defaults to

`utc-datestring`. The means the audit logging will by default log all

dates/times in UTC time. To restore the pre-3.8 behavior, please set

the option `--log.time-format` to `local-datestring`, which will make

the audit logger (and all other server log messages) use the server's

local time.

* Fix the activation of the agency supervision maintenance via the REST API

`/_admin/cluster/maintenance`. This API stored a boolean value instead of

an (expected) maintenance period end date/time string.

* Make the cancel operation safe for asynchronoulsly started JavaScript

transactions (via HTTP POST to `/_api/transaction` with the `x-arango-async`

header set).

* Updated OpenSSL to 1.1.1h.

* Added new scheduler metrics:

- `arangodb_scheduler_jobs_done`: Total number of scheduler queue jobs done

- `arangodb_scheduler_jobs_submitted`: Total number of jobs submitted to the

scheduler queue

- `arangodb_scheduler_jobs_dequeued`: Total number of jobs dequeued from the

scheduler queue

- `arangodb_scheduler_num_working_threads`: Number of currently working

scheduler threads

* Added new metric `arangodb_replication_cluster_inventory_requests` to track

the number of requests received for cluster inventories. The cluster

inventory API is called at the beginning of a dump process or by arangosync.

* Added startup option `--server.unavailability-queue-fill-grade`. This option

has a consequence for the `/_admin/server/availability` API only, which is

often called by load-balancers and other availability probing systems.

The `/_admin/server/availability` API will now return HTTP 200 if the fill

grade of the scheduler's queue is below the configured value, or HTTP 503 if

the fill grade is above it. This can be used to flag a server as unavailable

in case it is already highly loaded.

The default value for this option is `0.75`, i.e. 75%. This is a change

compared to previous versions of ArangoDB, where the default value was `1`.

To prevent sending more traffic to an already overloaded server, it can be

sensible to reduce the default value to even `0.5`.

This would mean that instances with a queue longer than 50% of their

maximum queue capacity would return HTTP 503 instead of HTTP 200 when their

availability API is probed.

nb: the default value for the scheduler queue length is 4096.

* Added metrics for the system CPU usage:

- `arangodb_server_statistics_user_percent`: Percentage of time that the

system CPUs have spent in user mode

- `arangodb_server_statistics_system_percent`: Percentage of time that

the system CPUs have spent in kernel mode

- `arangodb_server_statistics_idle_percent`: Percentage of time that the

system CPUs have been idle

- `arangodb_server_statistics_iowait_percent`: Percentage of time that

the system CPUs have been waiting for I/O

These metrics resemble the overall CPU usage metrics in `top`.

They are available on Linux only.

* Fix internal issue #777: Fixed memory access while substituting

stored values for ArangoSearch view optimization

* Make the number of network I/O threads properly configurable via the

startup option `--network.io-threads`. This option existed before, but its

configured value was effectively clamped to a value of `1`.

* Improvements for the Pregel distributed graph processing feature:

- during the loading/startup phase, the in-memory edge cache is now

intentionally bypassed. The reason for this is that any edges are

looked up exactly once, so caching them is not beneficial, but would

only lead to cache pollution.

- the loading/startup phase can now load multiple collections in parallel,

whereas previously it was only loading multiple shards of the same

collection in parallel. This change helps to reduce load times in case

there are many collections with few shards, and on single server.

- the loading and result storage phases code has been overhauled so that

it runs slightly faster.

- for Pregel runs that are based on named graphs (in contrast to explicit

naming of the to-be-used vertex and edge collections), only those edge

collections are considered that, according to the graph definition, can

have connections with the vertex. This change can reduce the loading

time substantially in case the graph contains many edge definitions.

- the number of executed rounds for the underlying Pregel algorithm now

does not vary for different `parallelism` values.

* Fix HTTP/1.1 status response header in fuerte responses

This change makes fuerte return the full status header, including the

numeric status code and the status string in the `http/1.1` header of

fuerte responses.

Previously, the return header lacked the numeric status code, so it

looked like