devel

-----

* Fix DEVSUP-753: now it is safe to call visit on exhausted disjunction

iterator.

* Slightly improve specific warning messages for better readability.

* Add 3 AQL functions: DECAY_GAUSS, DECAY_EXP and DECAY_LINEAR.

* Fix URL request parsing in case data is handed in in small chunks.

Previously the URL could be cut off if the chunk size was smaller than

the URL size.

* Backport bugfix from upstream rocksdb repository for calculating the

free disk space for the database directory. Before the bugfix, rocksdb

could overestimate the amount of free space when the arangod process

was run as non-privileged users.

* Add soft coordinator shutdown: This is a new option `soft=true` for the

DELETE /_admin/shutdown API. Has only meaning for coordinators, otherwise

ignored. A number of things are allowed to finish but no new things are

allowed when in soft coordinator shutdown:

- AQL cursors

- transactions

- asynchronous operations

- Pregel runs

Once all of the ongoing operations of these have finished and all requests

on the low priority queue have been executed, the coordinator shuts down

the normal way. This is supposed to make a coordinator restart less

intrusive for clients.

* Fix BTS-398: Cannot force index hint for primary index if FILTER has multiple

OR conditions that require different indexes.

* Fixed a problem with active failover, where a failover could take 5 mins

because the follower was caught in a bad state during replication. This

fixes BTS-425.

* Added check to utils/generateAllMetricsDocumentation.py to check that

the file name and the value of the name attribute are the same in the

metrics documentation snippets. Correct a few such names.

* Fix BTS-456, BTS-457: Make geo intersection between point and rectangle

symmetrical.

* Fix BTS-430: Added missing explain output about indexes for SHORTEST_PATH,

K_SHORTEST_PATHS and K_PATHS.

* Updated arangosync to 2.3.0.

* Added check for data type compatibility between members of pipeline

ArangoSearch analyzer.

* Implemented an optimization for Traversals. If you apply a POST filter on

the vertex and/or edge result this filter will now be applied during the

traversal to avoid generating the full output for AQL. This will have

positive effect if you filter on the vertex/edge but return the path,

this way the system does only need to produce a path that is allowed to

be passed through.

e.g.

FOR v,e,p IN 10 OUTBOUND @start GRAPH "myGraph"

FILTER v.isRelevant == true

RETURN p

can now be optimized, and the traversal statement will only produce

paths where the last vertex has `isRelevant == true`.

* Fix BTS-450: RandomGenerator caught assertion during a value generation within

`dump_maskings` testsuite. Ensure correct conversion between 64 and 32bit.

* Fix BTS-442: a query with fullCount on a sharded collection hangs

indefinitely when LIMIT is less than number of available documents.

* Bug-Fix (MacOs): In MacOs there is an upper bound for descriptors defined by

the system, which is independend of the settings in `ulimit -n`. If the

hard limit is set above this upper bound value ArangoDB tries to raise the

soft limit to the hard limit on boot. This will fail due to the system

limit. This could cause ArangoDB to not start, asking you to lower the

minimum of required file descriptors. The system set upper bound is now

honored and the soft limit will be set to either hard limit or system limit

whichever is lower.

* Allow an INSERT operation for a multi-shard collection to execute in

parallel on different DB servers.

* Fix BTS-409: return error 1948 when a negative edge was detected during or was

used as default weight in a SHORTEST_PATH or a K_SHOTRTEST_PAHS traversal.

* Fix BTS-446: When finding a not yet fully initialized agency, do not

immediately fatal exit. Keep trying for (very generous) 5

minutes.

* Removed unused documentation snippets (non-Rest DocuBlocks) as well as the

documentation about the long deprecated features Simple Queries and

JavaScript-based graph traversal. Also removed the descriptions of the

JS API methods `collection.range()`, `collection.closedRange()`,

`cursor.setBatchSize()` and `cursor.getBatchSize()`. All the functionality

is superseded by AQL.

* Implemented APM-86: add query option `fillBlockCache` to control population

of RocksDB block cache with data read by the query. The default value for

this per-query option is `true`, which mimics the previous behavior.

Setting the option to off allows not storing data in RocksDB's block cache

for queries that are known to read only semi-relevant or unimportant data.

* Improve Merkle tree memory usage and allow left-growth of trees, too. This

can help with insertions of arbitrarily old data.

* Added metric `arangodb_sync_rebuilds_total` to track the full rebuild of a

shard follower after too many subsequent shard synchronization failures. This

metric should always have a value of 0. Everything else indicates a serious

problem.

* Fixed BTS-422: SingleRemoteModification in AQL behaves different.

This disables the optimizer rule `optimize-cluster-single-document-operations`

for array inputs, e.g.

INSERT [...] INTO collection

REMOVE [...] IN collection

For the cases, the optimization is not pulled off, and the normal insert/

update/replace/remove behavior is executed, which will fail because of an

array being used as input.

* Fixed issue BTS-424: fix invalid input row handling in WINDOW execution.

* Fixed ES-881: ensure that LDAP options for async, referrals and restart set

the off value correctly. Otherwise, this can result in an "operations error".

* Fixed DEVSUP-764 (SEARCH-7): inconsistent BM25 scoring for LEVENSHTEIN_MATCH

function.

* Return error 1948 when a negative edge was detected during a

weighted traversal or was used as default weight.

* Added 2 options to allow HTTP redirection customization for root ("/") call of

HTTP API:

`--http.permanently-redirect-root`: if true (default), use a permanent

redirection (use HTTP 301 code), if false fall back to temporary redirection

(use HTTP 302 code);

`--http.redirect-root-to`: redirect of root URL to a specified path (redirects

to "/_admin/aardvark/index.html" if not set (default)).

* Fixes BTS-416. During shutdown, a shard leader wrongly reported that

it could not drop a shard follower instead of correctly indicating

the shutdown as reason.

* Fix various issues related to the new WINDOW operation (see BTS-402)

- Improved explain output for ISO 8601 duration strings and fixed missing week

component.

- Improved validation of input data and error messages.

- Prevent FILTERs from being moved beyond a WINDOW.

* Fixes BTS-417. In some cases an index did not consider both bounds (lower and

upper) for a close range scan if both bounds are expressed using the same

operator, e.g., `FILTER doc.beginDate >= lb AND ub >= doc.beginDate`.

* When writing to starting shard leader respond with specific

503. Fixes BTS-390.

* Reduced the agency store public members, for simpler support long-term.

* Fixed bug in error reporting when a database create did not work, which

lead to a busy loop reporting this error to the agency.

* Added a number of tests for the Agency Store public members.

* Improve error reporting for Merkle tree operations and improve memory usage

for unused trees by hibernating them. In addition, add some backoff to shard

synchronization in case there are repeated sync failures for the same shard.

* Fixes pregel lifetime management. Previously shutting down the server while a

pregel job was still running could result in a segfault or a shutdown hanger.

* Updated bundled version of Snappy library to 1.1.9.

* Fixed various issues (mainly data races) reported by ThreadSanitizer.

* Improve "Shards" view in web UI so that the shards of individual collections

can be expanded and collapsed without affecting the display of any other

shards. Also added a "Toggle all" button the web UI to expand/collapse the

shards for all collections.

* Fixed BTS-403: Hot restores must also clear relevant `Current` keys. The

overriding of the `Plan` entries needs to be reflected in `Current` to avoid

conflicts in maintenance jobs.

* Log a proper message if an unexpected state is encountered when taking over

shard leadership. In addition, make the change to the internal followerinfo

state atomic so that it cannot be semi-changed.

* Improve exception safety for maintenance thread and shard unlock

operations.

* Fixed two bugs in fuerte with HTTP/2 and VST connections.

One could lead to ordered timeouts not being honoured. The other could

lead to an ordered callback be called multiple times.

* Fix response when isBuilding could not be removed from newly created

collection, when agency precondition fails. This can happen, when own

rebootId increment has triggered plan entry to be removed.

* When writing to starting shard leader respond with specific

503. Fixes BTS-390.

* Introduce a new internal error code for cases where a call cannot succeed

because the server startup phase is still in progress. This error will be

mapped to the HTTP status code 503 (service unavailable).

One example where this can happen is when trying to authenticate a request,

but the _users collection is not yet available in the cluster.

* Fixed issue BTS-354: Assertion related to getCollection.

* Fixed a use after free bug in the connection pool.

* Fix DEVSUP-749: Fix potential deadlock when executing concurrent view/link

DDL operations and index DDL operations on the same collection.

* Fixed issue #14122: when the optimizer rule "inline-subqueries" is applied,

it may rename some variables in the query. The variable renaming was however

not carried out for traversal PRUNE conditions, so the PRUNE conditions

could still refer to obsolete variables, which would make the query fail with

errors such as

Query: AQL: missing variable ... for node ... while planning registers

* Fixed bug in error reporting when a database create did not work, which led

to a busy loop reporting this error to the agency.

* Fixed the error response if the HTTP version is not 1.0 or 1.1 and if

the Content-Length is too large (> 1 GB).

* Add a connection cache for internal replication requests.

* Improve legibility of size values (by adding KB, MB, GB, TB suffixes) to

output generated by client tools.

* Timely updates of rebootId / cluster membership of DB servers and

coordinators in ClusterInfo. Fixes BTS-368 detected in chaos tests.

* Guarded access only to ActionBase::_result.

* Fixed proper return value in sendRequestRetry if server is shutting down.

* Updated arangosync to 2.2.0.

* Fixed internal issue #798: In rare case when remove request

completely cleans just consolidated segment commit could be cancelled

and documents removed from collection may be left dangling in the ArangoSearch index.

Also fixes ES-810 and BTS-279.

* Retry if an ex-leader can no longer drop a follower because it is no longer

leading.

* Fixed a small problem in fuerte which could lead to an assertion failure.

* Upgrade jemalloc version to latest stable dev.

* Fixed issue BTS-373: ASan detected possible heap-buffer-overflow at

arangodb::transaction::V8Context::exitV8Context().

* Allow to specify a fail-over LDAP server. Instead of "--ldap.OPTION" you need

to specify "--ldap2.OPTION". Authentication / Authorization will first check

the primary LDAP server. If this server cannot authenticate a user, it will

try the secondary one. It is possible to specify a file containing all users

that the primary (or secondary) LDAP server is handling by specifying the

option "--ldap.responsible-for". This file must contain the usernames

line-by-line.

* Make the time-to-live (TTL) value of a streaming cursor only count after

the response has been sent to the client.

* Improve performance of batch CRUD operations (insert, update, replace,

remove) if some of the documents in the batch run into write-write conflicts.

Rolling back partial operations in case of a failure is very expensive

because it requires rebuilding RocksDB write batches for the transaction

from scratch. Rebuilding write batches takes time proportional to the number

of operations in the batch, and for larger batches the cost can be

prohibitive.

Now we are not rolling back write batches in some situations when this is

not required, so that in many cases running into a conflict does not have

that high overhead. There can still be issues when conflicts happen for index

entries, but a lot of previously problematic cases should now work better.

* Allow AQL variable names starting with an underscore, as stated in the docs.

* Fix crashes during arangorestore operations due to usage of wrong pointer

value for updating user permissions.

* Added option `--query-max-runtime` to arangoexport, in order to control

maximum query runtime.

* Fix BTS-340: AQL expressions similar to `x < 3 || x` are no longer erroneously

be reduced to `x < 3` by the optimizer rule remove-redundant-or.

* Changed default value of arangodump's `--envelope` option from `true` to

`false`. This allows using higher parallelism in arangorestore when

restoring large collection dumps. As a side-effect, this will also decrease

the size of dumps taken with arangodump, and should slightly improve dump

speed.

* Improve parallelism capabilities of arangorestore.

arangorestore can now dispatch restoring data chunks of a collection to idle

background threads, so that multiple restore requests can be in flight for

the same collection concurrently.

This can improve restore speed in situations when there are idle threads

left (number of threads can be configured via arangorestore's `--threads`

option) and the dump file for the collection is large.

The improved parallelism is only used when restoring dumps that are in the

non-enveloped format. This format has been introduced with ArangoDB 3.8.

The reason is that dumps in the non-enveloped format only contain the raw

documents, which can be restored independent of each other, i.e. in any

order. However, the enveloped format may contain documents and remove

operations, which need to be restored in the original order.

* Fix BTS-374: thread race between ArangoSearch link unloading and storage

engine WAL flushing.

* Fix thread race between ArangoSearch link unloading and storage engine

WAL flushing.

* change arangosh client behavior:

- *_RAW methods will never add a `body` to HEAD responses

- *_RAW methods will now always return velocypack-typed responses in Buffers

- `--server.force-json` will now be applied as default, overrideable

by user code

* Add HTTP REST API endpoint POST `/_api/cursor/<cursor-id>` as a drop-in

replacement for PUT `/_api/cursor/<cursor-id>`. The POST API is functionally

equivalent to the existing PUT API. The benefit of using the POST API is

that HTTP POST requests will not be considered as idempotent, so proxies

may not retry them if they fail. This was the case with the existing PUT

API, as HTTP PUT requests can be considered idempotent according to the

HTTP specification.

The POST API is now used internally by ArangoDB's own requests, including

the web UI and the client tools. That means the web UI and client tools

will only work with ArangoDB versions that have support for the new POST

API. This is true for recent 3.7 and 3.8 versions, as the POST API will be

backported there as well.

* Fixed BTS-360 and ES-826: sporadic ArangoSearch error `Invalid RL encoding in

'dense_fixed_offset_column_key'`.

* Add value of `_key` to more insert/update/replace/remove error messages

so it is easier to figure out which document caused unique constraint

violations and/or write-write conflict during a multi-document write

operation.

* Fix cluster internal retry behavior for network communications. In particular

retry on 421 (leader refuses operation). This leads to the cluster letting

less internal errors out to clients.

* Don't display obsoleted startup options and sections in `--help` and

`--help-.` commands. Also rename "global" to "general" options.

* Added option `--query.require-with` to make AQL in single server mode also

require `WITH` clauses where the cluster would need them.

The option is turned off by default, but can be turned on in single servers

to remove this behavior difference between single servers and clusters,

making later a transition from single server to cluster easier.

* Fixed a problem in document batch operations, where errors from one shard

were reported multiple times, if the shard is completely off line.

* Removed assertion for success of a RocksDB function. Throw a proper

exception instead.

* Show peak memory usage in AQL query profiling output.

* Micro improvements for Pregel job API and documentation:

- Added a few useful attributes to Pregel HTTP API docs.

- Added "parallelism" attribute to the result of Pregel job status responses,

so that the effective parallelism is reported back.

- Make sure "computationTime" in Pregel job status response does not

underflow in case of errors.

* Prevent arangod from terminating with "terminate called without an active

exception" (SIGABRT) in case an out-of-memory exception occurs during

creating an ASIO socket connection.

* UI builds are now using the yarn package manager instead of the previously

used node package manager.

* Fixed issue #13169: arangoimport tsv conversion of bools and null, although

switched off by `--convert false`.

Importing unquoted `null`, `false` and `true` literals from delimited files

get imported as strings now if `convert` is explicitly turned off. It

previously affected unquoted numbers only.

* Web UI: Highlight binary and hexadecimal integer literals in AQL queries.

* Fix BTS-350, BTS-358: Fixed potential startup errors due to global

replication applier being started before end of database recovery procedure.

Also fixed potential shut down errors due to global replication applier

being shut down in parallel to a concurrent shut down attempt.

* Experimentally switch to wyhash (from xxhash) for velocypack. This is an

experiment in devel to check if it produces any observable speedups.

* Updated ArangoDB Starter to 0.15.0.

* Remove deprecated HTTP REST API `/_api/export`. This API was deprecated

in a previous version because it was not supported in clusters and was

also covered completely by streaming AQL queries for the RocksDB storage

engine.

* Added error handling for figures command in cluster. Previously errors

returned by shards were ignored when aggregating the individual responses.

* Updated ArangoDB Starter to 0.15.0-preview-4.

* Fixed CPPCHECK warning or added suppression.

* Added enterprise-build-repository and oskar-build-repository to `--version`

as `enterprise-build-repository` and `oskar-build-repository`.

* Clean up replication code and remove a 3.2-compatibility mode that was

only useful when replicating from a leader < ArangoDB version 3.3.

* Obsolete option `--database.old-system-collections`. This option has no

meaning in ArangoDB 3.9, as old system collections will not be created

anymore in this version. The option was deprecated in 3.8 and announced

to be obsoleted.

* Upgrade velocypack to latest, C++17-only version.

* Make arangovpack more powerful, by supporting different input and output

formats (json and vpack, plain or hex-encoded).

The arangovpack options `--json` and `--pretty` have been removed and have

been replaced with separate options for specifying the input and output

types:

* `--input-type` ("json", "json-hex", "vpack", "vpack-hex")

* `--output-type` ("json", "json-pretty", "vpack", "vpack-hex")

The previous option `--print-non-json` has been replaced with the option

`--fail-on-non-json` which makes arangovpack fail when trying to emit

non-JSON types to JSON output.

* Fix undefined behavior in dynarray constructor when running into

an out-of-memory exception during construction. In arangod, this can only

happen during metrics objects construction at program start.

* Added option `--headers-file` to arangoimport, to optionally read CSV/TSV

headers from a separate file.

* Updated ArangoDB Starter to 0.15.0-preview-3.

* Fixed issue BTS-353: memleak when running into an out-of-memory situation

while repurposing an existing AqlItemBlock.

* Fix logging of urls when using `--log.level requests=debug`. There was an

issue since v3.7.7 with the wrong URL being logged in request logging if

multiple requests were sent over the same connection. In this case, the

request logging only reported the first URL requested in the connection,

even for all subsequent requests.

* Deprecate option `--rocksdb.exclusive-writes`, which was meant to serve

only as a stopgap measure while porting applications from the MMFiles

storage engine to RocksDB.

* Added startup option `--query.allow-collections-in-expressions` to control

whether collection names can be used in arbitrary places in AQL expressions,

e.g. `collection + 1`. This was allowed before, as a collection can be seen

as an array of documents. However, referring to a collection like this in a

query would materialize all the collection's documents in RAM, making such

constructs prohibitively expensive for medium-size to large-size collections.

The option can now be set to `false` to prohibit accidental usage of

collection names in AQL expressions. With that setting, using a collection

inside an arbitrary expression will trigger the error `collection used as

expression operand` and make the query fail.

Even with the option being set to `false`, it is still possible to use

collection names in AQL queries where they are expected, e.g. `FOR doc IN

collection RETURN doc`.

* Remove obsolete API endpoint /_admin/repair/distributeShardsLike`. This

API was intended to correct some bad state introduced before 3.2.12 or 3.3.4,

respectively. It had to be invoked manually by callers and there was never

any driver support for it.

* Remove now-unused SubqueryExecutor. This is an internal change only and

should not have any effect on queries, as from 3.8 onwards only spliced

subqueries should be used in query execution plans and during query

execution.

* Remove CMake control variable `UNCONDITIONALLY_BUILD_LOG_MESSAGES`.

Now, any maintainer mode build will build all log messages automatically,

so we will have full coverage of log message construction during our

tests. In non-maintainer mode, log messages are still only built when

actually required. This simplifies the build and increases coverage.

* Updated ArangoDB Starter to 0.15.0-preview-2.

* Updated OpenSSL to 1.1.1k and OpenLDAP to 2.4.58.

* Updated arangosync to 2.0.1.

* Introduce metrics for AQL query memory limit violations:

- `arangodb_aql_global_query_memory_limit_reached`: Total number of times the

global query memory limit was violated.

- `arangodb_aql_local_query_memory_limit_reached`: Total number of times a

local query memory limit was violated.

* Set the default value for `--query.global-memory-limit` to around 90% of RAM,

so that a global memory limit is now effective by default.

The default global memory limit value is calculated by a formula depending on

the amount of available RAM and will result in the following values for

common RAM sizes:

RAM: 0 (0MiB) Limit: 0 unlimited, %mem: n/a

RAM: 134217728 (128MiB) Limit: 33554432 (32MiB), %mem: 25.0

RAM: 268435456 (256MiB) Limit: 67108864 (64MiB), %mem: 25.0

RAM: 536870912 (512MiB) Limit: 255013683 (243MiB), %mem: 47.5

RAM: 805306368 (768MiB) Limit: 510027366 (486MiB), %mem: 63.3

RAM: 1073741824 (1024MiB) Limit: 765041049 (729MiB), %mem: 71.2

RAM: 2147483648 (2048MiB) Limit: 1785095782 (1702MiB), %mem: 83.1

RAM: 4294967296 (4096MiB) Limit: 3825205248 (3648MiB), %mem: 89.0

RAM: 8589934592 (8192MiB) Limit: 7752415969 (7393MiB), %mem: 90.2

RAM: 17179869184 (16384MiB) Limit: 15504831938 (14786MiB), %mem: 90.2

RAM: 25769803776 (24576MiB) Limit: 23257247908 (22179MiB), %mem: 90.2

RAM: 34359738368 (32768MiB) Limit: 31009663877 (29573MiB), %mem: 90.2

RAM: 42949672960 (40960MiB) Limit: 38762079846 (36966MiB), %mem: 90.2

RAM: 68719476736 (65536MiB) Limit: 62019327755 (59146MiB), %mem: 90.2

RAM: 103079215104 (98304MiB) Limit: 93028991631 (88719MiB), %mem: 90.2

RAM: 137438953472 (131072MiB) Limit: 124038655509 (118292MiB), %mem: 90.2

RAM: 274877906944 (262144MiB) Limit: 248077311017 (236584MiB), %mem: 90.2

RAM: 549755813888 (524288MiB) Limit: 496154622034 (473169MiB), %mem: 90.2

* The old metrics API contains the following gauges which should actually be

counters:

* arangodb_scheduler_jobs_dequeued

* arangodb_scheduler_jobs_submitted

* arangodb_scheduler_jobs_done

Therefore the new v2 metric api adds the following counters:

* arangodb_scheduler_jobs_dequeued_total

* arangodb_scheduler_jobs_submitted_total

* arangodb_scheduler_jobs_done_total

These counters are only visible in the new v2 metrics API and replace the old

metrics which are suppressed for v2.

* Fix connectionTime statistic. This statistic should provide the distribution

of the connection lifetimes, but in previous versions the tracking was broken

and no values were reported.

* Add an option for locking down all endpoints in the `/_admin/cluster` REST

API for callers without a proper JWT set in the request. There is a new

startup option `--cluster.api-jwt-policy` that allows *additional* checks

for a valid JWT in requests to sub-routes of `/_admin/cluster`. The

possible values for the startup option are:

- "jwt-all": requires a valid JWT for all accesses to `/_admin/cluster` and

its sub-routes. If this configuration is used, the "Cluster" and "Nodes"

sections of the web interface will be disabled, as they are relying on the

ability to read data from several cluster APIs.

- "jwt-write": requires a valid JWT for write accesses (all HTTP methods

except HTTP GET) to `/_admin/cluster`. This setting can be used to allow

privileged users to read data from the cluster APIs, but not to do any

modifications. All existing permissions checks for the cluster API routes

are still in effect with this setting, meaning that read operations without

a valid JWT may still require dedicated other permissions (as in 3.7).

- "jwt-compat": no *additional* access checks are in place for the cluster

APIs. However, all existing permissions checks for the cluster API routes

are still in effect with this setting, meaning that all operations may

still require dedicated other permissions (as in 3.7).

The default value for the option is `jwt-compat`, which means this option

will not cause any extra JWT checks compared to 3.7.

* Increase default idle timeout in streaming transactions from 10 seconds to

60 seconds, and make the timeout configurable via a startup parameter

`--transaction.streaming-idle-timeout`.

* Use RebootTracker to abort cluster transactions on DB servers should the

originating coordinator die or be rebooted. The previous implementation left

the coordinator's transactions open on DB servers until they timed out there.

Now, the coordinator's unavailability or reboot will be detected as early as

it is reported by the agency, and all open transactions from that coordinator

will be auto-aborted on DB servers.

* Fix shortName labels in metrics, in particular for agents.

* Fix a race in LogAppender::haveAppenders.

`haveAppenders` is called as part of audit logging. It accesses internal maps

but previously did not hold a lock while doing so.

* Fix implicit capture of views in a context of JS transaction.

* Fix a crash caused by returning a result produced by ANALYZER function.

* Update the Web UI's list of built-in AQL functions for proper syntax

highlighting in the query editor.

* Bug-fix in the case of very rare network issues there was a chance that

an AQL query could get stuck during a cleanup and after a commit.

This would cause the client to receive a timeout, and the Coordinator

blocking a Scheduler thread. This situation is sorted out and the thread

will not be blocked anymore. We also added logs in case the query

could not successfully be cleaned up, which would leave locks on shards

behind.

* Switched to GCC 10 as the default compiler and use Sandy Bridge as the

default required architecture (Linux, macOS binaries).

* Fix an assertion failure that occurred when restoring view definitions from

a cluster into a single server.

* Added new ArangoSearch analyzer type "stopwords".

* Fix error message in case of index unique constraint violations. They were

lacking the actual error message (i.e. "unique constraint violated") and

only showed the index details. The issue was introduced only in devel in Feb.

* Removed obsolete metrics in new v2 metric API. Those metrics' values were

identical to the sum value of histograms.

* Allow process-specific logfile names.

This change allows replacing '$PID' with the current process id in the

`--log.output` and `--audit.output` startup parameters.

This way it is easier to write process-specific logfiles.

* Backport a bugfix from upstream RocksDB for opening encrypted files with

small sizes. Without the bugfix, the server may run into assertion failures

during recovery.

* Fix duplicate leaving of V8 contexts when returning streaming cursors.

The `exitContext` call done on query shutdown could previously try to exit

the V8 context multiple times, which would cause undefined behavior. Now

we are tracking if we already left the context to prevent duplicate invocation.

* In a cluster, do not create the collections `_statistics`, `_statistics15` and

`statisticsRaw` on DB servers. These collections should only be created by the

coordinator, and should translate into 2 shards each on DB servers. But there

shouldn't be shards named `_statistics*` on DB servers.

* Fixed two bogus messages about hotbackup restore:

- Coordinators unconditionally logged the message "Got a hotbackup restore

event, getting new cluster-wide unique IDs..." on shutdown. This was not

necessarily related to a hotbackup restore.

- DB servers unconditionally logged the message "Strange, we could not

unregister the hotbackup restore callback." on shutdown, although this was

meaningless.

* Rename "save" return attribute to "dst" in AQL functions `DATE_UTCTOLOCAL` and

`DATE_LOCALTOUTC`.

* Fix potentially undefined behavior when creating a CalculationTransactionContext

for an arangosearch analyzer. An uninitialized struct member was passed as an

argument to its base class. This potentially had no observable effects, but

should be fixed.

* Retry a cluster internal network request if the connection comes from the

pool and turns out to be stale (connection immediately closed). This fixes

some spurious errors after a hotbackup restore.

* Fix progress reporting for arangoimport with large files on Windows.

Previously, progress was only reported for the first 2GB of data due to an

int overflow.

* Log the actual signal instead of "control-c" and also include the process id

of the process that sent the signal.

* Fixed Github issue #13665: Improve index selection when there are multiple

candidate indexes.

* When dropping a collection or an index with a larger amount of documents, the

key range for the collection/index in RocksDB gets compacted. Previously, the

compaction was running in foreground and thus would block the deletion operations.

Now, the compaction is running in background, so that the deletion operations

can return earlier.

The maximum number of compaction jobs that are executed in background can be

configured using the new startup parameter `--rocksdb.max-parallel-compactions`,

which defaults to 2.

* Put Sync/LatestID into hotbackup and restore it on hotbackup restore

if it is in the backup. This helps with unique key generation after

a hotbackup is restored to a young cluster.

* Fixed a bug in the index count optimization that doubled counted documents

when using array expansions in the fields definition.

* Don't store selectivity estimate values for newly created system collections.

Not storing the estimates has a benefit especially for the `_statistics`

system collections, which are written to periodically even on otherwise

idle servers. In this particular case, the actual statistics data was way

smaller than the writes caused by the index estimate values, causing a

disproportional overhead just for maintaining the selectivity estimates.

The change now turns off the selectivity estimates for indexes in all newly

created system collections, and for new user-defined indexes of type

"persistent", "hash" or "skiplist", there is now an attribute "estimates"

which can be set to `false` to disable the selectivity estimates for the index.

The attribute is optional. Not setting it will lead to the index being

created with selectivity estimates, so this is a downwards-compatible change

for user-defined indexes.

* Added startup option `--query.global-memory-limit` to set a limit on the

combined estimated memory usage of all AQL queries (in bytes).

If this option has a value of `0`, then no memory limit is in place.

This is also the default value and the same behavior as in previous versions

of ArangoDB.

Setting the option to a value greater than zero will mean that the total memory

usage of all AQL queries will be limited approximately to the configured value.

The limit is enforced by each server in a cluster independently, i.e. it can

be set separately for coordinators, DB servers etc. The memory usage of a

query that runs on multiple servers in parallel is not summed up, but tracked

seperately on each server.

If a memory allocation in a query would lead to the violation of the configured

global memory limit, then the query is aborted with error code 32 ("resource

limit exceeded").

The global memory limit is approximate, in the same fashion as the per-query

limit provided by the option `--query.memory-limit` is. Some operations,

namely calls to AQL functions and their intermediate results, are currently

not properly tracked.

If both `--query.global-memory-limit` and `--query.memory-limit` are set,

the former must be set at least as high as the latter.

To reduce the cost of globally tracking the memory usage of AQL queries, the

global memory usage counter is only updated in steps of 32 kb, making

this also the minimum granularity of the global memory usage figure.

In the same fashion, the granularity of the peak memory usage counter inside

each query was also adjusted to steps of 32 kb.

* Added startup option `--query.memory-limit-override` to control whether

individual AQL queries can increase their memory limit via the `memoryLimit`

query option. This is the default, so a query that increases its memory limit

is allowed to use more memory.

The new option `--query.memory-limit-override` allows turning this behavior

off, so that individual queries can only lower their maximum allowed memory

usage.

* Added metric `arangodb_aql_global_memory_usage` to expose the total amount

of memory (in steps of 32 kb) that is currently in use by all AQL queries.

* Added metric `arangodb_aql_global_memory_limit` to expose the memory limit

from startup option `--query.global-memory-limit`.

* Allow setting path to the timezone information via the `TZ_DATA` environment

variable, in the same fashion as the currently existing `ICU_DATA` environment

variable. The `TZ_DATA` variable is useful in environments` that start arangod

from some unusual locations, when it can't find its `tzdata` directory

automatically.

* Fixed a bug in query cost estimation when a NoResults node occured in a spliced

subquery. This could lead to a server crash.

* Fix slower-than-necessary arangoimport behavior:

arangoimport has a built-in rate limiter, which can be useful for importing

data with a somewhat constant rate. However, it is enabled by default and

limits imports to 1MB per second. These settings are not useful.

This change turns the rate limiting off by default, and sets the default

chunk size to 8MB (up from 1MB) as well. This means that arangoimport will

send larger batches to the server by default. The already existing `--batch-size`

option can be used to control the maximum size of each batch.

The new parameter `--auto-rate-limit` can now be used to toggle rate limiting.

It defaults to off, whereas previously rate limiting was enabled by default

unless `--batch-size` was specified when arangoimport was invoked.

* The cluster dashboard charts in the web UI are now more readable during the

initialization phase. Additionally, the amount of agents are now displayed

there as well. An agent failure will also appear here in case it exists.

* Added more useful information during the SmartGraph creation in the web UI

in case the current database is a OneShard database.

* Add support for building with Zen 3 CPU when optimizing for the local

architecture.

* The web UI's node overview now displays also agent information (cluster only).

* The statistics view in the web UI does now provide more system specific

information in case the Metrics API is enabled. Different statistics may

be visible depending on the operating system.

* Added metrics documentation snippets and infrastructure for that.

* Added a new cluster distribution view to the web UI. The view includes general

details about cluster-wide distribution in general as well as more detailed

shard distribution specific information.

* Follower primaries respond with

TRI_ERROR_CLUSTER_SHARD_FOLLOWER_REFUSES_OPERATION

to any read request. Fixes a wrongly responded 404 from chaos

tests.

* Fixed Github issue #13632: Query Fails on Upsert with Replace_nth.

* Reasonably harden MoveShard against invalid VelocyPack input.

* Removed older reference to VelocyPackDumper.

* Added `--documents-per-batch` option to arangoexport.

This option allows to control the number of documents to be returned by each

server-side batch. It can be used to limit the number of documents per batch

when exporting collections with large documents.

* Added a new metrics view to the web UI. This view can be used in a clustered

environment as well as in a single instance. Metrics are displayed either in

a tabular format or as plain text (Prometheus Text-based format).

Additionally, the metrics can be downloaded there.

* Added a new maintenance mode tab to the web UI in cluster mode.

The new tab shows the current state of the cluster supervision maintenance

and allows to enable/disable the maintenance mode from there. The tab will

only be visible in the `_system` database. The required privileges for

displaying the maintenance mode status and/or changing it are the as for

using the REST APIs for the maintenance mode.

* Fixed a problem that coordinators would vanish from the UI and the Health

API if one switched the agency Supervision into maintenance mode and kept

left that maintenance mode on for more than 24h.

* Fixed a bug in the web interface that displayed the error "Not authorized to

execute this request" when trying to create an index in the web interface in a

database other than `_system` with a user that does not have any access

permissions for the `_system` database.

The error message previously displayed error actually came from an internal

request made by the web interface, but it did not affect the actual index

creation.

* Added ability to display Coordinator and DBServer logs from inside the Web UI

in a clustered environment when privileges are sufficient.

Additionally, displayed log entries can now be downloaded from the web UI in

single server and in cluster mode.

* The Web UI's info view of a collection now displays additional properties and

statistics (e.g. RocksDB related figures, sharding information and more).

* Improve progress reporting for shard synchronization in the web UI.

The UI will now show how many shards are actively syncing data, and will

provide a better progress indicator, especially if there is more than one

follower for a shard.

* Fixed issue BTS-309: The Graph API (Gharial) did not respond with the correct

HTTP status code when validating edges. It now responds with 400 (Bad Request)

as documented and a new, more precise error code (1947) and message if a vertex

collection referenced in the _from or _to attribute is not part of the graph.

* Added `--shard` option to arangodump, so that dumps can be restricted to one or

multiple shards only.

* Enable statistics in web UI in non-`_system` databases in cluster mode.

In cluster mode, the web UI dashboard did not display statistics properly

when not being logged in to the `_system` database. For all other databases

than `_system`, no statistics were displayed but just some "No data..."

placeholders.

Statistics for non-`_system` databases were not properly displayed since

3.7.6 due to an internal change in the statistics processing.

In addition, a new startup option `--server.statistics-all-databases`

controls whether cluster statistics are displayed in the web interface for

all databases (if the option is set to `true`) or just for the system

database (if the option is set to `false`).

The default value for the option is `true`, meaning statistics will be

displayed in the web interface for all databases.

* Add optional hostname logging to log messages.

Whether or not the hostname is added to each log message can be controlled via

the new startup option `--log.hostname`. Its default value is the empty string,

meaning no hostname will be added to log messages.

Setting the option to an arbitrary string value will make this string be logged

in front of each regular log message, and inside the `hostname` attribute in

case of JSON-based logging. Setting the option to a value of `auto` will use

the hostname as returned by `gethostbyname`.

* Added logging of elapsed time of ArangoSearch commit/consolidation/cleanup

jobs.

* Added list-repeat AIR primitive that creates a list containing n copies of the input value.

* Updated OpenSSL to 1.1.1j and OpenLDAP to 2.4.57.

* Prevent arangosh from trying to connect after every executed command.

This fixes the case when arangosh is started with default options, but no

server is running on localhost:8529. In this particular case, arangosh will

try to connect on startup and after every executed shell command. The

connect attempts all fail and time out after 300ms.

In this case we now don't try to reconnect after every command.

* Added 'custom-query' testcase to arangobench to allow execution of custom

queries.

This also adds the options `--custom-query` and `--custom-query-file` for

arangobench.

* Addition to the internal Refactoring of K_PATHS feature: K_PATHS queries are

now being executed on the new refactored graph engine in a clustered

environment. This change should not have any visible effect on users.

* Reduce memory footprint of agency Store in Node class.

* Cleanup old hotbackup transfer jobs in agency.

* On Windows create a minidump in case of an unhandled SEH exception for

post-mortem debugging.

* Add JWT secret support for arangodump and arangorestore, i.e. they now also

provide the command-line options `--server.ask-jwt-secret` and

`--server.jwt-secret-keyfile` with the same meanings as in arangosh.

* Add optional hyperlink to program option sections for information purposes,

and add optional sub-headlines to program options for better grouping.

These changes will be visible only when using `--help`.

* For Windows builds, remove the defines `_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS`

and `_ENABLE_ATOMIC_ALIGNMENT_FIX` that were needed to build Boost components

with MSVC in older versions of Boost and MSVC.

Both of these defines are obsolete nowadays.

* Database initial sync considers document count on leader for

estimating timeouts when over 1 million docs on leader.

* EE only bugfix: On DisjointSmartGraphs that are used in anonymous way,

there was a chance that the query could fail, if non-disjoint collections

were part of the query. Named DisjointSmartGraphs have been save to this bug.

Example:

DisjointSmartGraph (graph) on vertices -edges-> vertices

Query:

WITH vertices, unrelated

FOR out IN 1 OUTBOUND "v/1:1" edges

FOR u IN unrelated

RETURN [out, u]

The "unrelated" collection was pulled into the DisjointSmartGraph, causing

the AQL setup to create erroneous state.

This is now fixed and the above query works.

This query:

WITH vertices, unrelated

FOR out IN 1 OUTBOUND "v/1:1" GRAPH "graph"

FOR u IN unrelated

RETURN [out, u]

was not affected by this bug.

* Fixed issue BTS-268: fix a flaky Foxx self-heal procedure.

* Fixed issue DEVSUP-720: Within an AQL query, the "COLLECT WITH COUNT INTO"

statement could lead to a wrong count output when used in combination with

an index which has been created with an array index attribute.

* Fixed issue #13117: Aardvark: Weird cursor offsets in query editor.

Disabled font ligatures for Ace editor in Web UI to avoid rare display issue.

* Fixed ES-784 regression related to encryption cipher propagation to

ArangoSearch data.

* Improved the wording for sharding options displayed in the web interface.

Instead of offering `flexible` and `single`, now use the more intuitive

`Sharded` and `OneShard` options, and update the help text for them.

* Make all AQL cursors return compact result arrays.

* Fix profiling of AQL queries with the `silent` and `stream` options sets in

combination. Using the `silent` option makes a query execute, but discard all

its results instantly. This led to some confusion in streaming queries, which

can return the first query results once they are available, but don't

necessarily execute the full query.

Now, `silent` correctly discards all results even in streaming queries, but

this has the effect that a streaming query will likely be executed completely

when the `silent` option is set. This is not the default however, and the

`silent` option is normally not set. There is no change for streaming queries

if the `silent` option is not set.

As a side-effect of this change, this makes profiling (i.e. using

`db._profileQuery(...)` work for streaming queries as well. Previously,

profiling a streaming query could have led to some internal errors, and even

query results being returned, even though profiling a query should not return

any query results.

* Make dropping of indexes in cluster retry in case of precondition failed.

When dropping an indexes of a collection in the cluster, the operation could

fail with a "precondition failed" error in case there were simultaneous

index creation or drop actions running for the same collection. The error

was returned properly internally, but got lost at the point when

`<collection>.dropIndex()` simply converted any error to just `false`.

We can't make `dropIndex()` throw an exception for any error, because that

would affect downwards-compatibility. But in case there is a simultaneous

change to the collection indexes, we can just retry our own operation and

check if it succeeds then. This is what `dropIndex()` will do now.

* Try to raise file descriptors limit in local start scripts (in `scripts/`

directory - used for development only).

* Fix error reporting in the reloadTLS route.

* Fix potential undefined behavior when iterating over connected nodes in an