devel

-----

* Fix a bug in the agency Supervision which could lead to removeFollower

jobs constantly being created and immediately stopped again.

* Limit additional replicas in failover cases to +2.

* Print a version mismatch (major/minor version difference) between the

arangosh version and the remote arangod version at arangosh startup.

* Internal Refactoring of K_PATH feature, with the goal to have all GraphAlgorithms on

the same framework. This change should not have any visible effect on users.

* Fixed an endless busy loop which could happen if a coordinator tries to

roll back a database creation, but the database has already been dropped

by other means.

* Removed server-side JavaScript object `ArangoClusterComm`, so it cannot be

used from inside JavaScript operations or Foxx.

The `ArangoClusterComm` object was previously used inside a few internal

JavaScript operations, but was not part of the public APIs.

* Restrict access to functions inside JavaScript objects `ArangoAgency` and

`ArangoAgent` to JavaScript code that is running in privileged mode, i.e.

via the server's emergency console, the `/_admin/execute` API (if turned on)

or internal bootstrap scripts.

* Added startup option `--javascript.transactions` to allow turning off JavaScript

transactions if not needed. The default value for this option is `true`, meaning

JavaScript transactions are available as before.

However, using this option they can be turned off by admins to limit the amount

of JavaScript user code that is executed.

* Introduce a default memory limit for AQL queries, to prevent rogue queries from

consuming the entire memory available to an arangod instance.

The limit is introduced via changing the default value of the option `--query.memory-limit`

from previously `0` (meaning: no limit) to a dynamically calculated value.

The per-query memory limits defaults are now:

Available memory: 134217728 (128MiB) Limit: 0 (0MiB), %mem: 0.0

Available memory: 268435456 (256MiB) Limit: 0 (0MiB), %mem: 0.0

Available memory: 536870912 (512MiB) Limit: 201326592 (192MiB), %mem: 37.5

Available memory: 805306368 (768MiB) Limit: 402653184 (384MiB), %mem: 50.0

Available memory: 1073741824 (1024MiB) Limit: 603979776 (576MiB), %mem: 56.2

Available memory: 2147483648 (2048MiB) Limit: 1288490189 (1228MiB), %mem: 60.0

Available memory: 4294967296 (4096MiB) Limit: 2576980377 (2457MiB), %mem: 60.0

Available memory: 8589934592 (8192MiB) Limit: 5153960755 (4915MiB), %mem: 60.0

Available memory: 17179869184 (16384MiB) Limit: 10307921511 (9830MiB), %mem: 60.0

Available memory: 25769803776 (24576MiB) Limit: 15461882265 (14745MiB), %mem: 60.0

Available memory: 34359738368 (32768MiB) Limit: 20615843021 (19660MiB), %mem: 60.0

Available memory: 42949672960 (40960MiB) Limit: 25769803776 (24576MiB), %mem: 60.0

Available memory: 68719476736 (65536MiB) Limit: 41231686041 (39321MiB), %mem: 60.0

Available memory: 103079215104 (98304MiB) Limit: 61847529063 (58982MiB), %mem: 60.0

Available memory: 137438953472 (131072MiB) Limit: 82463372083 (78643MiB), %mem: 60.0

Available memory: 274877906944 (262144MiB) Limit: 164926744167 (157286MiB), %mem: 60.0

Available memory: 549755813888 (524288MiB) Limit: 329853488333 (314572MiB), %mem: 60.0

As previously, a memory limit value of `0` means no limitation.

The limit values are per AQL query, so they may still be too high in case queries

run in parallel. The defaults are intentionally high in order to not stop any valid,

previously working queries from succedding.

* Added startup options `--audit.queue` to control audit logging queuing

behavior (Enterprise Edition only):

The option controls whether audit log messages are submitted to a queue

and written to disk in batches or if they should be written to disk directly

without being queued.

Queueing audit log entries may be beneficial for latency, but can lead to

unqueued messages being lost in case of a power loss or crash. Setting

this option to `false` mimics the behavior from 3.7 and before, where

audit log messages were not queued but written in a blocking fashion.

* Fixed some situations of

[...]

SUBQUERY

FILTER

LIMIT

[...]

in AQL queries, yielding incorrect responses. A distributed

state within the subquery was not resetted correctly.

This could also lead into "shrink" errors of AQL item blocks,

or much higher query runtimes.

Fixes:

- BTS-252

- ES-687

- github issue: #13099

- github issue: #13124

- github issue: #13147

- github issue: #13305

- DEVSUP-665

* Added metric `arangodb_server_statistics_cpu_cores` to provide the number of

CPU cores visibile to the arangod process. This is the number of CPU cores

reported by the operating system to the process.

If the environment variable `ARANGODB_OVERRIDE_DETECTED_NUMBER_OF_CORES` is

set to a positive value at instance startup, this value will be returned

instead.

* `COLLECT WITH COUNT INTO x` and `COLLECT var = expr WITH COUNT INTO x` are now

internally transformed into `COLLECT AGGREGATE x = LENGTH()` and

`COLLECT var = expr AGGREGATE x = LENGTH()` respectively. In addition, any

argument passed to the `COUNT`/`LENGTH` aggregator functions are now optimized

away. This not only simplified the code, but also allows more query optimizations:

- If the variable in `COLLECT WITH COUNT INTO var` is not used, the implicit

aggregator is now removed.

- All queries of the form `COLLECT AGGREGATE x = LENGTH()` are now executed

using the count executor, which can result in significantly improved

performance.

* Minor and rare AQL performance improvement, in nested subqueries:

LET sq1 ([..] FILTER false == true LET sq2 = (<X>) [..])

where sq1 produces no data (e.g. by the above filter) for sq2,

the part <X> have been asked two times (second returns empty result),

instead of one, if and only if the mainquery executes sq1 exactly one time.

Now we get away with one call only.

In the case sq1 has data, or sq1 is executed more often, only one call was needed

(assuming the data fits in one batch).

* Updated OpenSSL to 1.1.1i and OpenLDAP to 2.4.56.

* Make internal ClusterInfo::getPlan() wait for initial plan load from agency.

* Added AQL timezone functions `DATE_TIMEZONE` and `DATE_TIMEZONES`.

* Make DB servers report storage engine health to the agency, via a new "health"

attribute in requests sent to Sync/ServerStates/<id>.

The supervision can in the future check this attribute if it is posted,

and mark servers as BAD or FAILED in case an unhealthy status is reported.

DB server health is currently determined by whether or not the storage engine

(RocksDB) has reported a background error, and by whether or not the free disk

space has reached a critical low amount. The current threshold for free disk

space is set at 1% of the disk capacity (only the disk is considered that

contains the RocksDB database directory).

The minimum required free disk space percentage can be configured using the new

startup option `--rocksdb.minimum-disk-free-percent`, which needs to be between

0 and 1 (including). A value of 0 disables the check.

The minimum required free disk space can also be configured in bytes using the

new startup option `--rocksdb.minimum-disk-free-bytes`. A value of 0 disables

this check, too.

* Failed servers are now reported consistently in the web interface, at

approximately the same time in the navigation bar and in the nodes view.

Previously these two places had their own, independent poll mechanism for the

nodes' health, and they were updated independently, which could cause an

inconsistent view of the nodes' availability.

Using only one poll mechanism instead also saves some period background requests

for the second availability check.

* Updated arangosync to 1.2.1.

* Clean up callback bin and empty promises in single-host-agency.

* Stabilize a Foxx cleanup test.

* Drop a pair of braces {} in /_admin/metrics in case of empty labels, which

makes the API adhere better to the official Prometheus syntax.

* Add some more metrics to the ConnectionPool.

* Remove HTTP "Connection" header when forwarding requests in the cluster

from one coordinator to another, and let the internal network layer

handle closing of connections and keep-alive.

* Added new metric: "arangodb_collection_lock_sequential_mode" this will count

how many times we need to do a sequential locking of collections. If this

metric increases this indicates lock contention in transaction setup.

Most likely this is caused by exlcusive locks used on collections with

more than one shard.

* Fix for BTS-213

Changed the transaction locking mechanism in the cluster case.

For all installations that do not use "exclusive" collection locks this change

will not be noticable. In case of "exclusive" locks, and collections with more

than one shard, it is now less likely to get a LOCK_TIMEOUT (ErrorNum 18).

It is still possible to get into the LOCK_TIMEOUT case, especially if

the "exclusive" operation(s) are long-running.

* Reduce overhead of audit logging functionality if audit logging is turned

off.

* Add several more attributes to audit-logged queries, namely query execution

time and exit code (0 = no error, other values correspond to general ArangoDB

error codes).

* Added "startupTime", "computationTime" and "storageTime" to Pregel result

statistics.

* Fixed a bug in maintainer mode sorting followerinfo lists the wrong way.

* Limit value of `--rocksdb.block-cache-size` to 1 GB for agent instances to

reduce agency RAM usage, unless configured otherwise. In addition, limit the

value of `--rocksdb.total-write-buffer-size` to 512 MB on agent instances for

the same reason.

* Fixed issue #13238 Thread naming API on Windows are now used only if

available in KERNEL32.DLL

* When querying the list of currently running or slow AQL queries, ignore not-yet

created databases on other coordinators.

* Added support for fetching the list of currently running and slow AQL queries

from all databases at once, by adding an `all` parameter to the following

query APIs:

* `require("@arangodb/aql/queries").current({ all: true })`: will return the

currently running queries from all databases, not just the currently

selected database.

* HTTP GET `/_api/query/current?all=true`: same, but for the HTTP REST API.

* `require("@arangodb/aql/queries").slow({ all: true })`: will return the

slow query history from all databases, not just the currently selected

database.

* HTTP GET `/_api/query/slow?all=true`: same, but for the HTTP REST API.

* `require("@arangodb/aql/queries").clearSlow({ all: true })`: will clear

the slow query history for all databases, not just the currently selected

database.

* HTTP DELETE `/_api/query/slow?all=true`: same, but for the HTTP REST API.

Using the `all` parameter is only allowed when making the call inside the

`_system` database and with superuser privileges.

* Improve performance and memory efficiency of agency restart from persisted

database directory.

* Added the following agency-related metrics:

- `arangodb_agency_client_lookup_table_size`: current number of entries

in agency client id lookup table. This gauge is available only on

agent instances.

- `arangodb_agency_cache_callback_count`: current number of entries in

agency cache callbacks table. This gauge will be effective on coordinators

and DB servers.

- `arangodb_agency_callback_count`: current number of agency callbacks

registered. This gauge will be effective on coordinators and DB servers.

* Clean up agency change log, cluster info caches.

* Added new `rocksdb_write_stalls` and `rocksdb_write_stops` counter metrics,

which should be more accurate than existing metrics related to the underlying

conditions.

* Increased the default value of `--rocksdb.min-write-buffer-number-to-merge` in

some cases when we have allocated a sufficient amount of memory to the write

buffers for this to make sense. The increased value should help prevent

compaction-induced write stalls/stops, and should only be enabled when under

conditions such that it shouldn't greatly increase the chance of flush-induced

write stalls/stops.

* Changed the default values for `--rocksdb.cache-index-and-filter-blocks` and

`--rocksdb.cache-index-and-filter-blocks-with-high-priority` to true to

improve control over memory usage.

* Lowered the minimum allowed value for `--rocksdb.max-write-buffer-number` from

9 to 4 to allow more fine-grained memory usage control.

* Fix for issue #772: Optimized document counting for ArangoSearch views.

Added new ArangoSearch view option 'countApproximate' for customizing

view count strategy.

* Fix AR-113. Disallow non-values in the AQL geo-index-optimizer rule.

* Views on SmartGraph Edge collections do not contain some documents

twice.

* Fixed agency redirect in poll api.

* Fixed issue #12248: Web UI - Added missing HTML escaping in the setup script

section of a foxx app.

* The scheduler will now run a minimum of 4 threads at all times, and the

default and minimal value for `--server.maximal-threads` has been lowered from

64 to the greater of 32 and twice the number of detected cores.

* Throttle work coming from low priority queue, according to a constant

and to an estimate taking into account fanout for multi-shard operations.

* Move to 4 priority levels "low", "medium", "high" and "maintenance" in

scheduler to ensure that maintenance work and diagnostics is always

possible, even in the case of RocksDB throttles. Do not allow any

RocksDB work on "maintenance".

* Commit replications on high priority queue.

* Essentially get rid of timeout in replication to drop followers. This

is now entirely handled via reboot and failure tracking. The timeout

has now a default minimum of 15 minutes but can still be configured via

options.

* Additional metrics for all queue lengths and low prio ongoing work.

* New metric for number and total time of replication operations.

* New metrics for number of internal requests in flight, internal request

duration, and internal request timeouts

* Fix `Gauge` assignment operators.

* Fixed and extended LDAP log messages.

* Added LDAP_OFF if referrals and restart are false.

* If LDAP search fails, also retry (update to given number of retries).

* Add cluster support for collection.checksum() method to calculate CRC

checksums for collections.

* Fix cluster-internal replication of documents with special keys (percent

character, which has a special meaning when used inside URLs).

* Fix AQL cost estimate of spliced subqueries which could lead to overly large

numbers in the explain output of such queries.

* Make all Pregel HTTP and JavaScript APIs also accept stringified execution

number values, in addition to numeric ones.

This allows passing larger execution numbers as strings, so that any data

loss due to numeric data type conversion (uint32 => double) can be avoided.

The change also makes the Pregel HTTP and JavaScript APIs for starting a

run return a stringified execution number, e.g. "12345" instead of 12345.

* Updated ArangoDB Starter to 0.14.15-1.

* Fix Windows directory creation error handling.

* Add an AQL query kill check during early pruning. Fixes issue #13141.

* Remove a case in which followers were dropped unnecessarily in streaming

transactions that replicated to the same follower.

* Fixed ES-662 by introducing refactored thread pool to make more efficient

consolidation and commit routines for links of ArangoSearch views.

Added new command line options for fine-grained ArangoSearch maintenance

control:

- `--arangosearch.commit-threads` - max number of ArangoSearch commit

threads

- `--arangosearch.commit-threads-idle` - min number of ArangoSearch

commit threads

- `--arangosearch.consolidation-threads` - max number of ArangoSearch

consolidation threads

- `--arangosearch.consolidation-threads-idle` - min number of ArangoSearch

consolidation threads

Deprecated the following command line options:

- `--arangosearch.threads` - will be used in --arangosearch.commit-threads`

and `--arangosearch.consolidation-threads` as provided value divided by 2

if set to non-autodetect value > 0

- `--arangosearch.threads-limit`

* Turn off `StatisticsWorker` thread on DB servers.

This thread was previously only running queries on the local RocksDB

instance, but using the cluster-wide collection names. So effectively it

did nothing except use a bit of background CPU. In this case it is better

to turn off the background thread entirely on the DB servers.

* Avoid the usage of std::regex when constructing date/time string values

for log messages. This is a performance optimization only.

* Reimplement coordshort request handler. The new implementation only runs

two DB queries without any additional requests to other coordinators,

resulting in reduced load on the cluster. Previously this involved

requests to all coordinators, where each of them ran two DB queries.

* Fixed initial population of local AgencyCache values after a server restart.

Previously the local cache was populated from the agency using a commit index

value of 1, whereas it should have been 0 to get the full agency snapshot.

* Better tracking of memory used in AQL graph traversals, COLLECT and SORT

operations. From this version onwards, certain AQL queries can report a

higher memory usage than in previous versions of ArangoDB. This is not

because the queries use more memory than before, but because the memory

usage tracking has been improved.

A side effect of this change is that queries with a memory limit set may

now be aborted whereas in previous versions they ran through successfully

(but actually violated the limit). In this case it may be necessary to

adjust (i.e. raise) query memory limits accordingly.

* Increase background garbage-collection interval for cluster transactions

from 1 second to 2 seconds. This change should reduce the amount of

background task activity a tiny bit (though hardly measurable on an

otherwise idle server).

* Make the audit log honor the configured logging date/time output format

(i.e. `--log.time-format` option). Previously the audit logging always

created a time value in the server's local time, and logged it in

format YYYY-MM-DDTHH:MM:SS.

From 3.8 onwards, the audit logger will honor the date/time format

specified via the `--log.time-format` option, which defaults to

`utc-datestring`. The means the audit logging will by default log all

dates/times in UTC time. To restore the pre-3.8 behavior, please set

the option `--log.time-format` to `local-datestring`, which will make

the audit logger (and all other server log messages) use the server's

local time.

* Fix the activation of the agency supervision maintenance via the REST API

`/_admin/cluster/maintenance`. This API stored a boolean value instead of

an (expected) maintenance period end date/time string.

* Make the cancel operation safe for asynchronoulsly started JavaScript

transactions (via HTTP POST to `/_api/transaction` with the `x-arango-async`

header set).

* Updated OpenSSL to 1.1.1h.

* Added new scheduler metrics:

- `arangodb_scheduler_jobs_done`: Total number of scheduler queue jobs done

- `arangodb_scheduler_jobs_submitted`: Total number of jobs submitted to the

scheduler queue

- `arangodb_scheduler_jobs_dequeued`: Total number of jobs dequeued from the

scheduler queue

- `arangodb_scheduler_num_working_threads`: Number of currently working

scheduler threads

* Added new metric `arangodb_replication_cluster_inventory_requests` to track

the number of requests received for cluster inventories. The cluster

inventory API is called at the beginning of a dump process or by arangosync.

* Added startup option `--server.unavailability-queue-fill-grade`. This option

has a consequence for the `/_admin/server/availability` API only, which is

often called by load-balancers and other availability probing systems.

The `/_admin/server/availability` API will now return HTTP 200 if the fill

grade of the scheduler's queue is below the configured value, or HTTP 503 if

the fill grade is above it. This can be used to flag a server as unavailable

in case it is already highly loaded.

The default value for this option is `0.75`, i.e. 75%. This is a change

compared to previous versions of ArangoDB, where the default value was `1`.

To prevent sending more traffic to an already overloaded server, it can be

sensible to reduce the default value to even `0.5`.

This would mean that instances with a queue longer than 50% of their

maximum queue capacity would return HTTP 503 instead of HTTP 200 when their

availability API is probed.

nb: the default value for the scheduler queue length is 4096.

* Added metrics for the system CPU usage:

- `arangodb_server_statistics_user_percent`: Percentage of time that the

system CPUs have spent in user mode

- `arangodb_server_statistics_system_percent`: Percentage of time that

the system CPUs have spent in kernel mode

- `arangodb_server_statistics_idle_percent`: Percentage of time that the

system CPUs have been idle

- `arangodb_server_statistics_iowait_percent`: Percentage of time that

the system CPUs have been waiting for I/O

These metrics resemble the overall CPU usage metrics in `top`.

They are available on Linux only.

* Fix internal issue #777: Fixed memory access while substituting

stored values for ArangoSearch view optimization

* Make the number of network I/O threads properly configurable via the

startup option `--network.io-threads`. This option existed before, but its

configured value was effectively clamped to a value of `1`.

* Improvements for the Pregel distributed graph processing feature:

- during the loading/startup phase, the in-memory edge cache is now

intentionally bypassed. The reason for this is that any edges are

looked up exactly once, so caching them is not beneficial, but would

only lead to cache pollution.

- the loading/startup phase can now load multiple collections in parallel,

whereas previously it was only loading multiple shards of the same

collection in parallel. This change helps to reduce load times in case

there are many collections with few shards, and on single server.

- the loading and result storage phases code has been overhauled so that

it runs slightly faster.

- for Pregel runs that are based on named graphs (in contrast to explicit

naming of the to-be-used vertex and edge collections), only those edge

collections are considered that, according to the graph definition, can

have connections with the vertex. This change can reduce the loading

time substantially in case the graph contains many edge definitions.

- the number of executed rounds for the underlying Pregel algorithm now

does not vary for different `parallelism` values.

* Fix HTTP/1.1 status response header in fuerte responses

This change makes fuerte return the full status header, including the

numeric status code and the status string in the `http/1.1` header of

fuerte responses.

Previously, the return header lacked the numeric status code, so it

looked like

```

"http/1.1" : "Ok"

```

Now, with the numeric status code, the response header will look like

```

"http/1.1" : "200 Ok"

```

This PR also adds a protocol() method for arango client connections in

order to check the protocol in use. The possible return values are

- "http" for HTTP/1.1 connections

- "http2" for HTTP/2 connections

- "vst" for VST connections

- "unknown" for everyhting else

This is needed during testing, but can also be used for other purposes.

* Updated arangosync to 0.7.12.

* Fix log topic of general shutdown message from "cluster" to general.

* Automatically add "www-authenticate" headers to server HTTP 401 responses,

as required by the HTTP specification.

* Fixed an AQL bug that ignored PRUNE statements in OneShard setups.

* Make the DOCUMENT AQL function eligible for running on DB servers in

OneShard deployment mode. This allows pushing more query parts to DB servers

for execution.

* Enable HTTP request statistics and provide metrics even in case

`--server.statistics-history` is set to `false` (this option will set

itself to off automatically on agency instances on startup if not

explicitly set).

This change provides more metrics on all server instances, without the

need to persist them in the instance's RocksDB storage engine.

* Fixed a deadlock between AQL write transactions and hotbackup, since

in AQL write transactions follower transactions did not know they are

follower transactions.

* Added metrics for collection locks:

- `arangodb_collection_lock_timeouts_exclusive`: Number of lock timeouts

when trying to acquire collection exclusive locks

- `arangodb_collection_lock_timeouts_write`: Number of lock timeouts when

trying to acquire collection write locks

- `arangodb_collection_lock_acquisition_micros`: Total amount of collection

lock acquisition time [μs]

- `arangodb_collection_lock_acquisition_time`: Total collection lock

acquisition time histogram [s]

* Reduce lock timeout on followers to 15 seconds.

Rationale: we should not have any locking conflicts on followers, generally.

Any shard locking should be performed on leaders first, which will then,

eventually replicate changes to followers. replication to followers is only

done once the locks have been acquired on the leader(s).

* Fix a memory leak because cluster internal connections were not cleaned

up for agency communication.

* Added compile option USE_JEMALLOC_PROF to enable memory profiling.

* Remove extra CMake option `DEBUG_SYNC_REPLICATION` and use the already

existing `USE_FAILURE_TESTS` options for its purpose.

* Updated bundled version of Snappy compression/decompression library to 1.1.8.

* Fix REST API endpoint PUT `/_api/collection/<collection>/recalculateCount` on

coordinators. Coordinators sent a wrong message body to DB servers here, so

the request could not be handled properly.

* Fixed issue #12778: fails validation if additionalProperties: false.

* Fixed BTS-233 issue: Fixed invalid IndexId comparator.

* Fixed potential deadlock in cluster transactions if a transaction is

returned that was soft-aborted by transaction garbage collection before.

This deadlock should rarely ever occur in practice, as it can only be

triggered once during the server shutdown sequence.

* Added support of `GEO_DISTANCE`, `GEO_CONTAINS`, `GEO_INTERSECTS`,

`GEO_IN_RANGE` to ArangoSearch.

* Added new `GeoJSON` ArangoSearch analyzer.

* Added new `GeoPoint` ArangoSearch analyzer.

* Added new `GEO_IN_RANGE` AQL function.

* Fixed handling of failedLeaderJob. In case of a plan modification, that

removes a server from the plan, e.g. reduce replication factor. Directly

followed by a failure of the current shard leader, would reinsert

the just removed server in the plan, which is undesired, we first need

to have a full "desync" cycle on this server to be reusable in the plan

again.

* Fixed very spurious errors if the `holdReadLockCollection` replication API for

the getting-in-sync procedure of shards was called during server shutdown.

In this case that method could ask the transaction manager for a specific

transaction, but wasn't returning one due to the server shutdown.

* Added new 'aql' type for ArangoSearch analyzers.

* Obsoleted the startup options `--database.throw-collection-not-loaded-error`

and `--ttl.only-loaded-collection`.

These options were meaningful for the MMFiles storage engine only, but for

the RocksDB storage engine they did not make any difference. Using these startup

options is still possible, but will have no effect other than generating a

warning at server startup.

* Added CMake option `USE_MINIMAL_DEBUGINFO`.

This option is turned off by default. If turned on, the created binaries

will contain only a minimum amount of debug symbols, reducing the size

of the executables. If turned off (which is the default), the binaries

will contain full debug information, which will make them bigger in size

unless the debug information is later stripped again.

* Modified the returned error code for calling the `shards()` function on a

collection in single-server from "internal error" (error number 4) to "shards

API is only available in cluster" and error number 9, HTTP status code 501.

* Bug-fix: Allow to unlink a view created on a SmartGraphEdge collection.

* If a collection (or database) is dropped during the instantiation of an AQL query,

the setup code now aborts with an ERROR_QUERY_COLLECTION_LOCK_FAILED and earlier.

Before the setup code could abort with TRI_ERROR_INTERNAL in the same case.

* Added WINDOW keyword to AQL to allow aggregations on related rows.

* Added new graph method K_PATHS to AQL. This will enumerate all paths between a

source and a target vertex that match the given length.

For example, the query

```

FOR path IN 2..4 OUTBOUND K_PATHS "v/source" TO "v/target" GRAPH "g"

RETURN path

```

will yield all paths in format

{

vertices: [v/source, ... , v/target],

edges: [v/source -> v/1, ..., v/n -> v/target

}

that have length exactly 2 or 3 or 4, start at v/source and end at v/target.

The order of those paths in the result set is not guaranteed.

* Fixed issue BTS-195: AQL update queries using the `keepNull` option set to

false had an inconsistent behavior. For example, given a collection `test`

with an empty document with just key `testDoc`, the following query

would return different results when running for the first time or the second

time:

UPDATE 'testDoc'

WITH {test: {sub1: true, sub2: null}} IN test

OPTIONS { keepNull: false, mergeObjects: true }

For its first run, the query would return

{

"_key": "testDoc",

"test": {

"sub1": true,

"sub2": null

}

}

(with the `null` attribute value not being removed). For all subsequent runs,

the same query would return

{

"_key": "testDoc",

"test": {

"sub1": true,

}

}

(with the `null` value removed as requested).

This inconsistency was due to how the `keepNull` attribute was handled if

the attribute already existed in the to-be-updated document or not. The

behavior is now consistent, so `null` values are now properly removed from

sub-attributes even if in the to-be-updated document the target attribute

did not yet exist. This makes such updates idempotent again.

This a behavior change compared previous versions, but it will only have

effect when `keepNull` is set to `false` (the default value is `true` however),

and only when just-inserted object sub-attributes contained `null` values.

* Optimization of empty append entries.

* Bug-fix: Creating an additional index on the edge collection of a disjoint

SmartGraph could falsely result into an error:

`Could not find all smart collections ...`

This is now ruled out and indexes can be created as expected.

* Added startup option `--foxx.force-update-on-startup` to toggle waiting

for all Foxx services in all databases to be propagated to a coordinator

before it completes the boot sequence.

In case the option is set to `false` (i.e. no waiting), the coordinator

will complete the boot sequence faster, and the Foxx services will be

propagated lazily. Until the initialization procedure has completed for

the local Foxx apps, any request to a Foxx app will be responded to with

an HTTP 503 error and message

waiting for initialization of Foxx services in this database

This can cause an unavailability window for Foxx services on coordinator

startup for the initial requests to Foxx apps until the app propagation

has completed.

When not using Foxx, this option should be set to `false` (default) to

benefit from a faster coordinator startup.

Deployments relying on Foxx apps being available as soon as a coordinator

is integrated or responding should set this option to `true`.

The option only has an effect for cluster setups.

On single servers and in active failover mode, all Foxx apps will be

available from the very beginning.

Note: ArangoDB 3.6 and 3.7 also introduced this option, but with a default

value of `true`. ArangoDB 3.8 changes the default to `false`.

* Changed the server-side implementation of the following internal JavaScript

APIs to no-ops:

* `internal.reloadAqlFunctions()`: this is a no-op function now

* `@arangodb/actions.buildRouting()`: this is a no-op function now

* `@arangodb/actions.routingTree`: will return an empty object

* `@arangodb/actions.routingList`: will return an empty object

All the above APIs were intended to be used for internal means only. These

APIs are deprecated now and will be removed in ArangoDB v3.9.

* Instead of failing to connect to INADDR_ANY refuse it as a parameter, with a

descriptive error message for novice users (issue #12871).

* Remove any special handling for obsoleted collection attributes

`indexBuckets`, `journalSize`, `doCompact` and `isVolatile`. These

attributes were meaningful only with the MMFiles storage engine and have

no meaning with the RocksDB storage engine. Thus any special handling

for these attributes can be removed in the internal code.

Client applications and tests that rely on the behavior that setting

any of these attributes produces an error when using the RocksDB engine

may need adjustment now.

* Added a --continue options to arango restore. arangorestore now keeps track of the progress

and can continue the restore operation when some error occured.

* Added the following metrics for synchronous replication in the cluster:

- `arangodb_refused_followers_count`: Number of times a shard leader received

a refusal answer from a follower during synchronous replication.

- `arangodb_sync_wrong_checksum`: Number of times a mismatching shard

checksum was detected when syncing shards. In case this happens, a resync

will be triggered for the shard.

* Don't respond with misleading error in smart vertex collections.

When inserting a document with a non-conforming key pattern into

a smart vertex collection, the response error code and message are

1466 (ERROR_CLUSTER_MUST_NOT_SPECIFY_KEY) and "must not specify _key

for this collection".

This is misleading, because it is actually allowed to specify a key

value for documents in such collection. However, there are some

restrictions for valid key values (e.g. the key must be a string and

contain the smart graph attribute value at the front, followed by a

colon.

If any of these restrictions are not met, the server currently

responds with "must not specify key for this collection", which is

misleading. This change rectifies it so that the server responds with

error 4003 (ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTE)

and message "in smart vertex collections _key must be a string and

prefixed with the value of the smart graph attribute". This should

make it a lot easier to understand what the actual problem is.

* Fixed handling of failoverCandidates. Sometimes, a server can still be a

failoverCandidate even though it has been taken out of the Plan. With this

fix, such a server is quickly taken out of failoverCandidates and it can never

be re-added to the Plan before this has happened.

* Added configuration option `--query.tracking-slow-queries` to decide whether

slow queries are tracked extra.

* Added configuration option `--query.tracking-with-querystring` to decide

whether the query string is shown in the slow query log and the list of

currently running queries. The option is true by default.

When turned off, querystrings in the slow query log and the list of currently

running queries are just shown as "<hidden>".

* Added configuration option `--query.tracking-with-datasources` to toggle

whether the names of data sources used by queries are shown in the slow query

log and the list of currently running queries. The option is false by default.

When turned on, the names of data sources used by the query will be shown in

the slow query log and the list of currently running queries.

* Fix an issue in arangoimport improperly handling filenames with less than 3

characters. The specified input filename was checked for a potential ".gz"

ending, but the check required the filename to have at least 3 characters.

This is now fixed.

* Added optional verbose logging for agency write operations. This logging

is configurable by using the new log topic "agencystore".

The following log levels can be used for for the "agencystore" log topic

to log writes to the agency:

- DEBUG: will log all writes on the leader

- TRACE: will log all writes on both leaders and followers

The default log level for the "agencystore" log topic is WARN, meaning no

agency writes will be logged.

Turning on this logging can be used for auditing and debugging, but it is

not recommended in the general case, as it can lead to large amounts of

data being logged, which can have a performance impact and will lead to

higher disk space usage.

* Fix #12693: SORT inside a subquery could sometimes swallow part of its input

when it crossed boundaries of internal row batches.

* Added configuration option `--rocksdb.sync-delay-threshold`.

This option can be used to track if any RocksDB WAL sync operation is

delayed by more than the configured value (in milliseconds). The intention

is to get aware of severely delayed WAL sync operations.

* Fix for BTS-191: Made transaction API database-aware.

* Minor clean up of and less verbosity in agent callbacks.

* Speed up initial replication of collections/shards data by not wrapping

each document in a separate `{"type":2300,"data":...}` envelope. In

addition, the follower side of the replication will request data from

leaders in VelocyPack format if the leader is running at least version

3.8.

Stripping the envelopes and using VelocyPack for transfer allows for

smaller data sizes when exchanging the documents and faster processing,

and thus can lead to time savings in document packing and unpacking as

well as reduce the number of required HTTP requests.

* Add database, shard name and error information to several shard-related log

messages.

* Display shard names of a collection in the web interface when in the details

view of the collection.

* Added HTTP requests metrics for tracking the number of superuser and normal

user requests separately:

- `arangodb_http_request_statistics_superuser_requests`: Total number of HTTP

requests executed by superuser/JWT

- `arangodb_http_request_statistics_user_requests`: Total number of HTTP

requests executed by clients

* Added metric `arangodb_agency_callback_registered counter` for tracking the

total number of agency callbacks that were registered.

* Fixed a bug in handling of followers which refuse to replicate operations.

In the case that the follower has simply been dropped in the meantime, we now

avoid an error reported by the shard leader.

* Added weighted traversal. Use `mode: "weighted"` as option to enumerate

paths by increasing weights. The cost of an edge can be read from an

attribute which can be specified using `weightAttribute` option.

* Fix a performance regression when a LIMIT is combined with a COLLECT WITH

COUNT INTO. Reported in ES-692.

* Fixed issue ES-696: SEARCH vs FILTER lookup performance.

Consolidation functionality for ArangoSearch view links was able to hit non-

mergable enormous amount of segments due to improper scheduling logic.

* Data definition reconciliation in cluster has been modified

extensively to greatly accelerate the creation of 1000s of

databases through following means:

- AgencyCache offers change sets API based on Raft index.

- ClusterInfo caches are only updated using change sets.

- Maintenance uses local as well as agency change sets to limit

the scope of every runtime to these change sets.

* Make scheduler react and start new threads slightly faster in case a lot

of new work arrives.

* Make scheduler properly count down the number of working threads in case

an exception happens in a worker thread.

* Added startup option `--database.old-system-collections` to toggle automatic

creation of system collections `_modules` and `_fishbowl`, along with their

internal usage. These collections are useful only in very few cases, so it

is normally not worth to create them in all databases.

The `_modules` collection is only used to register custom JavaScript modules,

for which there exists no API, and `_fishbowl` is used to store the temporary

list of Foxx apps retrieved from the Github Foxx store.

If the option value is `false` (which is the default from v3.8 onwards), the

two collections will not be created for any new database. The `_fishbowl`

collection will still be created dynamically when needed. If the option value

is `true`, the collections will be created regularly as before.

The option will also be introduced to v3.7, where it will have a default

value of `true`, meaning the collections will still be created there.

Any functionality related to the `_modules` system collection is deprecated

and will be removed in ArangoDB v3.9.

Two side effects of turning this option off (which is the default) are:

* there will be no iteration over all databases at server startup just to check

the contents of all `_modules` collections.

* less collections/shards will be around for deployments that create a large

number of databases.

Already existing `_modules` and `_fishbowl` system collections will not be

modified by this change, even though they will likely be empty and unused.

* Don't iterate over all databases at server startup in order to initialize the

routing information. This is not necessary, as the routing information is

global and not tied to a specific database.

Any functionality related to the `_modules` system collection is deprecated

and will be removed in ArangoDB v3.9.

* Use rclone built from v1.51.0 source with go1.15.2 instead of prebuilt

v1.53.0 release.

* Fixed a possible crash during instantiation of an AQL graph traversal.

Reported in #12597.

* Added new ArangoSearch "pipeline" analyzer type

* Reduce the number of dropped followers when running larger (>= 128 MB)

write transactions.

* Fixed a bug in AQL COLLECT with OPTIONS { "hash" } that led to a quadratic

runtime in the number of output rows.

* Make the reboot tracker catch failed coordinators, too. Previously the

reboot tracker was invoked only when a DB server failed or was restarted,

and when a coordinator was restarted. Now it will also act if a coordinator

just fails (without restart).

* Added scheduler thread creation/destruction metrics:

- `arangodb_scheduler_threads_started`: Number of scheduler threads started

- `arangodb_scheduler_threads_stopped`: Number of scheduler threads stopped

* Added replication metrics `arangodb_replication_initial_sync_bytes_received`

for the number of bytes received during replication initial sync operations

and `arangodb_replication_tailing_bytes_received` for the number of bytes

received for replication tailing requests.

Also added `arangodb_replication_failed_connects` to track the number of

connection failures or non-OK response during replication.

* Added metrics `rocksdb_free_inodes` and `rocksdb_total_inodes` to track the

number of free inodes and the total/maximum number of inodes for the file

system the RocksDB database directory is located in. These metrics will

always be 0 on Windows.

* Fixed infinite reload of the login window after logout of an LDAP user.

* Added startup option `--query.max-runtime` to limit the maximum runtime of

all AQL queries to a specified threshold value (in seconds). By default,

the threshold is 0, meaning that the runtime of AQL queries is not limited.

Setting it to any positive value will restrict the runtime of all AQL

queries unless it is overwritten in the per-query "maxRuntime" query option.

Please note that setting this option will affect *all* queries in all

databases, and also queries issues for administration and database-internal

purposes.

If a query exceeds the configured runtime, it will be killed on the next

occasion when the query checks its own status. Killing is best effort,

so it is not guaranteed that a query will no longer than exactly the

configured amount of time.

* Updated rclone to 1.53.0.

* Fixed slightly wrong log level for authentication and also added login event

to the standard log.

* Ensure that the argument to an AQL OPTIONS clause is always an object

which does not contain any dynamic (run-time) values. Previously, this

was only enforced for traversal options and options for data-modification

queries. This change extends the check to all occurrences of OPTIONS.

* Added `details` option to figures command of a collection:

`collection.figures(details)`

Setting `details` to `true` will return extended storage engine-specific

details to the figures. The details are intended for debugging ArangoDB

itself and their format is subject to change. There is not much use in using

the details from a client application.

By default, `details` is set to `false`, so no details are returned and the

behavior is identical to previous versions of ArangoDB.

* Enforce a maximum result register usage limit in AQL queries. In an AQL

query, every user-defined or internal (unnamed) variable will need a

register to store results in.

AQL queries that use more result registers than allowed (currently 1000)

will now abort deterministically during the planning stage with error 32

(`resource limit exceeded`) and the error message

"too many registers (1000) needed for AQL query".

Before this fix, an AQL query that used more than 1000 result registers

crashed the server when assertions were turned on, and the behavior was

undefined when assertions were turned off.

* Implement RebootTracker usage for AQL queries in case of coordinator

restarts or failures. This will clean up the rest of an AQL query

on dbservers more quickly and in particular release locks faster.

* Serialize maintenance actions for each shard. This addresses lost document

problems found in chaos testing.

* Fixed an issue with audit logging misreporting some document requests as

internal instead of logging the proper request information

* Add option `--rocksdb.max-write-buffer-size-to-maintain` with default of 0.

This configures how much memory RocksDB is allowed to use for immutable

flushed memtables/write-buffers. The default of 0 will usually be good

for all purposes and restores the 3.6 memory usage for write-buffers.

* Updated arangosync to 0.7.10.

* Make followers in active failover run a compaction after they process a

truncate operation and the truncate removed more than 4k documents. This

can help to reclaim disk space on the follower earlier than without running

the truncate.