devel

-----

* Add attributes `database` and `user` when tracking current and slow AQL queries.

`database` contains the name of the database the query is/was running in, `user`

contains the name of the user that started the query.

These attributes will be returned in addition when calling the APIs for current

and slow query inspection:

* GET `/_api/query/current` and `require("arangodb/aql/queries").current()`

* GET `/_api/query/slow` and `require("arangodb/aql/queries").slow()`

The "slow query" log message has also been augmented to contain the database

name and the user name.

The `user` attribute is now also displayed in the web interface in the "Running

queries" and "Slow queries" views.

* Introduce an internal high-water mark for the maximum row number that was

written to in an AqlItemBlock. Using this number several operations on the

whole block, such as cleaning up or copying can be made more efficient when

run on only partially filled blocks.

* Updated arangosync to 0.7.10.

* Make UPSERT statement with collection bind parameter behave identical to its

non-bind parameter counterpart.

For example, the query

FOR d IN ["key_1", "key_2", "key_3", "key_1"]

UPSERT d INSERT d UPDATE d IN @@collection

would fail with "unique constraint violation" when used with a collection bind

parameter, but the equivalent query

FOR d IN ["key_1", "key_2", "key_3", "key_1"]

UPSERT d INSERT d UPDATE d IN collectionName

with a hard-coded collection name would succeed. This is now fixed so both

queries have the same behavior (no failure) in single server.

* Fixed internal issue #744: LIMIT with only offset and constrained heap

optimization will use estimation value for ArangoSearch views.

* Fix internal issue #742: Add tick storage in index meta payload after

truncate operation

* Fixed: During a move-shard job which moves the leader there is a

situation in which the old owner of a shard can reclaim ownership

(after having resigned already), with a small race where it allows to

write documents only locally, but then continue the move-shard to a

server without those documents. An additional bug in the MoveShard

Supervision job would then leave the shard in a bad configuration

with a resigned leader permanently in charge.

* Fixed issue #12304: insert in transaction causing com.arangodb.ArangoDBException:

Response: 500, Error: 4 - Builder value not yet sealed.

This happened when too deeply-nested documents (more than 63 levels of nesting)

were inserted. While indefinite nesting is still not supported, the error message

has been corrected from the internal HTTP 500 error "Builder value not yet sealed"

to the correct HTTP 400 "Bad parameter".

* Show optimizer rules with highest execution times in explain output.

* Renamed "master" to "leader" and "slave" to "follower" in replication messages.

This will change the contents of replication log messages as well the string

contents of replication-related error messages.

The messages of the error codes 1402, 1403 and 1404 were also changed accordingly,

as well as the identifiers:

- `TRI_ERROR_REPLICATION_MASTER_ERROR` renamed to `TRI_ERROR_REPLICATION_LEADER_ERROR`

- `TRI_ERROR_REPLICATION_MASTER_INCOMPATIBLE` renamed to `TRI_ERROR_REPLICATION_LEADER_INCOMPATIBLE`

- `TRI_ERROR_REPLICATION_MASTER_CHANGE` renamed to `TRI_ERROR_REPLICATION_LEADER_CHANGE`

This change also renames the API endpoint `/_api/replication/make-slave` to

`/_api/replication/make-follower`. The API is still available under the old

name, but using it is deprecated.

* Fixed that dropping a vanished follower works again. An exception response

to the replication request is now handled properly.

* Fixed bad behaviour in agency supervision in some corner cases involving

already resigned leaders in Current.

* Fixed a problem with potentially lost updates because a failover could

happen at a wrong time or a restarted leader could come back at an

unlucky time.

* Fixed issue BTS-168: Fixed undefined behaviour that did trigger

segfaults on cluster startups. It is only witnessed for

MacOS based builds. The issue could be triggered by any network connection.

This behaviour is not part of any released version.

* Fixed issue ES-664: the optimizer rule `inline-subqueries` must pull out

subqueries that contains a COLLECT statement if the subquery is itself called

from within a loop. Otherwise the COLLECT will be applied to the values in the

outer FOR loop, which can produce a different result.

* Fixed a blockage on hotbackup when writes are happening concurrently, since

followers could no longer replicate leader transactions.

* Updated arangosync to 0.7.9.

* Fixed hotbackup S3 credentials validation and error reporting for upload

and download.

* Make AQL user-defined functions (UDFs) work in a cluster in case the UDF runs

an AQL query inside its own function code (BTS-159).

* Fix: writeConcern is now honored correctly (ES-655).

* Fix: The 'sorted' COLLECT variant would return undefined instead of null when

grouping by a null value.

* Hard-code returned "planVersion" attribute of collections to a value of 1.

Before 3.7, the most recent Plan version from the agency was returned inside

"planVersion".

In 3.7, the attribute contained the Plan version that was in use when the

in-memory LogicalCollection object was last constructed. The object was

always reconstructed in case the underlying Plan data for the collection

changed or when a collection contained links to arangosearch views.

This made the attribute relatively useless for any real-world use cases, and

so we are now hard-coding it to simplify the internal code. Using the attribute

in client applications is also deprecated.

* Slightly improve the performance of cluster DDL maintenance operations.

* Don't prevent concurrent synchronization of different shards from the same

database. Previously only one shard was synchronized at a time per database.

* Fixed OASIS-278 issue: Added proper sort/calc nodes cleanup for late materialzation

after OneShard optimization

* Improve performance of many non-subquery AQL queries, by optimizing away

some storage overhead for subquery context data.

* Improve performance of internal cluster Plan and Current reload operations.

* Fixed issue #12349: arangosh compact Arangoerror 404.

* Wait until restore task queue is idle before shutting down.

* Fix a race problem in the unit tests w.r.t. PlanSyncer.

* Always fetch data for /_api/cluster/agency-dump from leader of the agency.

Add option "redirectToLeader=true" to internal /_api/agency/state API.

* Fixed issue #12297: ArangoDB 3.6.5 Swagger Error?

This issue caused the Swagger UI for displaying the APIs of user-defined Foxx

services to remain invisible in the web UI, because of a JavaScript exception.

This PR fixes the JavaScript exception, so the services API is displayed

again properly.

* Fixed issue #12248: Web UI on 3.6.5: 404 error on adding new index.

This issue caused "404: not found" errors when creating indexes via the web

UI. The indexes were created successfully despite the error message popping

up. This fix removes the misleading unconditional error message.

* Slightly improved the performance of some k-shortest-path queries.

* Added startup option `--rocksdb.encryption-key-rotation` to activate/deactivate

the encryption key rotation REST API. The API is disabled by default.

* Add internal caching for LogicalCollection objects inside ClusterInfo::loadPlan.

This allows avoiding the recreation of LogicalCollection objects that did not

change from one loadPlan run to the next. It reduces CPU usage considerably on

both Coordinators and DB-servers.

* Fixed undefined behavior in AQL COLLECT with multiple group variables (issue

#12267).

If you are grouping on "large" values that occur multiple times in different

groups, and two different groups with the same large value are written to

different batches in the output then the memory could be invalid.

e.g. the following query is affected:

```

FOR d IN documents

FOR batchSizeFiller IN 1..1001

COLLECT largeVal = d.largeValue, t = batchSizeFiller

RETURN 1

```

* Revive faster out-of-range comparator for secondary index scans that do a full

collection index scan for index types "hash", "skiplist", "persistent".

* Fixed internal issue #733: Primary sort compression in views now used properly.

* Errors with error code 1200 (Arango conflict) will now get the HTTP response

code 409 (Conflict) instead of 412 (Precondition failed), unless "if-match" header

was used in `_api/document` or `_api/gharial`.

* Fix spurious lock timeout errors when restoring collections.

* Improve performance of agency cache by not copying the hierarchical Node

tree result for serialization, but serializing it directly.

* Make sure cluster statistics in web UI work in case a coordinator is down.

* Change HTTP response code for error 1450 ("too many shards") from HTTP 500 to

HTTP 400, as this is clearly a client error.

* Turn off maintenance threads on Coordinators, as they are not needed there.

* Fixed crash in cleanup of parallel traversal queries.

* Updated arangosync to 0.7.8.

* Fixed hotbackup upload and download with encryption at rest key indirection.

* Fixed a race between a new request and the keepAlive timeout.

* Added cluster metrics `arangodb_load_plan_accum_runtime_msec` and

`arangodb_load_current_accum_runtime_msec` to track the total time spent

in `loadPlan()` and `loadCurrent()` operations.

* Fixed wrong reporting of failures in all maintenace failure counter metrics

(`arangodb_maintenance_action_failure_counter`). Previously, each successful

maintenance operation was reported as a failure, so the failure counters

actually were counters for the number of successful maintenance actions.

* Adjusted the scale of the `arangodb_maintenance_action_queue_time_msec` to

cover a more useful range.

* The filter executor will now overfetch data again if followed by a limit, same as in 3.6 series.

The following queries are effected:

```

something

FILTER a == b

LIMIT 10

```

something will now be asked for a full batch, instead of only 10 documents.

* In rare cases SmartBFS could use a wrong index for looking up edges. This is fixed now.

* The internally used JS-based ClusterComm send request function can now again use JSON, and does

not require VelocyPack anymore. This fixes an issue with Foxx-App management (install, updated, remove)

got delayed in a sharded environment, all servers do get all apps eventually, now the fast-path

will work again.

* Fixed a rare race in Agents, if the leader is rebooted quickly there is a chance

that it is still assumed to be the leader, but delivers a state shortly in the

past.

* Fixed a race in the ConnectionPool which could lease out a connection

that got its idle timeout after the lease was completed. This could lead

to sporadic network failures in TLS and to inefficiencies with TCP.

* Fixed restoring a SmartGraph into a database that already contains that same graph.

The use case is restoring a SmartGraph from backup, apply some modifications, which are

undesired, and then resetting it to the restored state, without dropping the database.

One way to achieve this is to use arangorestore with the `overwrite` option on the same dataset,

effectively resetting the SmartGraph to the original state.

Without this fix, the workaround for is to either drop the graph (or the database) before the

restore call, yielding an identical result.

* Keep the list of last-acknowledged entires in Agency more consistent.

During leadership take-over it was possible to get into a situation that

the new leader does not successfully report the agency config, which

was eventually fixed by the Agent itself. Now this situation is impossible.

* Allow changing collection properties for smart edge collections as well.

* Fixed that the hotbackup agency lock is released under all circumstances

using scope guards. This addresses a rare case in which the lock was left

behind.

* Privatized load plan / current in cluster info and cleanup following

agency cache implementation.

* Fix cluster-internal request forwarding for VST requests that do not have any

Content-Type header set. Such requests could have been caused by the Java

driver (ES-635).

* Fixed issue OASIS-252: Hotbackup agency locks without clientId.

* The `_from` and `_to` attributes of an edge document can now be edited from

within the UI.

* Added vertex collection validation in case of a SmartGraph edge definition

update.

* Updated arangosync to 0.7.7.

* Added support `db._engineStats()` API in coordinator. Previously calling this

API always produced an empty result. Now it will return the engine statistics

as an object, with an entry for each individual DB-Server.

* Fixed a document parsing bug in the Web UI. This issue occured in the

document list view in case a document had an attribute called `length`.

The result was an incorrect representation of the document preview.

* Improve readability of running and slow queries in web UI by properly left-

aligning the query strings.

* The Web UI is not disabling the query import button after file upload takes

place.

* The Web UI is now reporting errors properly in case of editing ArangoSearch

Views with invalid properties.

* In case of a graph deletion failure, the Web UI displays now the correct

error message.

* In case a document got requested via the UI of a collection which does not

exist, the UI now properly displays an error view instead of having a bad

display state.

* Removed the edge id's hover styling in case of embedded document editor in

the Web UI as this functionality is disabled. This was misleading because

the elements are not clickable.

* The Web UI now displays an error message inside the node information view in

case the user has no access to retreive the neccessary information.

* Web UI: Removed unnecessary menubar entry in case of database node inspection.

* Fixed a potential agency crash if trace logging is on.

* Re-enable access to GET `/_admin/cluster/numberOfServers` for all users by

default. Requests to PUT `/_admin/cluster/numberOfServers` require admin

user privileges. This restores the pre-3.7 behavior.

In contrast to pre-3.7 behavior, the `--server.harden` option now can be

used to restrict access to GET `/_admin/cluster/numberOfServers` to admin

users, too. This can be used to lock down this API for non-admin users

entirely.

* Network layer now reports connection setup issues in more cases

this replaces some INTERNAL_ERROR reports by more precise errors,

those are only reached during failover scenarios.

* Improve readability of running and slow queries in web UI by properly left-

aligning the query strings.

* Allow changing collection properties for smart edge collections as well.

Previously, collection property changes for smart edge collections were not

propagated.

* Adjust arangodump integration test to desired behavior, and make sure

arangodump behaves as specified when invoking it with non-existing

collections.

* Fixed BTS-110: Fulltext index with minLength <= 0 not allowed.

* Disallow using V8 dependent functions in SEARCH statement.

* Remove superflous `%>` output in the UI modal dialog in case the JSON editor

was embedded.

* Fixed a misleading error message in AQL.

* Fix undistribute-remove-after-enum-coll which would allow calculations

to be pushed to a DBServer which are not allowed to run there.

* Fixed issue ES-609: "Transaction already in use" error when running

transaction.

Added option `--transaction.streaming-lock-timeout` to control the timeout in

seconds in case of parallel access to a streaming transaction.

* Returned `AQL_WARNING()` to emit warnings from UDFs.

* Fixed internal issue BTS-107, offset over the main query passed through a

subquery which has modification access to shards could yield incorrect

results, if shards are large enough and skipping was large enough, both to

overflow the batch-size in AQL for each shard individually.

The following query would be affected:

FOR x IN 1..100

LET sub = (

FOR y IN Collection

UPDATE y WITH {updated: true} IN Collection

RETURN new

)

LIMIT 5, 10

REUTRN {x,sub}

If a shard in Collection, has enough entries to fill a batch, the second shard

could run into the issue that it actually does not skip the first 5 main

queries, but reports their results in addition. This has the negative side

effect that merging the subqueries back together was off.

* Correct some log entries.

* Allow removal of existing schemas by saving a schema of either `null` or `{}`

(empty object). Using an empty string as schema will produce an error in the

web interface and will not remove the schema.

The change also adjusts the behavior for the SCHEMA_VALIDATE AQL function in

case the first parameter was no document/object. In this case, the function

will now return null and register a warning in the query, so the user can

handle it.

* Internal issue BTS-71: Added a precondition to prevent creating a collection

with an invalid `distributeShardsLike` property.

* Internal issue BTS-71: In a cluster, for collections in creation, suspend

supervision jobs concerning replication factor until creation is completed.

Previously, it could cause collection creation to fail (e.g. when a server

failed during creation), even when it didn't have to.

* Internal issue BTS-71: Fixed error handling regarding communication with the

agency. This could in a specific case cause collection creation in a cluster

report success when it failed.

* Fixed internal issue #725: Added analyzers revision for _system database in

queries.

* Allow restoring collections from v3.3.0 with their all-numeric collection

GUID values, by creating a new, unambiguous collection GUID for them.

v3.3.0 had a bug because it created all-numeric GUID values, which can be

confused with numeric collection ids in lookups. v3.3.1 already changed the

GUID routine to produce something non-numeric already, but collections

created with v3.3.0 can still have an ambiguous GUID. This fix adjusts

the restore routine to drop such GUID values, so it only changes something

if v3.3.0 collections are dumped, dropped on the server and then restored

with the flawed GUIDs.

* Fixed bug in IResearchViewExecutor that lead to only up to 1000 rows being

produced.

* Changing the current users profile icon in the Web UI now renders the new

icon directly without the need of a full UI browser reload.

* The Web UI's replication view is now checking the replication state

automatically without the need of a manual reload.

* Fixed an error scenario where a call could miss count skip.

It was triggered in the case of Gather in Cluster, if we skipped over a full

shard, and the shard did actually skip, but there are more documents to skip

on another shard.

* Fixed hotbackup agency lock cleanup procedure.

* Only advance shard version after follower is reported in sync in agency.

* Fixed cluster behavior with HotBackup and non-existing backups on DB-Servers.

* Fixed that, when performing a graph AQL query while a (graceful) failover for

the leader of the system collections is in progress, ArangoDB would report a

"Graph not found error".

The real error, namely that an agency transaction failed, was swallowed in

the graph lookup code due to a wrong error code being used from Fuerte.

We now generate a more appropriate 503 - Service Unavailable error.

* added option `--log.use-json-format` to switch log output to JSON format.

Each log message then produces a seperate line with JSON-encoded log data,

which can be consumed by applications.

* added option `--log.process` to toggle the logging of the process id

(pid) in log messages. Logging the process id is useless when running

arangod in Docker containers, as the pid will always be 1. So one may

as well turn it off in these contexts.

* added option `--log.in-memory` to toggle storing log messages in memory,

from which they can be consumed via the `/_admin/log` and by the web UI. By

default, this option is turned on, so log messages are consumable via API

and the web UI. Turning this option off will disable that functionality and

save a tiny bit of memory for the in-memory log buffers.

* Allow for faster cluster shutdown. This should reduce the number of shutdown

hangers in case the agents are stopped already and then coordinators or

DB-Servers are shut down.

* Fixed issue ES-598. Web UI now shows correct permissions in case wildcard

database level and wildcard collection level permissions are both set.

* Fixed non-deterministic test failure in Pregel WCC test.

* Fixed unintentional connection re-use for cluster-internal communications.

* Fixed problem with newer replication protocol and ArangoSearch which could

lead to server crashes during normal operation.

* Fixed bad behavior that led to unnecessary additional revision tree rebuilding

on server restart.

* Allow AQL queries on DB-Servers again. This is not an official supported

feature, but is sometimes used for debugging. Previous changes made it

impossible to run a query on a local shard.

* Fix restoring old arangodumps from ArangoDB 3.3 and before, which had index

information stored in slightly different places in the dump.

* Fix invalid calls to `AgencyCommResult::slice()` method, which must only be

made in case of an agency result was retrieved successfully. In case the call

to the agency was not successful, `slice()` must not be called on it. This

change makes sure this invariant holds true.

* Fix internal test helper function `removeCost` to really remove costs.

* Fix potential AQL query shutdown hanger.

* Use smarter default value preset in web UI for replication factor in case

there are constraints established for the replication factor by the

startup options `--cluster.min-replication-factor` and

`--cluster.max-replication-factor`.

* Modified the exception handling in the RestHandler. Asynchronous communication

could lead to less detailed failure information.

* Added permissions check before trying to read data from `_analyzers`

collection.

If these permissions are not there, no load is performed (user can not use

analyzers from database anyway).

* Updated arangosync to 0.7.6.

* Make the reboot tracker catch a failed server and permanently removed

servers. This allows other servers in the cluster to move on more quickly,

when a server fails and does not immediately come back.

* Added WCC pregel algorithm for weakly connected components.

* Display "geoJson" value for geo indexes in index overview in web UI.

* Allow specifiying arbitrary vertex attributes when creating a vertex via the

graph viewer in the web UI. This is necessary for collections that have a

schema and that require new vertices to have certain attributes/values.

* Pass through AQL query errors to UI graph viewer without turning them into

"Internal Server Error"s.

* Augment nodes view in web UI with node id and full datetime of startup.

* Fixed "Create Debug Package" in web UI in cluster mode.

* Pick up selected "write concern" value entered in web UI when creating a

collection in the cluster. Previously, the "write concern" value was not

picked up correctly and collections were created with the default write

concern.

* Fixed the creation of edges in the web UI, which did not work due to a

JavaScript error.

* Fixed velocypack validator for proper check of keys.

* Handle missing "shardingStrategy" in Plan agency entries for collections.

"shardingStrategy" entries can be missing for collections created with

ArangoDB clusters in version 3.3 or earlier. A missing "shardingStrategy"

entry can cause issues for certain queries that need to check the shard

distribution on the DB-Servers.

This change transparently handles missing "shardingStrategy" entries.

* Taken collection dropping from fast track in maintenance. This avoids

blocking fast track maintenance threads when a shard cannot immediately

be dropped because of some pending lock.

* Updated ArangoDB Starter to 0.14.15.

* Fixed a spuriously occuring shutdown deadlock between AgencyCallbackRegistry

and AgencyCache.

* Fixed `"Plan" is not an object in agency` error messages when restarting

DB-Servers that contained ArangoSearch Views with links.

* Fixed misordering of skipped number of rows report. Only triggered if you do

a modification on a subquery nesting level 2 or more, e.g.:

```

LET s1 = (

LET s2 = (

INSERT {} INTO collection

)

COLLECT WITH COUNT INTO c

RETURN c

)

```

Here the c would be off, or a count on main query would be off.

* Fix crash in execution of non-spliced subqueries if remainder of subquery is

skipped.

* Added missing mutex to ConnectionPool::cancelConnections().

* Foxx API now respects "idiomatic" flag being explicitly set to false.

* Fixed crash in execution of non-spliced subqueries if remainder of subquery is

skipped.

* Made modification subqueries non-passthrough. The passthrough logic only works

if exactly as many output rows are produced as input rows are injected.

If the subquery with modification is skipped however this API is violated, we

perform the subquery with the modification, but then discard the input if

skipped).

* Optimized Web UI styling for ArangoSearch Views when in read-only mode.

* Fixed a bug in the internal `recalculateCount()` method for adjusting

collection counts. The bug could have led to count adjustments being

applied with the wrong sign.

* Removed external file loading in a used 3rd party library (Web UI).

* More substantial metrics in agency.

* arangodump and arangorestore will now fail when using the `--collection`

option and none of the specified collections actually exist in the database

(on dump) or in the dump to restore (on restore). In case some of the specified

collections exist, arangodump/restore will issue warnings about the invalid

collections, but will continue to work for the valid collections.

* Improved network send request for more robustness.

* Added multiple RocksDB configuration options to arangod:

* `--rocksdb.cache-index-and-filter-blocks` to make the RocksDB block cache quota

also include RocksDB memtable sizes

* `--rocksdb.cache-index-and-filter-blocks-with-high-priority` to use cache index

and filter blocks with high priority making index and filter blocks be less

likely to be evicted than data blocks

* `--rocksdb.pin-l0-filter-and-index-blocks-in-cache` make filter and index blocks

be pinned and only evicted from cache when the table reader is freed

* `--rocksdb.pin-top-level-index-and-filter` make the top-level index of partitioned

filter and index blocks pinned and only be evicted from cache when the table reader

is freed

* Don't move potentially expensive AQL function calls into loops in the

`remove-unnecessary-calculations-rule`.

For example, in the query

LET x = NOOPT(1..100)

LET values = SORTED(x)

FOR j IN 1..100

FILTER j IN values

RETURN j

there is only one use of the `values` variable. So the optimizer can remove

that variable and replace the filter condition with `FILTER j IN SORTED(x)`.

However, that would move the potentially expensive function call into the

inner loop, which could be a pessimization.

Now the optimizer will not move the calculation of values into the loop when

it merges calculations in the `remove-unnecessary-calculations` optimizer rule.

* Fixed modification executors inside of a subquery, where the subquery decided to

fetch all rows from upstream first and the amount of rows is higher then the

batch size.

* Fixed reporting of skipped number of documents if we have a LIMIT x, 0

right after the modification.

* Added exceptions catching in agency callbacks.

* Fixed issue #11104 case with complex condition on second branch of OR

operator.

* Fixed bad behavior when dropping followers. A follower should be dropped

immediately when it is officially FAILED, not only after a longish timeout.

* Fixed a bug in CollectionNameResolver which could lead to an extended

busy spin on a core when a collection was dropped, but documents of it

still remained in the WAL.

* Fixed return value of fuerte::H1Connection in case of timeout.

* Usability improvements for arangodump and arangorestore:

- when a dump is restricted to one or multiple collections using the

`--collection` option of arangodump, warnings are issued for all specified

collections that are not present in the database. Previously there were

no warnings, so missing a collection by misspelling its name could easily

go unnoticed.

- when a restore is restricted to one or multiple collections using the

`--collection` option of arangorestore, warnings are issued for all specified

collections that are not present in the dump. Previously there were

no warnings, so missing a collection by misspelling its name could easily

go unnoticed.

- when a dump was taken using the `--overwrite` option, there was no check

that validated whether the encryption mode used in the existing dump

directory was the same as the requested encryption mode. This could have

led to dump directories with both encrypted and unencrypted files. This

was only the case when using `--overwrite true`, which is not the default.

- when a restore was performed using the `--encryption.keyfile` option,

there was no check whether the to-be-restored files were actually encrypted.

Now this check is enforced and arangorestore will bail out with an error

if the requested encryption mode for restore is not the same as for the

stored dump files.

* Fixed traversal issue: If you have a traversal with different minDepth and maxDepth

values and filter on path elements that are larger then minDepth, in a way that a

shorter path would match the condition, the shorter paths would in some cases

not be returned, even if they are valid. e.g.

FOR v, e, p IN 1..3 OUTBOUND @start Graph "myGraph"

FILTER p.vertices[2].label != "foo"

RETURN p

In the above query, a path of length 1 would be valid. There p.vertices[2]

does not exist => Evaluated to `null`. `null`.label is again evaluated to `null`

=> `null != "foo"` is true, so the path is valid.

* Fixed traversal issue: If you have a filter on the path that is based on a

variable value, which could not be deduced as constant during runtime, in

a sharded GeneralGraph the filter was not applied correctly.

SmartGraphs and SingleServer traversals are not effected by this issue.

Also OneShard traversals are not effected.

* Added `securityScheme`, `securityScope` and `security` methods to Foxx

endpoints to allow defining Swagger security schemes.

* Foxx routes now always have a Swagger `operationId`. If the route is unnamed,

a distinct operationId will be generated based on the HTTP method and URL.

* Fixed, if you have a collection access within a Subquery, where the main query is fully skipped

and the "splice-subqueries" rule is active. The information of the skip was not transported

correctly. This could cause incorrect counting reports.

If splice-subqueries are disabled, or the main-query is only partly skipped, everything worked

as expected.

* Expanded -std=c++17 flag to all compilers.

* Fixed issue ES-600: log out of LDAP automatically.

Option `--ldap.allow-offline` was added with `false` value by default to let

server use cached offline credentials in case of LDAP server communication was

lost and ArangoDB can't prolong cache online.

* Include all requests to the web interface at endpoint `/_admin/aardvark` in

a node's requests statistics. Previously, requests going to `/_admin/aardvark`

were excluded from the request statistics if and only if the requested

database was the `_system` database.

* Fixed an issue with truncate of a collection after a dbserver was restarted

very quickly. This could block arangosync from making progress because the

_jobs collection could no longer be truncated.

* Web UI is now using pre-compiled templates.

* Fixed issue #11652: Error: AQL: missing variable #2 (1) for node #6

(FilterNode) while planning registers.

* Fixed internal issue #650: Added analyzer cache update before ArangoSearch

view creation/update.

* Update OpenLDAP to 2.4.50.

* Fixed issue ES-554: failed to build filter while querying ArangoSearch view.

This change provides global fix for ArangoSearch analyzers distribution and

management within a cluster.

* Fixed issue ES-545: failing modification operations led to random crashes with

"Index out of bound" error message.

* Fix edge cases in RocksDB primary index range lookups for operators >= and <.

* Fixed issue #11525: Address security vulnerability by updating Swagger-UI

dependency (upgraded Swagger UI to 3.25.1).

* Updated arangosync to 0.7.5.

* Fixed ability to edit graph edge in Graph Viewer of web UI.

* Fixed issue #10371: For k-shortest-paths queries on certain graphs a condition

to abort search was too eager and hence missed the shortest path between two

vertices and returned a longer one.

* Added feature: Disjoint SmartGraphs

SmartGraphs have been extended to a new subtype, called **Disjoint SmartGraphs**.

A Disjoint SmartGraph prohibits edges between different SmartGraph components.

In case the graph schema can be represented without the need of connected

SmartGraph components, a Disjoint SmartGraph should be used as this knowledge

can be used by the internal optimizer to gain even better query traversal

execution times.

* Fixed issue ES-605: arangodump broken when using encryption.

* Fixed a lockup in dropCollection due to a mutex being held for too long.

* Add an optimizer rule that enables execution of certain subqueries on a

DB Server. For this optimization to work, the subquery must contain exactly

one DISTRIBUTE/GATHER pair and only access at most one collection.

This proves particularly useful for traversals, shortest path, and k-shortest

paths queries on Disjoint SmartGraphs where the entire traversal is executed

on the DB Server without involvement of a coordinator.

* ClusterInfo does its own updating of plan / current caches.

* Properly forward requests based on the transaction header.

* Fix a bug in Agency poll, which would sometimes return a wrong firstIndex.

* Made several improvements to AQL's internal register planning. This will

improve query performance in several cases; mainly of large queries with many

variables, and of spliced subqueries. This also fixes the github issues #10853

and #11358 which reported performance degradation caused by subquery splicing.

* Fixed issue #11590: Querying for document by _key returning only a single

seemingly random property on entity ("old", in this case).

This fixes single-key document lookups in the cluster for simple by-key

AQL queries, such as `FOR doc IN collection FILTER doc._key == @key RETURN

doc` in case the document has either an "old" or a "new" attribute.

* Restored behavior of Foxx API documentation being expanded to show all routes

rather than collapsing all sections by default.

* Add optimization for subqueries for which only the number of results

matters. The optimization will be triggered for read-only subqueries that

use a full collection scan or an index scan, without any additional filtering

(early pruning or document post-filtering) and without LIMIT.

It will help in the following situation:

FOR doc IN ...

LET count = COUNT(

FOR sub IN subCollection

FILTER sub._from == doc._id

RETURN sub

)

...

The restrictions are that the subquery result must only be used with the

COUNT/LENGTH function and not for anything else. The subquery itself must

be read-only (no data-modification subquery), not use nested FOR loops nor

LIMIT, nor a FILTER condition or calculation that requires accessing the

document data. Accessing index data is supported for filtering, but not

for further calculations.

If the optimization is triggered, it will show up in the query execution

plan under the name `optimize-count`.

* Integrated libiresearch log topic properly into ArangoDB logging system.

* Fix a bad deadlock because transaction cleanup was pushed with too low

priority.

* Allow specifying graph names as unquoted string in an AQL graph traversal

query, e.g. `FOR ... IN ... GRAPH abc`. Previously, the graph name had

to be a bind parameter or a string enclosed in quotes.

* loadPlan and loadCurrent have been fixed to not miss out on increments.

* ClusterInfo and all data definition changes use the local agency cache.

* Coordinators and DB servers cache the agency's read key value store.

* Agency offers the new poll API to subscribe to the Raft log stream.

* Added option `--rocksdb.edge-cache` to toggle in-memory caching for

edges. The option is turned on by default. This normally helps with

performance in read-only and read-mostly workloads.

* Fixed a bug in Maintenance which could prevent collection creation from

working (made CreateCollection action idempotent).

* Fix potential undefined behavior in some operations issued to the REST

handler at `/_api/collection` in cluster mode.

* `--cluster.agency-prefix` marked as obsolete. Did never work and is not supported.

* Removed old AgencyComm.

* Improve continuation behavior of AQL queries. We post the continuation handler

on lane CLUSTER_AQL instead of CLIENT_AQL. This leads to MEDIUM prio instead

of LOW.

* Fixed a sleeping barber in fuerte. Added TLA+ model to prove that there

is not another one hiding somewhere.

* Fix spurious bugs in `resilience_move` tests due to replication context of

to-be-dropped collections lingering around until timeout.

* Fixed issue 11243: `db._isSystem()` returns true for a non-system DB.

* Fix REST API `/_admin/cluster/removeServer`.

This also fixes the removal of servers via the web interface.

* Updated OpenSSL to 1.1.1g.

* Lower the default values for `--rocksdb.block-cache-size` and

`--rocksdb.total-write-buffer-size` for systems with less than 1 GB of RAM.

For these systems the default value were presumably too high and needed manual

adjustment.

* Allow to override the detected number of available CPU cores via an environment

variable ARANGODB_OVERRIDE_DETECTED_NUMBER_OF_CORES.

* Fix a bug in the optimizer which handled combinations of SmartJoins with

query components that had to run on a coordinator wrong.

* Remove the scatter-arangosearch-view-in-cluster optimizer rule; it was

integrated into scatter-in-cluster.

* Issue #11447: expose extra RocksDB options `--rocksdb.target-file-size-base`

and `--rocksdb.target-file-size-multiplier` to make them configurable.

* Allow STARTS_WITH function to process several prefixes with min match count

parameter.

* Added collection property changes into auditing.

* Fix creation of example graphs when `--cluster.min-replication-factor` is set

and no replication factor was specified for a graph. In this case, the new

graph will use the configured minimum replication factor.

* Show some index details in index overview of the web interface, e.g. the

`minLength` attribute for fulltext indexes or the `expireAfter` attribute for

TTL indexes.

* Added startup option `--server.validate-utf8-strings`.

This option controls whether strings in incoming JSON or VPack requests are

validated for UTF-8 conformity. If set to `true` (which is the default value),

all strings are validated, and requests containing invalid UTF-8 data will be

rejected. Setting the option to `false` will turn off the UTF-8 string

validation for incoming requests. This mode should only be used for

deployments that are known to already contain invalid UTF-8 data and to keep

them operational until the wrong string encoding is fixed in the data.

* Fixed a bug which occurred if a DB-Server was shut down exactly

when it was supposed to resign from its leadership for a shard.

* Improve continuation behavior of AQL queries. We post the continuation

handler on lane CLUSTER_AQL instead of CLIENT_AQL. This leads to MEDIUM

prio instead of LOW.

* When relaying requests to other coordinators in a load-balanced setup, don't

forward the "http/1.1" HTTP header from the remote response. Forwarding that

header in the same way as any other header is not required, as each response

will store and set its status separately anyway. Adding it to the headers

again will only cause confusion and some client applications to choke.

* Fixed issue #11104: `ANALYZER` function doesn't correctly process `!= OR`

queries.

* Obsoleted startup option `--database.maximal-journal-size`. This option was

useful for the MMFiles storage engine only, but did not have an effect with

the RocksDB engine. Using this startup option is not an error, but has

no effect anymore.

* Added `JACCARD` AQL function.

* storedValues property is removed from ArangoSearch link properties output.

* Added primarySortCompression property to ArangoSearch Views.

* Added compression property to ArangoSearch View storedValues.

* Added overwrite mode "ignore" for document inserts. This mode allows

ignoring primary key conflicts on insert when the target document already

exists.

The following overwrite modes now exist:

* "ignore": if a document with the specified `_key` value exists already,

nothing will be done and no write operation will be carried out. The

insert operation will return success in this case. This mode does not

support returning the old document version using the `returnOld`

attribute. `returnNew` will only set the `new` attribute in the response

if a new document was inserted.

* "replace": if a document with the specified `_key` value exists already,

it will be overwritten with the specified document value. This mode will

also be used when no overwrite mode is specified but the `overwrite`

flag is set to `true`.

* "update": if a document with the specified `_key` value exists already,

it will be patched (partially updated) with the specified document value.

* "conflict": if a document with the specified `_key` value exists already,

return a unique constraint violation error so that the insert operation

fails. This is also the default behavior in case the overwrite mode is

not set, and the *overwrite* flag is *false* or not set either.

The overwrite mode "ignore" can also be used from AQL INSERT operations

by specifying it in the INSERT's `OPTIONS`, e.g.

INSERT { _key: ..., .... } INTO collection OPTIONS { overwriteMode: "ignore" }

Again, when the overwrite mode "ignore" is used from AQL, it does not

support returning the old document version. Using "RETURN OLD" in

an INSERT operation that uses the "ignore" overwrite mode will trigger

a parse error, as there will be no old version to return. "RETURN NEW"

will only return the document in case it was inserted. In case the

document already existed, "RETURN NEW" will return "null".

The main use case of inserting documents with overwrite mode "ignore" is

to make sure that certain documents exist in the cheapest possible way.

In case the target document already exists, the "ignore" mode is most

efficient, as it will not retrieve the existing document from storage and

not write any updates to it.

* Added feature: SatelliteGraphs

SatelliteGraphs are a new type of graph, added in addition to the existing

ones, General Graphs and SmartGraphs.

When doing joins involving graph traversals, shortest path or k-shortest paths

computation in an ArangoDB cluster, data has to be exchanged between different

servers. In particular graph traversals are usually executed on a Coordinator,

because they need global information. This results in a lot of network traffic

and potentially slow query execution.

SatelliteGraphs are the natural extension of the concept of

SatelliteCollections to graphs. All of the usual benefits and caveats apply.

SatelliteGraphs are synchronously replicated to all DB-Servers that are part of

a cluster, which enables DB-Servers to execute graph traversals locally. This

includes (k-)shortest path(s) computation and possibly joins with traversals

and greatly improves performance for such queries.

* Removed deprecated MMFiles storage engine and also the `arango-dfdb`

(datafile debugger) executable that could be used to validate MMFiles