devel

-----

* Speed up collection creation process in cluster, if not all agency callbacks are

delivered successfully.

* increased performance of document inserts, by reducing the number of checks in unique / primary indexes

* fixed a callback function in the web UI where the variable `this` was out of scope.

* Fixed editing a user within the web UI if the user added a gravatar profile picture

* allow pregel to select the shard key via `shardKeyAttribute` in pregel start parameters

* Added --server.jwt-secret-keyfile to arangosh.

* fixed internal issue #4040: gharial api is now checking existence of `_from` and `_to` vertices

during edge replacements and edge updates

* fixed `Buffer.alloc` method

* `Buffer` is now iterable and accepts `ArrayBuffer` values as input

* fix timeout-response in case of simultaneous index create/drop in cluster

* allow pregel to select the shard key via `shardKeyAttribute` in pregel start parameters

* Added --server.jwt-secret-keyfile to arangosh.

* enabled dropping an index by its name

* fixed lookup of index from collection by fully qualified name, e.g.

`db.testCollection.index('testCollection/primary')`.

* fixed agency bug with TTL object writes discovered in 3.4.6

v3.5.0-rc.3 (2019-05-31)

------------------------

* fix issue #9106: Sparse Skiplist Index on multiple fields not used for FILTER + SORT query

Allow AQL query optimizer to use sparse indexes in more cases, specifically when

indexes could not be used for filtering and there finally was an `EnumerateCollectionNode`

in the query execution plan followed by a `SortNode`. In this case, sparse indexes were

not considered for enumeration in sorted order, because it was unclear to the optimizer

if the result set would contain null values or not.

* upgrade RocksDB to version 6.2

* updated ArangoDB Starter to 0.14.4

* The system collection '_jobs' will from now on be created with non-unique, non-sparse indexes.

* Bugfix for smart graph traversals with uniqueVertices: path, which could

sometimes lead to erroneous traversal results

* Pregel algorithms can be run with the option "useMemoryMaps: true" to be

able to run algorithms on data that is bigger than the available RAM.

* fix a race in TTL thread deactivation/shutdown

* fixed internal issue #3919: The web UI is now using precompiled ejs templates.

* fixed agency issue in abort of cleanOutServer job

v3.5.0-rc.2 (2019-05-23)

------------------------

* fixed "collection not found" exception during setup of 3-way smart join queries in the

cluster

* fixed an edge case of handling `null` values in the AQL function `MIN` for input

sequences that started with a `null` value. In this case, `null` was always returned as the

minimum value even though other non-null values may have followed, and `MIN` was supposed

to return `null` only if there are no input values or all input values are `null`.

* fixed a crash when posting an async request to the server using the "x-arango-async"

request header and the server's request queue was full

* added error code 1240 "incomplete read" for RocksDB-based reads which cannot retrieve

documents due to the RocksDB block cache being size-restricted (with size limit enforced)

and uncompressed data blocks not fitting into the block cache

The error can only occur for collection or index scans with the RocksDB storage engine

when the RocksDB block cache is used and set to a very small size, plus its maximum size is

enforced by setting the `--rocksdb.enforce-block-cache-size-limit` option to `true`.

Previously these incomplete reads could have been ignored silently, making collection or

index scans return less documents than there were actually present.

* fixed internal issue #3918: added optional second parameter "withId" to AQL

function PREGEL_RESULT

this parameter defaults to `false`. When set to `true` the results of the Pregel

computation run will also contain the `_id` attribute for each vertex and not

just `_key`. This allows distinguishing vertices from different vertex collections.

* make Swagger UI work with HTTPS-enabled ArangoDBs too

* improve scheduler performance for single-connection cases

* internally switch unit tests framework from catch to gtest

* disable selection of index types "hash" and "skiplist" in the web interface when

using the RocksDB engine. The index types "hash", "skiplist" and "persistent" are

just aliases of each other with the RocksDB engine, so there is no need to offer all

of them. After initially only offering "hash" indexes, we decided to only offer

indexes of type "persistent", as it is technically the most

appropriate description.

* fix client id lookup table in state

v3.5.0-rc.1 (2019-05-14)

------------------------

* removed bug during start up with a single agent, that leads to dbserver crash.

* fixed issue #7011: description when replacing a foxx application was misleading

* fixed issue #8841: Graph Viewer dropped ability to edit an edge after

rerendering.

* upgraded arangodb starter version to 0.14.3

* ArangoQueryStreamCursor.prototype.id needs to be a string, v8 32 bit integers

can't hold the full data.

* upgraded Swagger UI to 3.22.1

* add --compress-output flag to arangodump. Activates gzip compression for

collection data. Metadata files, such as .structure.json and .view.json,

do not get compressed. No option is needed for arangorestore to restore

.data.json.gz files.

* added options to make server more secure:

- `--server.harden`: denies access to certain REST APIs that return server internals

- `--foxx.api`: set to false disables Foxx management API

- `--foxx.store`: set to false disables Foxx UI

- `--javascript.allow-port-testing`: enables internal.testPort()

- `--javascript.allow-external-process-control`: enables external process control

- `--javascript.harden`: disables getPid(), processStatistics() and logLevel()

- `--javascript.startup-options-whitelist`: control startup options visible in JavaScript

- `--javascript.environment-variables-whitelist`: control environment variables visible in JavaScript

- `--javascript.endpoints-whitelist`: control accessible endpoints in JavaScript

- `--javascript.files-whitelist`: control file access in JavaScript

Note: There is a [detailed description of all options](https://www.arangodb.com/docs/devel/security-security-options.html).

* prevent arangod from making a call to www.arangodb.com at startup

This call was done to check for available updates, but it could have contributed

to small startup delays in case outgoing connections were blocked.

* removed support for undocumented HTTP header `x-arango-v8-context`, which

allowed controlling in which particular V8 context number a JavaScript-based

action was executed. This header was only used internally for testing.

* db._query now handles additional arguments correctly when passing an AQL

query object instead of a query string and separate bindVars

* added req.auth property to Foxx

* added collection.documentId method to derive document id from key

* Indexes created with the 'inBackground', will not hold an

exclusive collection lock for the entire index creation period (rocksdb only)

* fixed internal issue #536: ArangoSearch may crash server during term lookup

* fixed internal issue #2946: Create graph autocomplete was not working under

certain circumstances.

* added `filter` option to Foxx HTTP API for running tests.

* added function `db.<collection>.getResponsibleShard()` to find out which is the

responsible shard for a given document. Note that this function is only available

in a cluster coordinator.

* updated bundled version of jemalloc memory allocator to 5.2.0.

* don't create per-database system collection `_frontend` automatically.

This collection is only needed by the web UI, and it can be created lazily

when needed.

* added logging option `--log.time-format` to configure the time format used

in log output. The possible values for this option are:

- timestamp: unix timestamps, in seconds

- timestamp-millis: unix timestamps, in seconds, with millisecond precision

- timestamp-micros: unix timestamps, in seconds, with microsecond precision

- uptime: seconds since server start

- uptime-millis: seconds since server start, with millisecond precision

- uptime-micros: seconds since server start, with microsecond precision

- utc-datestring: UTC-based date and time in format YYYY-MM-DDTHH:MM:SSZ

- utc-datestring-millis: UTC-based date and time in format YYYY-MM-DDTHH:MM:SS.FFFZ

- local-datestring: local date and time in format YYYY-MM-DDTHH:MM:SS

This change deprecates the existing options `--log.use-microtime` and

`--log.use-localtime`, because the functionality provided by these options

is covered by `--log.time-format` too.

* added "smart joins" to the ArangoDB Enterprise Edition that allows running cluster

joins between two certain sharded collections with performance close to that of a

local join operation.

* fixed internal issue #3815: fixed the removal of connected edges when

removing a vertex graph node in a smart graph environment.

* show startup warning in case kernel setting `vm.overcommit_memory` is set

to a value of 2 and the jemalloc memory allocator is in use. This combination

does not play well together.

* added AQL function DECODE_REV for decomposing `_rev` values into their

individual parts

* added AQL functions CRC32 and FNV64 for hashing data

* renamed attribute key `openssl-version` in server/client tool version

details output to `openssl-version-compile-time`.

This change affects the output produced when starting one of the ArangoDB

executables with the `--version` command. It also changes the attribute

name in the detailed response of the `/_api/version` REST API.

* internal issue #2276: fixed the sorting of the databases in the database

selection dropdown in the web ui. The sort order differed based on whether

authentication was enabled or disabled.

* fixed internal issuel #3546: improved the shards view in the web ui if there

is only one shard to display.

* fixed internal issue #3789: restricted the allowed query names for user

defined custom queries within the web ui.

* upgraded bundled RocksDB version to 6.0

* added "--log.ids" option to arangod

The default value for this option is `true`. Setting the option to `false`

will prevent embedding unique log ids into all log messages generated by

ArangoDB C++ code. The unique ids allow for easy access to the location in

the source code from which a message originates. This should help customers

to configure custom monitoring/alerting based on specific log id occurrences

and will also be helpful for support and development in identifying problems.

* fixed issue #8294: wrong equals behavior on arrays with ArangoSearch

* fixed internal issue #528: ArangoSearch range query sometimes doesn't work

correctly with numeric values

* changed type of the startup option `--rocksdb.recycle-log-file-num` from

numeric to boolean, as this is also the type the options has in the RocksDB

library.

* renamed hidden startup option `--rocksdb.delayed_write_rate` to the more

consistent `--rocksdb.delayed-write-rate`. When the old option name is

used, the arangod startup will be aborted with a descriptive error message.

* if not explicitly configured, make agency nodes start removing their unused

WAL files a few seconds after the completed server startup already. This is

because on agency nodes, unused WAL files do not need to be retained for

potential replication clients to read them.

* added option `--all-databases` to arangodump and arangorestore

When set to true, this makes arangodump dump all available databases

the current user has access to. The option `--all-databases` cannot be

used in combination with the option `--server.database`.

When `--all-databases` is used, arangodump will create a subdirectory

with the data of each dumped database. Databases will be dumped one

after the other. However, inside each database, the collections of the

database can be dumped in parallel using multiple threads.

For arangorestore, this makes it restore all databases from inside the

subdirectories of the specified dump directory. Using the option for

arangorestore only makes sense for dumps created with arangodump and

the `--all-databases` option. As for arangodump, arangorestore cannot

be invoked with the options `--all-databases` and `--server.database`

at the same time. Additionally, the option `--force-same-database` cannot

be used together with `--all-databases`.

If the to-be-restored databases do not exist on the target server, then

restoring data into them will fail unless the option `--create-database`

is also specified. Please note that in this case a database user must

be used that has access to the `_system` database, in order to create

the databases on restore.

* added index hints feature to AQL

* added "name" property for indices

If a name is not specified on index creation, one will be auto-generated.

* Under normal circumstances there should be no need to connect to a

database server in a cluster with one of the client tools, and it is

likely that any user operations carried out there with one of the client

tools may cause trouble.

The client tools arangosh, arangodump and arangorestore will now emit

a warning when connecting with them to a database server node in a cluster.

* fix compation behavior of followers

* added "random" masking to mask any data type, added wildcard masking

* added option `--console.history` to arangosh for controlling whether

the command-line history should be loaded from and persisted in a file.

The default value for this option is `true`. Setting it to `false`

will make arangosh not load any command-line history from the history

file, and not store the current session's history when the shell is

exited. The command-line history will then only be available in the

current shell session.

* display the server role when connecting arangosh against a server (e.g.

SINGLE, COORDINATOR)

* fixed overflow in Windows NowNanos in RocksDB

* Allow MoveShard from leader to a follower, thus swapping the two

* Supervision fix: Satellite collections, various fixes

* Add coordinator route for agency dump

* Supervision fix: abort MoveShard job does not leave a lock behind,

* Supervision fix: abort MoveShard (leader) job moves forwards when point

of no return has been reached,

* Supervision fix: abort CleanOutServer job does not leave server in

ToBeCleanedServers,

* Supervision fix: move shard with data stopped to early due to wrong usage

of compare function

* Supervision fix: AddFollower only counts good followers, fixing a

situation after a FailedLeader job could not find a new working

follower

* Supervision fix: FailedLeader now also considers temporarily BAD

servers as replacement followers and does not block servers which

currently receive a new shard

* Supervision fix: Servers in ToBeCleanedServers are no longer considered

as replacement servers

* Maintenance fix: added precondition of unchanged Plan in phase2

* add "PRUNE <condition>" to AQL Traversals. This allows to early abort searching of

unnecessary branches within a traversal.

PRUNE is only allowed in the Traversal statement and only between the graph

definition and the options of the traversal.

e.g.:

FOR v, e, p IN 1..3 OUTBOUND @source GRAPH "myGraph"

PRUNE v.value == "bar"

OPTIONS {} /* These options remain optional */

RETURN v

for more details refer to the documentation chapter.

* fixed a display issue when editing a graph within the web UI

* fixed some escaping issues within the web UI

* follow up to fix JWT authentication in arangosh (#7530):

also fix reconnect

* now also syncing _jobs and _queues collections in active failover mode

* upgraded lodash to 4.17.11 because of CVE-2018-16487

* `--query.registry-ttl` is now honored in single-server mode, and cursor TTLs

are now honored on DBServers in cluster mode

* add "TTL" index type, for optional auto-expiration of documents

* disable selection of index types "persistent" and "skiplist" in the web

interface when using the RocksDB engine. The index types "hash", "skiplist"

and "persistent" are just aliases of each other with the RocksDB engine,

so there is no need to offer all of them.

* fixed JS AQL query objects with empty query strings not being recognized

as AQL queries

* update V8 to 7.1.302.28

New V8 behavior introduced herein:

- ES2016 changed the default timezone of date strings to be conditional on

whether a time part is included. The semantics were a compromise approach

based on web compatibility feedback from V8, but until now, we have been

shipping ES5.1 default timezone semantics. This patch implements the

new semantics, following ChakraCore and SpiderMonkey (though JSC

implements V8's previous semantics).

* fixed JS AQL query objects with empty query strings not being recognized as AQL queries

* report run-time openssl version (for dynamically linked executables)

* added greeting warning about maintainer mode

* improve insertion time into non-unique secondary indexes with the RocksDB

engine

* fixed possible segfault when using COLLECT with a LIMIT and an offset

* fixed COLLECT forgetting top-level variables after 1000 rows

* added sort-limit optimization in AQL; improves memory usage and execution

time for some queries

* upgraded to OpenSSL 1.1.0j

* added configurable masking of dumped data via `arangodump` tool to obfuscate

exported sensible data

* fixed arangoimp script for MacOSX CLI Bundle

* added "peakMemoryUsage" in query results figures, showing the peak memory

usage of the executed query. In a cluster, the value the peak memory usage

of all shards, but it is not summed up across shards.

* fixed an issue where a crashed coordinator can lead to some Foxx queue jobs

erroneously either left hanging or being restarted

* fix issue #7900: Bind values of `null` are not replaced by

empty string anymore, when toggling between json and table

view in the web-ui.

* fix issue #7903: Regression on ISO8601 string compatibility in AQL

millisecond parts of AQL date values were limited to up to 3 digits.

Now the length of the millisecond part is unrestricted, but the

millisecond precision is still limited to up to 3 digits.

* the RocksDB primary index can now be used by the optimizer to optimize queries

that use `_key` or `_id` for sorting or for range queries.

* the web UI will now by default show the documents of a collection lexicographically

sorted when sorting documents by their `_key` values.

Previous versions of ArangoDB tried to interpret `_key` values as numeric values if

possible and sorted by these. That previous sort strategy never used an index and

could have caused unnecessary overhead. The new version will now use an index for

sorting for the RocksDB engine, but may change the order in which documents are

shown in the web UI (e.g. now a `_key` value of "10" will be shown before a `_key`

value of "9").

* fixed known issue #445: ArangoSearch ignores `_id` attribute even if `includeAllFields`

is set to `true`.

* upgraded bundled boost library to version 1.69.0

* upgraded bundled curl library to version 7.63

* Use base64url to encode and decode JWT parts.

* Added --server.jwt-secret-keyfile option.

* speed up data-modification operations in exclusive transactions in the RocksDB

storage engine

* An AQL query that uses the edge index only and returns the opposite side of

the edge can now be executed in a more optimized way, e.g.

FOR edge IN edgeCollection FILTER edge._from == "v/1" RETURN edge._to

is fully covered by RocksDB edge index. For MMFiles this rule does not apply.

* reverted accidental change to error handling in geo index

In previous versions, if non-valid geo coordinates were contained in the

indexed field of a document, the document was simply ignored an not indexed.

In 3.4.0, this was accidentally changed to generate an error, which caused

the upgrade procedure to break in some cases.

* fixed TypeError being thrown instead of validation errors when Foxx manifest

validation fails

* fix internal issue #2786: improved confirmation dialog when clicking the

Truncate button in the Web UI

* make `--help-all` now also show all hidden program options

Previously hidden program options were only returned when invoking arangod or

a client tool with the cryptic `--help-.` option. Now `--help-all` simply

retuns them as well.

The program options JSON description returned by `--dump-options` was also

improved as follows:

- the new boolean attribute "dynamic" indicates whether the option has a dynamic

default value, i.e. a value that depends on the target host capabilities or

configuration

- the new boolean attribute "requiresValue" indicates whether a boolean option

requires a value of "true" or "false" when specified. If "requiresValue" is

false, then the option can be specified without a boolean value following it,

and the option will still be set to true, e.g. `--server.authentication` is

identical to `--server.authentication true`.

- the new "category" attribute will contain a value of "command" for command-like

options, such as `--version`, `--dump-options`, `--dump-dependencies` etc.,

and "option" for all others.

* fixed issue #7586: a running query within the user interface was not shown

if the active view was `Running Queries` or `Slow Query History`.

* fixed issue #7743: Query processing discrepancy between Rocks and MMFiles databases

this change enforces the invalidation of variables in AQL queries after usage of

a COLLECT statement as documented. The documentation for variable invalidation claims

that

The COLLECT statement will eliminate all local variables in the current scope.

After COLLECT only the variables introduced by COLLECT itself are available.

However, the described behavior was not enforced when a COLLECT was preceded by a

FOR loop that was itself preceded by a COLLECT. In the following query the final

RETURN statement accesses variable `key1` though the variable should have been

invalidated by the COLLECT directly before it:

FOR x1 IN 1..2

COLLECT key1 = x1

FOR x2 IN 1..2

COLLECT key2 = x2

RETURN [key2, key1]

In previous releases, this query was

parsed ok, but the contents of variable `key1` in the final RETURN statement were

undefined.

This change is about making queries as the above fail with a parse error, as an

unknown variable `key1` is accessed here, avoiding the undefined behavior. This is

also in line with what the documentation states about variable invalidation.

* fixed issue #7763: Collect after update does not execute updates

* fixed issue #7749: AQL query result changed for COLLECT used on empty data/array

* fixed issue #7757: Using multiple filters on nested objects produces wrong results

* fixed a rare thread local dead lock situation in replication:

If a follower tries to get in sync in the last steps it requires

a lock on the leader. If the follower cancels the lock before the leader

has succeeded with locking we can end up with one thread being deadlocked.

* allow usage of floating point values in AQL without leading zeros, e.g.

`.1234`. Previous versions of ArangoDB required a leading zero in front of

the decimal separator, i.e `0.1234`.

* Foxx `req.makeAbsolute` now will return meaningful values when ArangoDB is using

a unix socket endpoint. URLs generated when using a unix socket follow the format

http://unix:<socket-path>:<url-path> used by other JS tooling.

* Updated joi library (Web UI), improved foxx mount path validation

* do not create `_routing` collection for new installations/new databases,

as it is not needed anymore. Redirects to the web interface's login screen, which

were previously handled by entries in the `_routing` collection are now handled

from the responsible REST action handler directly

Existing `_routing` collections will not be touched as they may contain other

entries as well, and will continue to work.

* do not create `_modules` collection for new databases/installations

`_modules` is only needed for custom modules, and in case a custom

module is defined via `defineModule`, the _modules collection will

be created lazily automatically.

Existing modules in existing `_modules` collections will remain

functional even after this change

* disable in-memory cache for edge and traversal data on agency nodes, as it

is not needed there

* removed bundled Valgrind headers, removed JavaScript variable `valgrind`

from the `internal` module

* upgraded JEMalloc version to 5.1.0

* use `-std=c++14` for ArangoDB compilation

* in case of resigned leader, set isReady=false in clusterInventory

* abort RemoveFollower job if not enough in-sync followers or leader failure

* fix shrinkCluster for satelliteCollections

* fix crash in agency supervision when leadership is lost

* leader's timestamps in append entries protocol

* speed up agency supervision in case of many jobs

* fix log spam in agency supervision when leader resigned

* make AddFollower less aggressive

* lots of agency performance improvements, mostly avoiding copying

* priority queue for maintenance jobs

* do not wait for replication after each job execution in Supervision

* add support for configuring custom Analyzers via JavaScript and REST

v3.4.6 (2019-05-21)

-------------------

* fixed a crash when posting an async request to the server using the "x-arango-async"

request header and the server's request queue was full

* added error code 1240 "incomplete read" for RocksDB-based reads which cannot retrieve

documents due to the RocksDB block cache being size-restricted (with size limit enforced)

and uncompressed data blocks not fitting into the block cache

The error can only occur for collection or index scans with the RocksDB storage engine

when the RocksDB block cache is used and set to a very small size, plus its maximum size is

enforced by setting the `--rocksdb.enforce-block-cache-size-limit` option to `true`.

Previously these incomplete reads could have been ignored silently, making collection or

index scans return less documents than there were actually present.

* fixed internal issue #3918: added optional second parameter "withId" to AQL

function PREGEL_RESULT

this parameter defaults to `false`. When set to `true` the results of the Pregel

computation run will also contain the `_id` attribute for each vertex and not

just `_key`. This allows distinguishing vertices from different vertex collections.

* fixed issue #7011: description when replacing a foxx application was misleading

* fixed issue #8841: Graph Viewer dropped ability to edit an edge after

rerendering.

* upgraded arangodb starter version to 0.14.3

* fix removal of sort order for sorted results that were passed into a RETURN DISTINCT

in an AQL query running on a collection with more than a single shard. In this case,

the results from the individual shards were sorted, but the overall result was not

sorted according to the sort criterion.

* removed bug during start up with a single agent, that leads to dbserver crash.

* fix the creation of debug packages (in the web interface) for queries that

involve smart graphs and/or multiple edge collections from a traversal

* add --compress-output flag to arangodump. Activates gzip compression for

collection data. Metadata files, such as .structure.json and .view.json,

do not get compressed. No option is needed for arangorestore to restore

.data.json.gz files.

* fixed internal issue #3893: correct wrong option name `--database.ignore-logfile-errors`

in error message and correct it to `--database.ignore-datafile-errors`

* fixed internal issue #2946: create graph autocomplete was not working under

certain circumstances.

* fixed a memory leak in PRUNE operation for traversals.

The leak occurred only when PRUNE was actively used in queries.

* fixed a crash (SIGSEGV) when opening a RocksDB database directory that

contained an empty (0 bytes filesize) journal file and encryption was in use.

* improve test performance by doing the 0.1 delay between phase 1 and 2 only

if phase 1 is slow

* port back timestamp replication in agency from devel

* better logging when scheduler queue is half-full and full

v3.4.5 (2019-03-27)

-------------------

* fixed a shutdown issue when the server was shut down while there were

active Pregel jobs executing

* fixed internal issue #3815: fixed the removal of connected edges when

removing a vertex graph node in a smart graph environment.

* added AQL functions CRC32 and FNV64 for hashing data

* internal issue #2276: fixed the sorting of the databases in the database

selection dropdown in the web ui. The sort order differed based on whether

authentication was enabled or disabled.

* fixed internal issue #3789: restricted the allowed query names for user-

defined custom queries within the web ui.

* fixed internal issue #3546: improved the shards view in the web ui if there

is only one shard to display.

* fixed a display issues when editing a graph within the web ui

* fixed internal-issue #3787: Let user know about conflicting attribute in

AQL queries if information is available.

* fixed issue #8294: wrong equals behavior on arrays with ArangoSearch

* fixed internal issue #528: ArangoSearch range query sometimes doesn't work

correctly with numeric values

* fixed internal issue #3757: when restarting a follower in active failover mode,

try an incremental sync instead of a full resync. Fixed also the case where a

double resync was made

* don't check for the presence of ArangoDB upgrades available when firing up an

arangosh enterprise edition build

* added startup option `--rocksdb.allow-fallocate`

When set to true, allows RocksDB to use the fallocate call. If false, fallocate

calls are bypassed and no preallocation is done. Preallocation is turned on by

default, but can be turned off for operating system versions that are known to

have issues with it.

This option only has an effect on operating systems that support fallocate.

* added startup option `--rocksdb.limit-open-files-at-startup`

If set to true, this will limit the amount of .sst files RocksDB will inspect at

startup, which can reduce the number of IO operations performed at start.

* don't run compact() on a collection after a truncate() was done in the same

transaction

running compact() in the same transaction will only increase the data size on

disk due to RocksDB not being able to remove any documents physically due to the

snapshot that is taken at transaction start.

This change also exposes db.<collection>.compact() in the arangosh, in order to

manually run a compaction on the data range of a collection should it be needed

for maintenance.

* don't attempt to remove non-existing WAL files, because such attempt will

trigger unnecessary error log messages in the RocksDB library

* updated arangosync to 0.6.3

* added --log.file-mode to specify a file mode of newly created log files

* added --log.file-group to specify the group of newly created log files

* fixed some escaping issues within the web ui.

* fixed issue #8359: How to update a document with unique index (constraints)?

* when restarting a follower in active failover mode, try an incremental sync instead

of a full resync

* add "PRUNE <condition>" to AQL Traversals (internal issue #3068). This allows to early

abort searching of unnecessary branches within a traversal.

PRUNE is only allowed in the Traversal statement and only between the graphdefinition

and the options of the traversal.

e.g.:

FOR v, e, p IN 1..3 OUTBOUND @source GRAPH "myGraph"

PRUNE v.value == "bar"

OPTIONS {} /* These options remain optional */

RETURN v

for more details refer to the documentation chapter.

* added option `--console.history` to arangosh for controlling whether

the command-line history should be loaded from and persisted in a file.

The default value for this option is `true`. Setting it to `false`

will make arangosh not load any command-line history from the history

file, and not store the current session's history when the shell is

exited. The command-line history will then only be available in the

current shell session.

* display the server role when connecting arangosh against a server (e.g.

SINGLE, COORDINATOR)

* added replication applier state figures `totalDocuments` and `totalRemovals` to

access the number of document insert/replace operations and the number of document

removals operations separately. Also added figures `totalApplyTime` and

`totalFetchTime` for determining the total time the replication spent for

applying changes or fetching new data from the master. Also added are the figures

`averageApplyTime` and `averageFetchTime`, which show the average time spent

for applying a batch or for fetching data from the master, resp.

* fixed race condition in which the value of the informational replication applier

figure `ticksBehind` could underflow and thus show a very huge number of ticks.

* always clear all ongoing replication transactions on the slave if the slave

discovers the data it has asked for is not present anymore on the master and the

`requireFromPresent` value for the applier is set to `false`.

In this case aborting the ongoing transactions on the slave is necessary because

they may have held exclusive locks on collections, which may otherwise not be

released.

* added option `--rocksdb.wal-archive-size-limit` for controlling the

maximum total size (in bytes) of archived WAL files. The default is 0

(meaning: unlimited).

When setting the value to a size bigger than 0, the RocksDB storage engine

will force a removal of archived WAL files if the total size of the archive

exceeds the configured size. The option can be used to get rid of archived

WAL files in a disk size-constrained environment.

Note that archived WAL files are normally deleted automatically after a

short while when there is no follower attached that may read from the archive.

However, in case when there are followers attached that may read from the

archive, WAL files normally remain in the archive until their contents have

been streamed to the followers. In case there are slow followers that cannot

catch up this will cause a growth of the WAL files archive over time.

The option `--rocksdb.wal-archive-size-limit` can now be used to force a

deletion of WAL files from the archive even if there are followers attached

that may want to read the archive. In case the option is set and a leader

deletes files from the archive that followers want to read, this will abort

the replication on the followers. Followers can however restart the replication

doing a resync.

* agents need to be able to overwrite a compacted state with same _key

* in case of resigned leader, set isReady=false in clusterInventory

* abort RemoveFollower job if not enough in-sync followers or leader failure

* fix shrinkCluster for satelliteCollections

* fix crash in agency supervision when leadership is lost

* speed up supervision in agency for large numbers of jobs

* fix log spamming after leader resignation in agency

* make AddFollower less aggressive

* fix cases, where invalid json could be generated in agents' store dumps

* coordinator route for full agency dumps contains compactions and

time stamps

* lots of agency performance improvements, mostly avoiding copying

* priority queue for maintenance jobs

* do not wait for replication after each job execution in Supervision

* fix a blockage in MoveShard if a failover happens during the operation

* check health of servers in Current before scheduling removeFollower jobs

* wait for statistics collections to be created before running resilience

tests

* fix ttl values in agency when key overwritten with no ttl

v3.4.4 (2019-03-12)

-------------------

* added missing test for success in failed leader: this could lead

to a crash

* follow up to fix JWT authentication in arangosh (#7530):

also fix reconnect

* now also syncing _jobs and _queues collections in active failover mode

* fixed overflow in Windows NowNanos in RocksDB

* fixed issue #8165: AQL optimizer does not pick up multiple geo index

* when creating a new database with an initial user, set the database permission

for this user as specified in the documentation

* Supervision fix: abort MoveShard job does not leave a lock behind,

* Supervision fix: abort MoveShard (leader) job moves forwards when point

of no return has been reached,

* Supervision fix: abort CleanOutServer job does not leave server in

ToBeCleanedServers,

* Supervision fix: move shard with data stopped to early due to wrong usage

of compare function

* Supervision fix: AddFollower only counts good followers, fixing a

situation after a FailedLeader job could not find a new working

follower

* Supervision fix: FailedLeader now also considers temporarily BAD

servers as replacement followers and does not block servers which

currently receive a new shard

* Supervision fix: Servers in ToBeCleanedServers are no longer considered

as replacement servers

* Maintenance fix: added precondition of unchanged Plan in phase2

* Allow MoveShard from leader to a follower, thus swapping the two

* Supervision fix: Satellite collections, various fixes

* Add coordinator route for agency dump

* speed up replication of transactions containing updates of existing documents.

The replication protocol does not provide any information on whether a document

was inserted on the master or updated/replaced. Therefore the slave will always

try an insert first, and move to a replace if the insert fails with "unique

constraint violation". This case is however very costly in a bigger transaction,

as the rollback of the insert will force the underlying RocksDB write batch to be

entirely rewritten. To circumvent rewriting entire write batches, we now do a

quick check if the target document already exists, and then branch to either

insert or replace internally.

v3.4.3 (2019-02-19)

-------------------

* fixed JS AQL query objects with empty query strings not being recognized as AQL queries

* fixed issue #8137: NULL input field generates U_ILLEGAL_ARGUMENT_ERROR

* fixed issue #8108: AQL variable - not working query since upgrade to 3.4 release

* fixed possible segfault when using COLLECT with a LIMIT and an offset

* fixed COLLECT forgetting top-level variables after 1000 rows

* fix undefined behavior when calling user-defined AQL functions from an AQL

query via a streaming cursor

* fix broken validation of tick range in arangodump

* updated bundled curl library to version 7.63.0

* added "peakMemoryUsage" in query results figures, showing the peak memory

usage of the executed query. In a cluster, the value contains the peak memory

usage across all shards, but it is not summed up across shards.

* data masking: better documentation, fixed default phone number,

changed default range to -100 and 100 for integer masking function

* fix supervision's failed server handling to transactionally create

all failed leader/followers along

v3.4.2.1 (2019-02-01)

---------------------

* upgrade to new velocypack version

v3.4.2 (2019-01-24)

-------------------

* added configurable masking of dumped data via `arangodump` tool to obfuscate exported sensible data

* upgraded to OpenSSL 1.1.0j

* fixed an issue with AQL query IN index lookup conditions being converted into

empty arrays when they were shared between multiple nodes of a lookup condition

that used an IN array lookup in an OR that was multiplied due to DNF transformations

This issue affected queries such as the following

FILTER (... && ...) || doc.indexAttribute IN non-empty-array

* upgraded arangodb starter version to 0.14.0

* upgraded arangosync version to 0.6.2

* fixed an issue where a crashed coordinator can lead to some Foxx queue jobs

erroneously either left hanging or being restarted

* fix issue #7903: Regression on ISO8601 string compatibility in AQL

millisecond parts of AQL date values were limited to up to 3 digits.

Now the length of the millisecond part is unrestricted, but the

millisecond precision is still limited to up to 3 digits.

* fix issue #7900: Bind values of `null` are not replaced by

empty string anymore, when toggling between json and table

view in the web-ui.

* Use base64url to encode and decode JWT parts.

* added AQL function `CHECK_DOCUMENT` for document validity checks

* when detecting parse errors in the JSON input sent to the restore API, now

abort with a proper error containing the problem description instead of aborting

but hiding there was a problem.

* do not respond with an internal error in case of JSON parse errors detected

in incoming HTTP requests

* added arangorestore option `--cleanup-duplicate-attributes` to clean up input documents