devel

-----

* removed option `--cluster.my-local-info` in favor of persisted server UUIDs

The option `--cluster.my-local-info` was deprecated since ArangoDB 3.3.

* added new collection property `cacheEnabled` which enables in-memory caching for

documents and primary index entries. Available only when using RocksDB

* arangodump now supports `--threads` option to dump collections in parallel

* fix issue #4924: removeFollower now prefers to remove the last follower(s)

* Improvement: The AQL query planner in cluster is now a bit more clever and

can prepare AQL queries with less network overhead.

This should speed up simple queries in cluster mode, on complex queries it

will most likely not show any performance effect.

It will especially show effects on collections with a very high amount of Shards.

* removed remainders of dysfunctional `/_admin/cluster-test` and `/_admin/clusterCheckPort`

API endpoints and removed them from documentation

* added new query option `stream` to enable streaming query execution via the `POST /_api/cursor` rest interface.

* fix issue #4698: databases within the UI are now displayed in a sorted order.

* fix issue #4657: fixed incomplete content type header

* Behaviour of permissions for databases and collections changed:

The new fallback rule for databases for which an access level is not explicitly specified:

Choose the higher access level of::

* A wildcard database grant

* A database grant on the `_system` database

The new fallback rule for collections for which an access level is not explicitly specified:

Choose the higher access level of::

* Any wildcard access grant in the same database, or on "*/*"

* The access level for the current database

* The access level for the `_system` database

* fix issue #4583 - add AQL ASSERT and AQL WARN

* remove _admin/echo handler

* remove long disfunctional admin/long_echo handler

* fixed Foxx API:

* PUT /_api/foxx/service: Respect force flag

* PATCH /_api/foxx/service: Check whether a service under given mount exists

* renamed startup option `--replication.automatic-failover` to

`--replication.active-failover`

using the old option name will still work in ArangoDB 3.4, but the old option

will be removed afterwards

* Index selectivity estimates for RocksDB engine are now eventually consistent.

This change addresses a previous issue where some index updates could be

"lost" from the view of the internal selectivity estimate, leading to

inaccurate estimates. The issue is solved now, but there can be up to a second

or so delay before updates are reflected in the estimates.

* support `returnOld` and `returnNew` attributes for in the following HTTP REST

APIs:

* /_api/gharial/<graph>/vertex/<collection>

* /_api/gharial/<graph>/edge/<collection>

The exception from this is that the HTTP DELETE verb for these APIs does not

support `returnOld` because that would make the existing API incompatible

* fixed issue #4160: Run arangod with "--database.auto-upgrade" option always crash

silently without error log

* fix issue #4457: create /var/tmp/arangod with correct user in supervisor mode

* fix inconvenience: If we want to start server with a non-existing

--javascript.app-path it will now be created (if possible)

* fix internal issue #478: remove unused and undocumented REST API endpoints

_admin/statistics/short and _admin/statistics/long.

These APIs were available in ArangoDB's REST API, but have not been called by

ArangoDB itself nor have they been part of the documented API. They have been

superseded by other REST APIs and were partially dysfunctional. Therefore

these two endpoints have been removed entirely.

* fixed issue #1532: reload users on restore

* fixed internal issue #1475: when restoring a cluster dump to a single server

ignore indexes of type primary and edge since we mustn't create them here.

* fix internal issue 1770: collection creation using distributeShardsLike yields

errors and did not distribute shards correctly in the following cases:

1. If numberOfShards * replicationFactor % nrDBServers != 0

(shards * replication is not divisible by DBServers).

2. If there was failover / move shard case on the leading collection

and creating the follower collection afterwards.

* fixed: REST API `POST _api/foxx` now returns HTTP code 201 on success, as documented.

returned 200 before.

* fixed: REST API `PATCH _api/foxx/dependencies` now updates the existing dependencies

instead of replacing them.

* fixed: Foxx upload of single javascript file. You now can upload via http-url pointing

to a JavaScript file.

* fixed issue #4395: If your foxx app includes an `APP` folder it got accidently removed by selfhealing

this is not the case anymore.

* fix internal issue #1439: improve performance of any-iterator for RocksDB

* added option `--rocksdb.throttle` to control whether write-throttling is enabled

Write-throttling is turned on by default, to reduce chances of compactions getting

too far behind and blocking incoming writes.

* UI: fixed an issue where a collection name gets wrongly cached within the

documents overview of a collection.

* issue #1190: added option `--create-database` for arangoimport

* UI: updated dygraph js library to version 2.1.0

* honor specified COLLECT method in AQL COLLECT options

for example, when the user explicitly asks for the COLLECT method

to be `sorted`, the optimizer will now not produce an alternative

version of the plan using the hash method.

additionally, if the user explcitly asks for the COLLECT method to

be `hash`, the optimizer will now change the existing plan to use

the hash method if possible instead of just creating an alternative

plan.

`COLLECT ... OPTIONS { method: 'sorted' }` => always use sorted method

`COLLECT ... OPTIONS { method: 'hash' }` => use hash if this is technically possible

`COLLECT ...` (no options) => create a plan using sorted, and another plan using hash method

* added C++ implementation for AQL function `SHA512()`

* added AQL function `IS_KEY`

this function checks if the value passed to it can be used as a document key,

i.e. as the value of the `_key` attribute

* add AQL functions `SORTED` and `SORTED_UNIQUE`

`SORTED` will return a sorted version of the input array using AQL's internal

comparison order

`SORTED_UNIQUE` will do the same, but additionally removes duplicates.

* Added C++ implementation for the AQL TRANSLATE function

* renamed arangoimp to arangoimport for consistency

Release packages will still install arangoimp as a symlink so user scripts

invoking arangoimp do not need to be changed

* UI: Shard distribution view now has an accordion view instead of displaying

all shards of all collections at once.

* fix issue #4393: broken handling of unix domain sockets in

JS_Download

* internal issue #1726: Supervision to remove deleted nodes from

Health did not function for multiple servers at the same time, as

agency transaction was malformed.

* added C++ implementation for AQL functions `DATE_NOW()`, `DATE_ISO8601()`,

`DATE_TIMESTAMP()`, `IS_DATESTRING()`, `DATE_DAYOFWEEK()`, `DATE_YEAR()`,

`DATE_MONTH()`, `DATE_DAY()`, `DATE_HOUR()`, `DATE_MINUTE()`, `DATE_SECOND()`,

`DATE_MILLISECOND()`, `DATE_DAYOFYEAR()`, `DATE_ISOWEEK()`, `DATE_LEAPYEAR()`,

`DATE_QUARTER()`, `DATE_DAYS_IN_MONTH()`, `DATE_ADD()`, `DATE_SUBTRACT()`,

`DATE_DIFF()` and `DATE_COMPARE()`

* fixed a bug where supervision tried to deal with shards of virtual collections

* fixed a bug where clusterinfo missed changes to plan after agency callback is registred for create collection

v3.3.4 (XXXX-XX-XX)

-------------------

* fixed issue #4651: Simple query taking forever until a request timeout error

* fix timeout issues in replication client expiration

v3.3.3 (2018-01-16)

-------------------

* fix issue #4272: VERSION file keeps disappearing

* fix internal issue #81: quotation marks disappeared when switching table/json

editor in the query editor ui

* added option `--rocksdb.throttle` to control whether write-throttling is enabled

Write-throttling is turned on by default, to reduce chances of compactions getting

too far behind and blocking incoming writes.

* fixed issue #4308: Crash when getter for error.name throws an error (on Windows)

* UI: fixed a query editor caching and parsing issue

* Fixed internal issue #1683: fixes an UI issue where a collection name gets wrongly cached

within the documents overview of a collection.

* Fixed an issue with the index estimates in RocksDB in the case a transaction is aborted.

Former the index estimates were modified if the transaction commited or not.

Now they will only be modified if the transaction commited successfully.

* UI: optimized login view for very small screen sizes

* Truncate in RocksDB will now do intermediate commits every 10.000 documents

if truncate fails or the server crashes during this operation all deletes

that have been commited so far are persisted.

* make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB

default value. This will mostly mean the default number block cache shard

bits is lower than before, allowing each shard to store more data and cause

less evictions from block cache

* issue #4222: Permission error preventing AQL query import / export on webui

* UI: optimized error messages for invalid query bind parameter

* UI: upgraded swagger ui to version 3.9.0

* issue #3504: added option `--force-same-database` for arangorestore

with this option set to true, it is possible to make any arangorestore attempt

fail if the specified target database does not match the database name

specified in the source dump's "dump.json" file. it can thus be used to

prevent restoring data into the "wrong" database

The option is set to `false` by default to ensure backwards-compatibility

* make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB

default value. This will mostly mean the default number block cache shard

bits is lower than before, allowing each shard to store more data and cause

less evictions from block cache

* fixed issue #4255: AQL SORT consuming too much memory

* fixed incorrect persistence of RAFT vote and term

v3.3.2 (2018-01-04)

-------------------

* fixed issue #4199: Internal failure: JavaScript exception in file 'arangosh.js'

at 98,7: ArangoError 4: Expecting type String

* fixed issue in agency supervision with a good server being left in

failedServers

* distinguish isReady and allInSync in clusterInventory

* fixed issue #4197: AQL statement not working in 3.3.1 when upgraded from 3.2.10

* do not reuse collection ids when restoring collections from a dump, but assign new collection ids, this should prevent collection id conflicts

v3.3.1 (2017-12-28)

-------------------

* UI: displayed wrong wfs property for a collection when using RocksDB as

storage engine

* added `--ignore-missing` option to arangoimp

this option allows importing lines with less fields than specified in the CSV

header line

* changed misleading error message from "no leader" to "not a leader"

* optimize usage of AQL FULLTEXT index function to a FOR loop with index

usage in some cases

When the optimization is applied, this especially speeds up fulltext index

queries in the cluster

* UI: improved the behavior during collection creation in a cluster environment

* Agency lockup fixes for very small machines.

* Agency performance improvement by finer grained locking.

* Use steady_clock in agency whereever possible.

* Agency prevent Supervision thread crash.

* Fix agency integer overflow in timeout calculation.

v3.3.0 (2012-12-14)

-------------------

* release version

* added a missing try/catch block in the supervision thread

v3.3.rc8 (2017-12-12)

---------------------

* UI: fixed broken foxx configuration keys. Some valid configuration values

could not be edited via the ui.

* UI: pressing the return key inside a select2 box no longer triggers the modal's

success function

* UI: coordinators and db servers are now in sorted order (ascending)

v3.3.rc7 (2017-12-07)

---------------------

* fixed issue #3741: fix terminal color output in Windows

* UI: fixed issue #3822: disabled name input field for system collections

* fixed issue #3640: limit in subquery

* fixed issue #3745: Invalid result when using OLD object with array attribute in UPSERT statement

* UI: edge collections were wrongly added to from and to vertices select box during graph creation

* UI: added not found views for documents and collections

* UI: using default user database api during database creation now

* UI: the graph viewer backend now picks one random start vertex of the

first 1000 documents instead of calling any(). The implementation of

"any" is known to scale bad on huge collections with RocksDB.

* UI: fixed disappearing of the navigation label in some case special case

* UI: the graph viewer now displays updated label values correctly.

Additionally the included node/edge editor now closes automatically

after a successful node/edge update.

* fixed issue #3917: traversals with high maximal depth take extremely long

in planning phase.

v3.3.rc4 (2017-11-28)

---------------------

* minor bug-fixes

v3.3.rc3 (2017-11-24)

---------------------

* bug-fixes

v3.3.rc2 (2017-11-22)

---------------------

* UI: document/edge editor now remembering their modes (e.g. code or tree)

* UI: optimized error messages for invalid graph definitions. Also fixed a

graph renderer cleanup error.

* UI: added a delay within the graph viewer while changing the colors of the

graph. Necessary due different browser behaviour.

* added options `--encryption.keyfile` and `--encryption.key-generator` to arangodump

and arangorestore

* UI: the graph viewer now displays updated label values correctly.

Additionally the included node/edge editor now closes automatically

after a successful node/edge update.

* removed `--recycle-ids` option for arangorestore

using that option could have led to problems on the restore, with potential

id conflicts between the originating server (the source dump server) and the

target server (the restore server)

v3.3.rc1 (2017-11-17)

---------------------

* add readonly mode REST API

* allow compilation of ArangoDB source code with g++ 7

* upgrade minimum required g++ compiler version to g++ 5.4

That means ArangoDB source code will not compile with g++ 4.x or g++ < 5.4 anymore.

* AQL: during a traversal if a vertex is not found. It will not print an ERROR to the log and continue

with a NULL value, but will register a warning at the query and continue with a NULL value.

The situation is not desired as an ERROR as ArangoDB can store edges pointing to non-existing

vertex which is perfectly valid, but it may be a n issue on the data model, so users

can directly see it on the query now and do not "by accident" have to check the LOG output.

* introduce `enforceReplicationFactor` attribute for creating collections:

this optional parameter controls if the coordinator should bail out during collection

creation if there are not enough DBServers available for the desired `replicationFactor`.

* fixed issue #3516: Show execution time in arangosh

this change adds more dynamic prompt components for arangosh

The following components are now available for dynamic prompts,

settable via the `--console.prompt` option in arangosh:

- '%t': current time as timestamp

- '%a': elpased time since ArangoShell start in seconds

- '%p': duration of last command in seconds

- '%d': name of current database

- '%e': current endpoint

- '%E': current endpoint without protocol

- '%u': current user

The time a command takes can be displayed easily by starting arangosh with `--console.prompt "%p> "`.

* make the ArangoShell refill its collection cache when a yet-unknown collection

is first accessed. This fixes the following problem:

arangosh1> db._collections(); // shell1 lists all collections

arangosh2> db._create("test"); // shell2 now creates a new collection 'test'

arangosh1> db.test.insert({}); // shell1 is not aware of the collection created

// in shell2, so the insert will fail

* make AQL `DISTINCT` not change the order of the results it is applied on

* incremental transfer of initial collection data now can handle partial

responses for a chunk, allowing the leader/master to send smaller chunks

(in terms of HTTP response size) and limit memory usage

this optimization is only active if client applications send the "offset" parameter

in their requests to PUT `/_api/replication/keys/<id>?type=docs`

* initial creation of shards for cluster collections is now faster with

`replicationFactor` values bigger than 1. this is achieved by an optimization

for the case when the collection on the leader is still empty

* potential fix for issue #3517: several "filesystem full" errors in logs

while there's a lot of disk space

* added C++ implementations for AQL function `SUBSTRING()`, `LEFT()`, `RIGHT()` and `TRIM()`

* show C++ function name of call site in ArangoDB log output

this requires option `--log.line-number` to be set to *true*

* UI: added word wrapping to query editor

* UI: fixed wrong user attribute name validation, issue #3228

* make AQL return a proper error message in case of a unique key constraint

violation. previously it only returned the generic "unique constraint violated"

error message but omitted the details about which index caused the problem.

This addresses https://stackoverflow.com/questions/46427126/arangodb-3-2-unique-constraint-violation-id-or-key

* added option `--server.local-authentication`

* UI: added user roles

* added config option `--log.color` to toggle colorful logging to terminal

* added config option `--log.thread-name` to additionally log thread names

* usernames must not start with `:role:`, added new options:

--server.authentication-timeout

--ldap.roles-attribute-name

--ldap.roles-transformation

--ldap.roles-search

--ldap.superuser-role

--ldap.roles-include

--ldap.roles-exclude

* performance improvements for full collection scans and a few other operations

in MMFiles engine

* added `--rocksdb.encryption-key-generator` for enterprise

* removed `--compat28` parameter from arangodump and replication API

older ArangoDB versions will no longer be supported by these tools.

* increase the recommended value for `/proc/sys/vm/max_map_count` to a value

eight times as high as the previous recommended value. Increasing the

values helps to prevent an ArangoDB server from running out of memory mappings.

The raised minimum recommended value may lead to ArangoDB showing some startup

warnings as follows:

WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 512000

WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=512000"'

* Foxx now warns about malformed configuration/dependency names and aliases in the manifest.

v3.2.7 (2017-11-13)

-------------------

* Cluster customers, which have upgraded from 3.1 to 3.2 need to upgrade

to 3.2.7. The cluster supervision is otherwise not operational.

* Fixed issue #3597: AQL with path filters returns unexpected results

In some cases breadth first search in combination with vertex filters

yields wrong result, the filter was not applied correctly.

* fixed some undefined behavior in some internal value caches for AQL GatherNodes

and SortNodes, which could have led to sorted results being effectively not

correctly sorted.

* make the replication applier for the RocksDB engine start automatically after a

restart of the server if the applier was configured with its `autoStart` property

set to `true`. previously the replication appliers were only automatically restarted

at server start for the MMFiles engine.

* fixed arangodump batch size adaptivity in cluster mode and upped default batch size

for arangodump

these changes speed up arangodump in cluster context

* smart graphs now return a proper inventory in response to replication inventory

requests

* fixed issue #3618: Inconsistent behavior of OR statement with object bind parameters

* only users with read/write rights on the "_system" database can now execute

"_admin/shutdown" as well as modify properties of the write-ahead log (WAL)

* increase default maximum number of V8 contexts to at least 16 if not explicitly

configured otherwise.

the procedure for determining the actual maximum value of V8 contexts is unchanged

apart from the value `16` and works as follows:

- if explicitly set, the value of the configuration option `--javascript.v8-contexts`

is used as the maximum number of V8 contexts

- when the option is not set, the maximum number of V8 contexts is determined

by the configuration option `--server.threads` if that option is set. if

`--server.threads` is not set, then the maximum number of V8 contexts is the

server's reported hardware concurrency (number of processors visible

to the arangod process). if that would result in a maximum value of less than 16

in any of these two cases, then the maximum value will be increased to 16.

* fixed issue #3447: ArangoError 1202: AQL: NotFound: (while executing) when

updating collection

* potential fix for issue #3581: Unexpected "rocksdb unique constraint

violated" with unique hash index

* fixed geo index optimizer rule for geo indexes with a single (array of coordinates)

attribute.

* improved the speed of the shards overview in cluster (API endpoint /_api/cluster/shardDistribution API)

It is now guaranteed to return after ~2 seconds even if the entire cluster is unresponsive.

* fix agency precondition check for complex objects

this fixes issues with several CAS operations in the agency

* several fixes for agency restart and shutdown

* the cluster-internal representation of planned collection objects is now more

lightweight than before, using less memory and not allocating any cache for indexes

etc.

* fixed issue #3403: How to kill long running AQL queries with the browser console's

AQL (display issue)

* fixed issue #3549: server reading ENGINE config file fails on common standard

newline character

* UI: fixed error notifications for collection modifications

* several improvements for the truncate operation on collections:

* the timeout for the truncate operation was increased in cluster mode in

order to prevent too frequent "could not truncate collection" errors

* after a truncate operation, collections in MMFiles still used disk space.

to reclaim disk space used by truncated collection, the truncate actions

in the web interface and from the ArangoShell now issue an extra WAL flush

command (in cluster mode, this command is also propagated to all servers).

the WAL flush allows all servers to write out any pending operations into the

datafiles of the truncated collection. afterwards, a final journal rotate

command is sent, which enables the compaction to entirely remove all datafiles

and journals for the truncated collection, so that all disk space can be

reclaimed

* for MMFiles a special method will be called after a truncate operation so that

all indexes of the collection can free most of their memory. previously some

indexes (hash and skiplist indexes) partially kept already allocated memory

in order to avoid future memory allocations

* after a truncate operation in the RocksDB engine, an additional compaction

will be triggered for the truncated collection. this compaction removes all

deletions from the key space so that follow-up scans over the collection's key

range do not have to filter out lots of already-removed values

These changes make truncate operations potentially more time-consuming than before,

but allow for memory/disk space savings afterwards.

* enable JEMalloc background threads for purging and returning unused memory

back to the operating system (Linux only)

JEMalloc will create its background threads on demand. The number of background

threads is capped by the number of CPUs or active arenas. The background threads run

periodically and purge unused memory pages, allowing memory to be returned to the

operating system.

This change will make the arangod process create several additional threads.

It is accompanied by an increased `TasksMax` value in the systemd service configuration

file for the arangodb3 service.

* upgraded bundled V8 engine to bugfix version v5.7.492.77

this upgrade fixes a memory leak in upstream V8 described in

https://bugs.chromium.org/p/v8/issues/detail?id=5945 that will result in memory

chunks only getting uncommitted but not unmapped

v3.2.6 (2017-10-26)

-------------------

* UI: fixed event cleanup in cluster shards view

* UI: reduced cluster dashboard api calls

* fixed a permission problem that prevented collection contents to be displayed

in the web interface

* removed posix_fadvise call from RocksDB's PosixSequentialFile::Read(). This is

consistent with Facebook PR 2573 (#3505)

this fix should improve the performance of the replication with the RocksDB

storage engine

* allow changing of collection replication factor for existing collections

* UI: replicationFactor of a collection is now changeable in a cluster

environment

* several fixes for the cluster agency

* fixed undefined behavior in the RocksDB-based geo index

* fixed Foxxmaster failover

* purging or removing the Debian/Ubuntu arangodb3 packages now properly stops

the arangod instance before actuallying purging or removing

v3.2.5 (2017-10-16)

-------------------

* general-graph module and _api/gharial now accept cluster options

for collection creation. It is now possible to set replicationFactor and

numberOfShards for all collections created via this graph object.

So adding a new collection will not result in a singleShard and

no replication anymore.

* fixed issue #3408: Hard crash in query for pagination

* minimum number of V8 contexts in console mode must be 2, not 1. this is

required to ensure the console gets one dedicated V8 context and all other

operations have at least one extra context. This requirement was not enforced

anymore.

* fixed issue #3395: AQL: cannot instantiate CollectBlock with undetermined

aggregation method

* UI: fixed wrong user attribute name validation, issue #3228

* fix potential overflow in CRC marker check when a corrupted CRC marker

is found at the very beginning of an MMFiles datafile

* UI: fixed unresponsive events in cluster shards view

* Add statistics about the V8 context counts and number of available/active/busy

threads we expose through the server statistics interface.

v3.2.4 (2017-09-26)

-------------------

* UI: no default index selected during index creation

* UI: added replicationFactor option during SmartGraph creation

* make the MMFiles compactor perform less writes during normal compaction

operation

This partially fixes issue #3144

* make the MMFiles compactor configurable

The following options have been added:

* `--compaction.db-sleep-time`: sleep interval between two compaction runs

(in s)

* `--compaction.min-interval"`: minimum sleep time between two compaction

runs (in s)

* `--compaction.min-small-data-file-size`: minimal filesize threshold

original datafiles have to be below for a compaction

* `--compaction.dead-documents-threshold`: minimum unused count of documents

in a datafile

* `--compaction.dead-size-threshold`: how many bytes of the source data file

are allowed to be unused at most

* `--compaction.dead-size-percent-threshold`: how many percent of the source

datafile should be unused at least

* `--compaction.max-files`: Maximum number of files to merge to one file

* `--compaction.max-result-file-size`: how large may the compaction result

file become (in bytes)

* `--compaction.max-file-size-factor`: how large the resulting file may

be in comparison to the collection's `--database.maximal-journal-size' setting`

* fix downwards-incompatibility in /_api/explain REST handler

* fix Windows implementation for fs.getTempPath() to also create a

sub-directory as we do on linux

* fixed a multi-threading issue in cluster-internal communication

* performance improvements for traversals and edge lookups

* removed internal memory zone handling code. the memory zones were a leftover

from the early ArangoDB days and did not provide any value in the current

implementation.

* (Enterprise only) added `skipInaccessibleCollections` option for AQL queries:

if set, AQL queries (especially graph traversals) will treat collections to

which a user has no access rights to as if these collections were empty.

* adjusted scheduler thread handling to start and stop less threads in

normal operations

* leader-follower replication catchup code has been rewritten in C++

* early stage AQL optimization now also uses the C++ implementations of

AQL functions if present. Previously it always referred to the JavaScript

implementations and ignored the C++ implementations. This change gives

more flexibility to the AQL optimizer.

* ArangoDB tty log output is now colored for log messages with levels

FATAL, ERR and WARN.

* changed the return values of AQL functions `REGEX_TEST` and `REGEX_REPLACE`

to `null` when the input regex is invalid. Previous versions of ArangoDB

partly returned `false` for invalid regexes and partly `null`.

* added `--log.role` option for arangod

When set to `true`, this option will make the ArangoDB logger print a single

character with the server's role into each logged message. The roles are:

- U: undefined/unclear (used at startup)

- S: single server

- C: coordinator

- P: primary

- A: agent

The default value for this option is `false`, so no roles will be logged.

v3.2.3 (2017-09-07)

-------------------

* fixed issue #3106: orphan collections could not be registered in general-graph module

* fixed wrong selection of the database inside the internal cluster js api

* added startup option `--server.check-max-memory-mappings` to make arangod check

the number of memory mappings currently used by the process and compare it with

the maximum number of allowed mappings as determined by /proc/sys/vm/max_map_count

The default value is `true`, so the checks will be performed. When the current

number of mappings exceeds 90% of the maximum number of mappings, the creation

of further V8 contexts will be deferred.

Note that this option is effective on Linux systems only.

* arangoimp now has a `--remove-attribute` option

* added V8 context lifetime control options

`--javascript.v8-contexts-max-invocations` and `--javascript.v8-contexts-max-age`

These options allow specifying after how many invocations a used V8 context is

disposed, or after what time a V8 context is disposed automatically after its

creation. If either of the two thresholds is reached, an idl V8 context will be

disposed.

The default value of `--javascript.v8-contexts-max-invocations` is 0, meaning that

the maximum number of invocations per context is unlimited. The default value

for `--javascript.v8-contexts-max-age` is 60 seconds.

* fixed wrong UI cluster health information

* fixed issue #3070: Add index in _jobs collection

* fixed issue #3125: HTTP Foxx API JSON parsing

* fixed issue #3120: Foxx queue: job isn't running when server.authentication = true

* fixed supervision failure detection and handling, which happened with simultaneous

agency leadership change

v3.2.2 (2017-08-23)

-------------------

* make "Rebalance shards" button work in selected database only, and not make

it rebalance the shards of all databases

* fixed issue #2847: adjust the response of the DELETE `/_api/users/database/*` calls

* fixed issue #3075: Error when upgrading arangoDB on linux ubuntu 16.04

* fixed a buffer overrun in linenoise console input library for long input strings

* increase size of the linenoise input buffer to 8 KB

* abort compilation if the detected GCC or CLANG isn't in the range of compilers

we support

* fixed spurious cluster hangups by always sending AQL-query related requests

to the correct servers, even after failover or when a follower drops

The problem with the previous shard-based approach was that responsibilities

for shards may change from one server to another at runtime, after the query

was already instanciated. The coordinator and other parts of the query then

sent further requests for the query to the servers now responsible for the

shards.

However, an AQL query must send all further requests to the same servers on

which the query was originally instanciated, even in case of failover.

Otherwise this would potentially send requests to servers that do not know

about the query, and would also send query shutdown requests to the wrong

servers, leading to abandoned queries piling up and using resources until

they automatically time out.

* fixed issue with RocksDB engine acquiring the collection count values too

early, leading to the collection count values potentially being slightly off

even in exclusive transactions (for which the exclusive access should provide

an always-correct count value)

* fixed some issues in leader-follower catch-up code, specifically for the

RocksDB engine

* make V8 log fatal errors to syslog before it terminates the process.

This change is effective on Linux only.

* fixed issue with MMFiles engine creating superfluous collection journals

on shutdown

* fixed issue #3067: Upgrade from 3.2 to 3.2.1 reset autoincrement keys

* fixed issue #3044: ArangoDB server shutdown unexpectedly

* fixed issue #3039: Incorrect filter interpretation

* fixed issue #3037: Foxx, internal server error when I try to add a new service

* improved MMFiles fulltext index document removal performance

and fulltext index query performance for bigger result sets

* ui: fixed a display bug within the slow and running queries view

* ui: fixed a bug when success event triggers twice in a modal

* ui: fixed the appearance of the documents filter

* ui: graph vertex collections not restricted to 10 anymore

* fixed issue #2835: UI detection of JWT token in case of server restart or upgrade

* upgrade jemalloc version to 5.0.1

This fixes problems with the memory allocator returing "out of memory" when

calling munmap to free memory in order to return it to the OS.

It seems that calling munmap on Linux can increase the number of mappings, at least

when a region is partially unmapped. This can lead to the process exceeding its

maximum number of mappings, and munmap and future calls to mmap returning errors.

jemalloc version 5.0.1 does not have the `--enable-munmap` configure option anymore,

so the problem is avoided. To return memory to the OS eventually, jemalloc 5's

background purge threads are used on Linux.

* fixed issue #2978: log something more obvious when you log a Buffer

* fixed issue #2982: AQL parse error?

* fixed issue #3125: HTTP Foxx API Json parsing

v3.2.1 (2017-08-09)

-------------------

* added C++ implementations for AQL functions `LEFT()`, `RIGHT()` and `TRIM()`

* fixed docs for issue #2968: Collection _key autoincrement value increases on error

* fixed issue #3011: Optimizer rule reduce-extraction-to-projection breaks queries

* Now allowing to restore users in a sharded environment as well

It is still not possible to restore collections that are sharded

differently than by _key.

* fixed an issue with restoring of system collections and user rights.

It was not possible to restore users into an authenticated server.

* fixed issue #2977: Documentation for db._createDatabase is wrong

* ui: added bind parameters to slow query history view

* fixed issue #1751: Slow Query API should provide bind parameters, webui should display them

* ui: fixed a bug when moving multiple documents was not possible

* fixed docs for issue #2968: Collection _key autoincrement value increases on error

* AQL CHAR_LENGTH(null) returns now 0. Since AQL TO_STRING(null) is '' (string of length 0)

* ui: now supports single js file upload for Foxx services in addition to zip files

* fixed a multi-threading issue in the agency when callElection was called

while the Supervision was calling updateSnapshot

* added startup option `--query.tracking-with-bindvars`

This option controls whether the list of currently running queries

and the list of slow queries should contain the bind variables used

in the queries or not.

The option can be changed at runtime using the commands

// enables tracking of bind variables

// set to false to turn tracking of bind variables off

var value = true;

require("@arangodb/aql/queries").properties({

trackBindVars: value

});

* index selectivity estimates are now available in the cluster as well

* fixed issue #2943: loadIndexesIntoMemory not returning the same structure

as the rest of the collection APIs

* fixed issue #2949: ArangoError 1208: illegal name

* fixed issue #2874: Collection properties do not return `isVolatile`

attribute

* potential fix for issue #2939: Segmentation fault when starting

coordinator node

* fixed issue #2810: out of memory error when running UPDATE/REPLACE

on medium-size collection

* fix potential deadlock errors in collector thread

* disallow the usage of volatile collections in the RocksDB engine

by throwing an error when a collection is created with attribute

`isVolatile` set to `true`.

Volatile collections are unsupported by the RocksDB engine, so

creating them should not succeed and silently create a non-volatile

collection

* prevent V8 from issuing SIGILL instructions when it runs out of memory

Now arangod will attempt to log a FATAL error into its logfile in case V8

runs out of memory. In case V8 runs out of memory, it will still terminate the

entire process. But at least there should be something in the ArangoDB logs

indicating what the problem was. Apart from that, the arangod process should

now be exited with SIGABRT rather than SIGILL as it shouldn't return into the

V8 code that aborted the process with `__builtin_trap`.

this potentially fixes issue #2920: DBServer crashing automatically post upgrade to 3.2

* Foxx queues and tasks now ensure that the scripts in them run with the same

permissions as the Foxx code who started the task / queue

* fixed issue #2928: Offset problems

* fixed issue #2876: wrong skiplist index usage in edge collection

* fixed issue #2868: cname missing from logger-follow results in rocksdb

* fixed issue #2889: Traversal query using incorrect collection id

* fixed issue #2884: AQL traversal uniqueness constraints "propagating" to other traversals? Weird results

* arangoexport: added `--query` option for passing an AQL query to export the result

* fixed issue #2879: No result when querying for the last record of a query

* ui: allows now to edit default access level for collections in database

_system for all users except the root user.

* The _users collection is no longer accessible outside the arngod process, _queues is always read-only

* added new option "--rocksdb.max-background-jobs"

* removed options "--rocksdb.max-background-compactions", "--rocksdb.base-background-compactions" and "--rocksdb.max-background-flushes"

* option "--rocksdb.compaction-read-ahead-size" now defaults to 2MB

* change Windows build so that RocksDB doesn't enforce AVX optimizations by default

This fixes startup crashes on servers that do not have AVX CPU extensions

* speed up RocksDB secondary index creation and dropping