devel

-----

* removed `--recycle-ids` option for arangorestore

using that option could have led to problems on the restore, with potential

id conflicts between the originating server (the source dump server) and the

target server (the restore server)

* add readonly mode REST API

* allow compilation of ArangoDB source code with g++7

* AQL: during a traversal if a vertex is not found. It will not print an ERROR to the log and continue

with a NULL value, but will register a warning at the query and continue with a NULL value.

The situation is not desired as an ERROR as ArangoDB can store edges pointing to non-existing

vertex which is perfectly valid, but it may be a n issue on the data model, so users

can directly see it on the query now and do not "by accident" have to check the LOG output.

* potential fix for issue #3562: Document WITHIN_RECTANGLE not found

* introduce `enforceReplicationFactor` attribute for creating collections:

this optional parameter controls if the coordinator should bail out during collection

creation if there are not enough DBServers available for the desired `replicationFactor`.

* fixed issue #3516: Show execution time in arangosh

this change adds more dynamic prompt components for arangosh

The following components are now available for dynamic prompts,

settable via the `--console.prompt` option in arangosh:

- '%t': current time as timestamp

- '%p': duration of last command in seconds

- '%d': name of current database

- '%e': current endpoint

- '%E': current endpoint without protocol

- '%u': current user

The time a command takes can be displayed easily by starting arangosh with `--console.prompt "%p> "`.

* make the ArangoShell refill its collection cache when a yet-unknown collection

is first accessed. This fixes the following problem:

arangosh1> db._collections(); // shell1 lists all collections

arangosh2> db._create("test"); // shell2 now creates a new collection 'test'

arangosh1> db.test.insert({}); // shell1 is not aware of the collection created

// in shell2, so the insert will fail

* make AQL `DISTINCT` not change the order of the results it is applied on

* incremental transfer of initial collection data now can handle partial

responses for a chunk, allowing the leader/master to send smaller chunks

(in terms of HTTP response size) and limit memory usage

this optimization is only active if client applications send the "offset" parameter

in their requests to PUT `/_api/replication/keys/<id>?type=docs`

* initial creation of shards for cluster collections is now faster with

`replicationFactor` values bigger than 1. this is achieved by an optimization

for the case when the collection on the leader is still empty

* potential fix for issue #3517: several "filesystem full" errors in logs

while there's a lot of disk space

* added C++ implementations for AQL function `SUBSTRING()`, `LEFT()`, `RIGHT()` and `TRIM()`

* show C++ function name of call site in ArangoDB log output

this requires option `--log.line-number` to be set to *true*

* UI: added word wrapping to query editor

* UI: fixed wrong user attribute name validation, issue #3228

* make AQL return a proper error message in case of a unique key constraint

violation. previously it only returned the generic "unique constraint violated"

error message but omitted the details about which index caused the problem.

This addresses https://stackoverflow.com/questions/46427126/arangodb-3-2-unique-constraint-violation-id-or-key

* added option `--server.local-authentication`

* UI: added user roles

* added config option `--log.color` to toggle colorful logging to terminal

* added config option `--log.thread-name` to additionally log thread names

* usernames must not start with `:role:`, added new options:

--server.authentication-timeout

--ldap.roles-attribute-name

--ldap.roles-transformation

--ldap.roles-search

--ldap.superuser-role

--ldap.roles-include

--ldap.roles-exclude

* performance improvements for full collection scans and a few other operations

in MMFiles engine

* added `--rocksdb.encryption-key-generator` for enterprise

* removed `--compat28` parameter from arangodump and replication API

older ArangoDB versions will no longer be supported by these tools.

* increase the recommended value for `/proc/sys/vm/max_map_count` to a value

eight times as high as the previous recommended value. Increasing the

values helps to prevent an ArangoDB server from running out of memory mappings.

The raised minimum recommended value may lead to ArangoDB showing some startup

warnings as follows:

WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 512000

WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=512000"'

v3.2.7 (2017-11-13)

-------------------

* Cluster customers, which have upgraded from 3.1 to 3.2 need to upgrade

to 3.2.7. The cluster supervision is otherwise not operational.

* Fixed issue #3597: AQL with path filters returns unexpected results

In some cases breadth first search in combination with vertex filters

yields wrong result, the filter was not applied correctly.

* fixed some undefined behavior in some internal value caches for AQL GatherNodes

and SortNodes, which could have led to sorted results being effectively not

correctly sorted.

* make the replication applier for the RocksDB engine start automatically after a

restart of the server if the applier was configured with its `autoStart` property

set to `true`. previously the replication appliers were only automatically restarted

at server start for the MMFiles engine.

* fixed arangodump batch size adaptivity in cluster mode and upped default batch size

for arangodump

these changes speed up arangodump in cluster context

* smart graphs now return a proper inventory in response to replication inventory

requests

* fixed issue #3618: Inconsistent behavior of OR statement with object bind parameters

* only users with read/write rights on the "_system" database can now execute

"_admin/shutdown" as well as modify properties of the write-ahead log (WAL)

* increase default maximum number of V8 contexts to at least 16 if not explicitly

configured otherwise.

the procedure for determining the actual maximum value of V8 contexts is unchanged

apart from the value `16` and works as follows:

- if explicitly set, the value of the configuration option `--javascript.v8-contexts`

is used as the maximum number of V8 contexts

- when the option is not set, the maximum number of V8 contexts is determined

by the configuration option `--server.threads` if that option is set. if

`--server.threads` is not set, then the maximum number of V8 contexts is the

server's reported hardware concurrency (number of processors visible

to the arangod process). if that would result in a maximum value of less than 16

in any of these two cases, then the maximum value will be increased to 16.

* fixed issue #3447: ArangoError 1202: AQL: NotFound: (while executing) when

updating collection

* potential fix for issue #3581: Unexpected "rocksdb unique constraint

violated" with unique hash index

* fixed geo index optimizer rule for geo indexes with a single (array of coordinates)

attribute.

* improved the speed of the shards overview in cluster (API endpoint /_api/cluster/shardDistribution API)

It is now guaranteed to return after ~2 seconds even if the entire cluster is unresponsive.

* fix agency precondition check for complex objects

this fixes issues with several CAS operations in the agency

* several fixes for agency restart and shutdown

* the cluster-internal representation of planned collection objects is now more

lightweight than before, using less memory and not allocating any cache for indexes

etc.

* fixed issue #3403: How to kill long running AQL queries with the browser console's

AQL (display issue)

* fixed issue #3549: server reading ENGINE config file fails on common standard

newline character

* UI: fixed error notifications for collection modifications

* several improvements for the truncate operation on collections:

* the timeout for the truncate operation was increased in cluster mode in

order to prevent too frequent "could not truncate collection" errors

* after a truncate operation, collections in MMFiles still used disk space.

to reclaim disk space used by truncated collection, the truncate actions

in the web interface and from the ArangoShell now issue an extra WAL flush

command (in cluster mode, this command is also propagated to all servers).

the WAL flush allows all servers to write out any pending operations into the

datafiles of the truncated collection. afterwards, a final journal rotate

command is sent, which enables the compaction to entirely remove all datafiles

and journals for the truncated collection, so that all disk space can be

reclaimed

* for MMFiles a special method will be called after a truncate operation so that

all indexes of the collection can free most of their memory. previously some

indexes (hash and skiplist indexes) partially kept already allocated memory

in order to avoid future memory allocations

* after a truncate operation in the RocksDB engine, an additional compaction

will be triggered for the truncated collection. this compaction removes all

deletions from the key space so that follow-up scans over the collection's key

range do not have to filter out lots of already-removed values

These changes make truncate operations potentially more time-consuming than before,

but allow for memory/disk space savings afterwards.

* enable JEMalloc background threads for purging and returning unused memory

back to the operating system (Linux only)

JEMalloc will create its background threads on demand. The number of background

threads is capped by the number of CPUs or active arenas. The background threads run

periodically and purge unused memory pages, allowing memory to be returned to the

operating system.

This change will make the arangod process create several additional threads.

It is accompanied by an increased `TasksMax` value in the systemd service configuration

file for the arangodb3 service.

* upgraded bundled V8 engine to bugfix version v5.7.492.77

this upgrade fixes a memory leak in upstream V8 described in

https://bugs.chromium.org/p/v8/issues/detail?id=5945 that will result in memory

chunks only getting uncommitted but not unmapped

v3.2.6 (2017-10-26)

-------------------

* UI: fixed event cleanup in cluster shards view

* UI: reduced cluster dashboard api calls

* fixed a permission problem that prevented collection contents to be displayed

in the web interface

* removed posix_fadvise call from RocksDB's PosixSequentialFile::Read(). This is

consistent with Facebook PR 2573 (#3505)

this fix should improve the performance of the replication with the RocksDB

storage engine

* allow changing of collection replication factor for existing collections

* UI: replicationFactor of a collection is now changeable in a cluster

environment

* several fixes for the cluster agency

* fixed undefined behavior in the RocksDB-based geo index

* fixed Foxxmaster failover

* purging or removing the Debian/Ubuntu arangodb3 packages now properly stops

the arangod instance before actuallying purging or removing

v3.2.5 (2017-10-16)

-------------------

* general-graph module and _api/gharial now accept cluster options

for collection creation. It is now possible to set replicationFactor and

numberOfShards for all collections created via this graph object.

So adding a new collection will not result in a singleShard and

no replication anymore.

* fixed issue #3408: Hard crash in query for pagination

* minimum number of V8 contexts in console mode must be 2, not 1. this is

required to ensure the console gets one dedicated V8 context and all other

operations have at least one extra context. This requirement was not enforced

anymore.

* fixed issue #3395: AQL: cannot instantiate CollectBlock with undetermined

aggregation method

* UI: fixed wrong user attribute name validation, issue #3228

* fix potential overflow in CRC marker check when a corrupted CRC marker

is found at the very beginning of an MMFiles datafile

* UI: fixed unresponsive events in cluster shards view

* Add statistics about the V8 context counts and number of available/active/busy

threads we expose through the server statistics interface.

v3.2.4 (2017-09-26)

-------------------

* UI: no default index selected during index creation

* UI: added replicationFactor option during SmartGraph creation

* make the MMFiles compactor perform less writes during normal compaction

operation

This partially fixes issue #3144

* make the MMFiles compactor configurable

The following options have been added:

* `--compaction.db-sleep-time`: sleep interval between two compaction runs

(in s)

* `--compaction.min-interval"`: minimum sleep time between two compaction

runs (in s)

* `--compaction.min-small-data-file-size`: minimal filesize threshold

original datafiles have to be below for a compaction

* `--compaction.dead-documents-threshold`: minimum unused count of documents

in a datafile

* `--compaction.dead-size-threshold`: how many bytes of the source data file

are allowed to be unused at most

* `--compaction.dead-size-percent-threshold`: how many percent of the source

datafile should be unused at least

* `--compaction.max-files`: Maximum number of files to merge to one file

* `--compaction.max-result-file-size`: how large may the compaction result

file become (in bytes)

* `--compaction.max-file-size-factor`: how large the resulting file may

be in comparison to the collection's `--database.maximal-journal-size' setting`

* fix downwards-incompatibility in /_api/explain REST handler

* fix Windows implementation for fs.getTempPath() to also create a

sub-directory as we do on linux

* fixed a multi-threading issue in cluster-internal communication

* performance improvements for traversals and edge lookups

* removed internal memory zone handling code. the memory zones were a leftover

from the early ArangoDB days and did not provide any value in the current

implementation.

* (Enterprise only) added `skipInaccessibleCollections` option for AQL queries:

if set, AQL queries (especially graph traversals) will treat collections to

which a user has no access rights to as if these collections were empty.

* adjusted scheduler thread handling to start and stop less threads in

normal operations

* leader-follower replication catchup code has been rewritten in C++

* early stage AQL optimization now also uses the C++ implementations of

AQL functions if present. Previously it always referred to the JavaScript

implementations and ignored the C++ implementations. This change gives

more flexibility to the AQL optimizer.

* ArangoDB tty log output is now colored for log messages with levels

FATAL, ERR and WARN.

* changed the return values of AQL functions `REGEX_TEST` and `REGEX_REPLACE`

to `null` when the input regex is invalid. Previous versions of ArangoDB

partly returned `false` for invalid regexes and partly `null`.

* added `--log.role` option for arangod

When set to `true`, this option will make the ArangoDB logger print a single

character with the server's role into each logged message. The roles are:

- U: undefined/unclear (used at startup)

- S: single server

- C: coordinator

- P: primary

- A: agent

The default value for this option is `false`, so no roles will be logged.

v3.2.3 (2017-09-07)

-------------------

* fixed issue #3106: orphan collections could not be registered in general-graph module

* fixed wrong selection of the database inside the internal cluster js api

* added startup option `--server.check-max-memory-mappings` to make arangod check

the number of memory mappings currently used by the process and compare it with

the maximum number of allowed mappings as determined by /proc/sys/vm/max_map_count

The default value is `true`, so the checks will be performed. When the current

number of mappings exceeds 90% of the maximum number of mappings, the creation

of further V8 contexts will be deferred.

Note that this option is effective on Linux systems only.

* arangoimp now has a `--remove-attribute` option

* added V8 context lifetime control options

`--javascript.v8-contexts-max-invocations` and `--javascript.v8-contexts-max-age`

These options allow specifying after how many invocations a used V8 context is

disposed, or after what time a V8 context is disposed automatically after its

creation. If either of the two thresholds is reached, an idl V8 context will be

disposed.

The default value of `--javascript.v8-contexts-max-invocations` is 0, meaning that

the maximum number of invocations per context is unlimited. The default value

for `--javascript.v8-contexts-max-age` is 60 seconds.

* fixed wrong UI cluster health information

* fixed issue #3070: Add index in _jobs collection

* fixed issue #3125: HTTP Foxx API JSON parsing

* fixed issue #3120: Foxx queue: job isn't running when server.authentication = true

* fixed supervision failure detection and handling, which happened with simultaneous

agency leadership change

v3.2.2 (2017-08-23)

-------------------

* make "Rebalance shards" button work in selected database only, and not make

it rebalance the shards of all databases

* fixed issue #2847: adjust the response of the DELETE `/_api/users/database/*` calls

* fixed issue #3075: Error when upgrading arangoDB on linux ubuntu 16.04

* fixed a buffer overrun in linenoise console input library for long input strings

* increase size of the linenoise input buffer to 8 KB

* abort compilation if the detected GCC or CLANG isn't in the range of compilers

we support

* fixed spurious cluster hangups by always sending AQL-query related requests

to the correct servers, even after failover or when a follower drops

The problem with the previous shard-based approach was that responsibilities

for shards may change from one server to another at runtime, after the query

was already instanciated. The coordinator and other parts of the query then

sent further requests for the query to the servers now responsible for the

shards.

However, an AQL query must send all further requests to the same servers on

which the query was originally instanciated, even in case of failover.

Otherwise this would potentially send requests to servers that do not know

about the query, and would also send query shutdown requests to the wrong

servers, leading to abandoned queries piling up and using resources until

they automatically time out.

* fixed issue with RocksDB engine acquiring the collection count values too

early, leading to the collection count values potentially being slightly off

even in exclusive transactions (for which the exclusive access should provide

an always-correct count value)

* fixed some issues in leader-follower catch-up code, specifically for the

RocksDB engine

* make V8 log fatal errors to syslog before it terminates the process.

This change is effective on Linux only.

* fixed issue with MMFiles engine creating superfluous collection journals

on shutdown

* fixed issue #3067: Upgrade from 3.2 to 3.2.1 reset autoincrement keys

* fixed issue #3044: ArangoDB server shutdown unexpectedly

* fixed issue #3039: Incorrect filter interpretation

* fixed issue #3037: Foxx, internal server error when I try to add a new service

* improved MMFiles fulltext index document removal performance

and fulltext index query performance for bigger result sets

* ui: fixed a display bug within the slow and running queries view

* ui: fixed a bug when success event triggers twice in a modal

* ui: fixed the appearance of the documents filter

* ui: graph vertex collections not restricted to 10 anymore

* fixed issue #2835: UI detection of JWT token in case of server restart or upgrade

* upgrade jemalloc version to 5.0.1

This fixes problems with the memory allocator returing "out of memory" when

calling munmap to free memory in order to return it to the OS.

It seems that calling munmap on Linux can increase the number of mappings, at least

when a region is partially unmapped. This can lead to the process exceeding its

maximum number of mappings, and munmap and future calls to mmap returning errors.

jemalloc version 5.0.1 does not have the `--enable-munmap` configure option anymore,

so the problem is avoided. To return memory to the OS eventually, jemalloc 5's

background purge threads are used on Linux.

* fixed issue #2978: log something more obvious when you log a Buffer

* fixed issue #2982: AQL parse error?

* fixed issue #3125: HTTP Foxx API Json parsing

v3.2.1 (2017-08-09)

-------------------

* added C++ implementations for AQL functions `LEFT()`, `RIGHT()` and `TRIM()`

* fixed docs for issue #2968: Collection _key autoincrement value increases on error

* fixed issue #3011: Optimizer rule reduce-extraction-to-projection breaks queries

* Now allowing to restore users in a sharded environment as well

It is still not possible to restore collections that are sharded

differently than by _key.

* fixed an issue with restoring of system collections and user rights.

It was not possible to restore users into an authenticated server.

* fixed issue #2977: Documentation for db._createDatabase is wrong

* ui: added bind parameters to slow query history view

* fixed issue #1751: Slow Query API should provide bind parameters, webui should display them

* ui: fixed a bug when moving multiple documents was not possible

* fixed docs for issue #2968: Collection _key autoincrement value increases on error

* AQL CHAR_LENGTH(null) returns now 0. Since AQL TO_STRING(null) is '' (string of length 0)

* ui: now supports single js file upload for Foxx services in addition to zip files

* fixed a multi-threading issue in the agency when callElection was called

while the Supervision was calling updateSnapshot

* added startup option `--query.tracking-with-bindvars`

This option controls whether the list of currently running queries

and the list of slow queries should contain the bind variables used

in the queries or not.

The option can be changed at runtime using the commands

// enables tracking of bind variables

// set to false to turn tracking of bind variables off

var value = true;

require("@arangodb/aql/queries").properties({

trackBindVars: value

});

* index selectivity estimates are now available in the cluster as well

* fixed issue #2943: loadIndexesIntoMemory not returning the same structure

as the rest of the collection APIs

* fixed issue #2949: ArangoError 1208: illegal name

* fixed issue #2874: Collection properties do not return `isVolatile`

attribute

* potential fix for issue #2939: Segmentation fault when starting

coordinator node

* fixed issue #2810: out of memory error when running UPDATE/REPLACE

on medium-size collection

* fix potential deadlock errors in collector thread

* disallow the usage of volatile collections in the RocksDB engine

by throwing an error when a collection is created with attribute

`isVolatile` set to `true`.

Volatile collections are unsupported by the RocksDB engine, so

creating them should not succeed and silently create a non-volatile

collection

* prevent V8 from issuing SIGILL instructions when it runs out of memory

Now arangod will attempt to log a FATAL error into its logfile in case V8

runs out of memory. In case V8 runs out of memory, it will still terminate the

entire process. But at least there should be something in the ArangoDB logs

indicating what the problem was. Apart from that, the arangod process should

now be exited with SIGABRT rather than SIGILL as it shouldn't return into the

V8 code that aborted the process with `__builtin_trap`.

this potentially fixes issue #2920: DBServer crashing automatically post upgrade to 3.2

* Foxx queues and tasks now ensure that the scripts in them run with the same

permissions as the Foxx code who started the task / queue

* fixed issue #2928: Offset problems

* fixed issue #2876: wrong skiplist index usage in edge collection

* fixed issue #2868: cname missing from logger-follow results in rocksdb

* fixed issue #2889: Traversal query using incorrect collection id

* fixed issue #2884: AQL traversal uniqueness constraints "propagating" to other traversals? Weird results

* arangoexport: added `--query` option for passing an AQL query to export the result

* fixed issue #2879: No result when querying for the last record of a query

* ui: allows now to edit default access level for collections in database

_system for all users except the root user.

* The _users collection is no longer accessible outside the arngod process, _queues is always read-only

* added new option "--rocksdb.max-background-jobs"

* removed options "--rocksdb.max-background-compactions", "--rocksdb.base-background-compactions" and "--rocksdb.max-background-flushes"

* option "--rocksdb.compaction-read-ahead-size" now defaults to 2MB

* change Windows build so that RocksDB doesn't enforce AVX optimizations by default

This fixes startup crashes on servers that do not have AVX CPU extensions

* speed up RocksDB secondary index creation and dropping

* removed RocksDB note in Geo index docs

v3.2.0 (2017-07-20)

-------------------

* fixed UI issues

* fixed multi-threading issues in Pregel

* fixed Foxx resilience

* added command-line option `--javascript.allow-admin-execute`

This option can be used to control whether user-defined JavaScript code

is allowed to be executed on server by sending via HTTP to the API endpoint

`/_admin/execute` with an authenticated user account.

The default value is `false`, which disables the execution of user-defined

code. This is also the recommended setting for production. In test environments,

it may be convenient to turn the option on in order to send arbitrary setup

or teardown commands for execution on the server.

v3.2.beta6 (2017-07-18)

-----------------------

* various bugfixes

v3.2.beta5 (2017-07-16)

-----------------------

* numerous bugfixes

v3.2.beta4 (2017-07-04)

-----------------------

* ui: fixed document view _from and _to linking issue for special characters

* added function `db._parse(query)` for parsing an AQL query and returning information about it

* fixed one medium priority and two low priority security user interface

issues found by owasp zap.

* ui: added index deduplicate options

* ui: fixed renaming of collections for the rocksdb storage engine

* documentation and js fixes for secondaries

* RocksDB storage format was changed, users of the previous beta/alpha versions

must delete the database directory and re-import their data

* enabled permissions on database and collection level

* added and changed some user related REST APIs

* added `PUT /_api/user/{user}/database/{database}/{collection}` to change collection permission

* added `GET /_api/user/{user}/database/{database}/{collection}`

* added optional `full` parameter to the `GET /_api/user/{user}/database/` REST call

* added user functions in the arangoshell `@arangodb/users` module

* added `grantCollection` and `revokeCollection` functions

* added `permission(user, database, collection)` to retrieve collection specific rights

* added "deduplicate" attribute for array indexes, which controls whether inserting

duplicate index values from the same document into a unique array index will lead to

an error or not:

// with deduplicate = true, which is the default value:

db._create("test");

db.test.ensureIndex({ type: "hash", fields: ["tags[*]"], deduplicate: true });

db.test.insert({ tags: ["a", "b"] });

db.test.insert({ tags: ["c", "d", "c"] }); // will work, because deduplicate = true

db.test.insert({ tags: ["a"] }); // will fail

// with deduplicate = false

db._create("test");

db.test.ensureIndex({ type: "hash", fields: ["tags[*]"], deduplicate: false });

db.test.insert({ tags: ["a", "b"] });

db.test.insert({ tags: ["c", "d", "c"] }); // will not work, because deduplicate = false

db.test.insert({ tags: ["a"] }); // will fail

The "deduplicate" attribute is now also accepted by the index creation HTTP

API endpoint POST /_api/index and is returned by GET /_api/index.

* added optimizer rule "remove-filters-covered-by-traversal"

* Debian/Ubuntu installer: make messages about future package upgrades more clear

* fix a hangup in VST

The problem happened when the two first chunks of a VST message arrived

together on a connection that was newly switched to VST.

* fix deletion of outdated WAL files in RocksDB engine

* make use of selectivity estimates in hash, skiplist and persistent indexes

in RocksDB engine

* changed VM overcommit recommendation for user-friendliness

* fix a shutdown bug in the cluster: a destroyed query could still be active

* do not terminate the entire server process if a temp file cannot be created

(Windows only)

* fix log output in the front-end, it stopped in case of too many messages

v3.2.beta3 (2017-06-27)

-----------------------

* numerous bugfixes

v3.2.beta2 (2017-06-20)

-----------------------

* potentially fixed issue #2559: Duplicate _key generated on insertion

* fix invalid results (too many) when a skipping LIMIT was used for a

traversal. `LIMIT x` or `LIMIT 0, x` were not affected, but `LIMIT s, x`

may have returned too many results

* fix races in SSL communication code

* fix invalid locking in JWT authentication cache, which could have

crashed the server

* fix invalid first group results for sorted AQL COLLECT when LIMIT

was used

* fix potential race, which could make arangod hang on startup

* removed `exception` field from transaction error result; users should throw

explicit `Error` instances to return custom exceptions (addresses issue #2561)

* fixed issue #2613: Reduce log level when Foxx manager tries to self heal missing database

* add a read only mode for users and collection level authorization

* removed `exception` field from transaction error result; users should throw

explicit `Error` instances to return custom exceptions (addresses issue #2561)

* fixed issue #2677: Foxx disabling development mode creates non-deterministic service bundle

* fixed issue #2684: Legacy service UI not working

v3.2.beta1 (2017-06-12)

-----------------------

* provide more context for index errors (addresses issue #342)

* arangod now validates several OS/environment settings on startup and warns if

the settings are non-ideal. Most of the checks are executed on Linux systems only.

* fixed issue #2515: The replace-or-with-in optimization rule might prevent use of indexes

* added `REGEX_REPLACE` AQL function

* the RocksDB storage format was changed, users of the previous alpha versions

must delete the database directory and re-import their data

* added server startup option `--query.fail-on-warning`

setting this option to `true` will abort any AQL query with an exception if

it causes a warning at runtime. The value can be overridden per query by

setting the `failOnWarning` attribute in a query's options.

* added --rocksdb.num-uncompressed-levels to adjust number of non-compressed levels

* added checks for memory managment and warn (i. e. if hugepages are enabled)

* set default SSL cipher suite string to "HIGH:!EXPORT:!aNULL@STRENGTH"

* fixed issue #2469: Authentication = true does not protect foxx-routes

* fixed issue #2459: compile success but can not run with rocksdb

* `--server.maximal-queue-size` is now an absolute maximum. If the queue is

full, then 503 is returned. Setting it to 0 means "no limit".

* (Enterprise only) added authentication against an LDAP server

* fixed issue #2083: Foxx services aren't distributed to all coordinators

* fixed issue #2384: new coordinators don't pick up existing Foxx services

* fixed issue #2408: Foxx service validation causes unintended side-effects

* extended HTTP API with routes for managing Foxx services

* added distinction between hasUser and authorized within Foxx

(cluster internal requests are authorized requests but don't have a user)

* arangoimp now has a `--threads` option to enable parallel imports of data

* PR #2514: Foxx services that can't be fixed by self-healing now serve a 503 error

* added `time` function to `@arangodb` module

v3.2.alpha4 (2017-04-25)

------------------------

* fixed issue #2450: Bad optimization plan on simple query

* fixed issue #2448: ArangoDB Web UI takes no action when Delete button is clicked

* fixed issue #2442: Frontend shows already deleted databases during login

* added 'x-content-type-options: nosniff' to avoid MSIE bug

* set default value for `--ssl.protocol` from TLSv1 to TLSv1.2.

* AQL breaking change in cluster:

The SHORTEST_PATH statement using edge-collection names instead

of a graph name now requires to explicitly name the vertex-collection names

within the AQL query in the cluster. It can be done by adding `WITH <name>`

at the beginning of the query.

Example:

```

FOR v,e IN OUTBOUND SHORTEST_PATH @start TO @target edges [...]

```

Now has to be:

```

WITH vertices

FOR v,e IN OUTBOUND SHORTEST_PATH @start TO @target edges [...]

```

This change is due to avoid dead-lock sitations in clustered case.

An error stating the above is included.

* add implicit use of geo indexes when using SORT/FILTER in AQL, without

the need to use the special-purpose geo AQL functions `NEAR` or `WITHIN`.

the special purpose `NEAR` AQL function can now be substituted with the

following AQL (provided there is a geo index present on the `doc.latitude`

and `doc.longitude` attributes):

FOR doc in geoSort

SORT DISTANCE(doc.latitude, doc.longitude, 0, 0)

LIMIT 5

RETURN doc

`WITHIN` can be substituted with the following AQL:

FOR doc in geoFilter

FILTER DISTANCE(doc.latitude, doc.longitude, 0, 0) < 2000

RETURN doc

Compared to using the special purpose AQL functions this approach has the

advantage that it is more composable, and will also honor any `LIMIT` values

used in the AQL query.

* potential fix for shutdown hangs on OSX

* added KB, MB, GB prefix for integer parameters, % for integer parameters

with a base value

* added JEMALLOC 4.5.0

* added `--vm.resident-limit` and `--vm.path` for file-backed memory mapping

after reaching a configurable maximum RAM size

* try recommended limit for file descriptors in case of unlimited

hard limit

* issue #2413: improve logging in case of lock timeout and deadlocks

* added log topic attribute to /_admin/log api

* removed internal build option `USE_DEV_TIMERS`

Enabling this option activated some proprietary timers for only selected

events in arangod. Instead better use `perf` to gather timings.

v3.2.alpha3 (2017-03-22)

------------------------

* increase default collection lock timeout from 30 to 900 seconds

* added function `db._engine()` for retrieval of storage engine information at

server runtime

There is also an HTTP REST handler at GET /_api/engine that returns engine

information.

* require at least cmake 3.2 for building ArangoDB

* make arangod start with less V8 JavaScript contexts

This speeds up the server start (a little bit) and makes it use less memory.

Whenever a V8 context is needed by a Foxx action or some other operation and

there is no usable V8 context, a new one will be created dynamically now.

Up to `--javascript.v8-contexts` V8 contexts will be created, so this option

will change its meaning. Previously as many V8 contexts as specified by this

option were created at server start, and the number of V8 contexts did not

change at runtime. Now up to this number of V8 contexts will be in use at the

same time, but the actual number of V8 contexts is dynamic.

The garbage collector thread will automatically delete unused V8 contexts after

a while. The number of spare contexts will go down to as few as configured in

the new option `--javascript.v8-contexts-minimum`. Actually that many V8 contexts

are also created at server start.

The first few requests in new V8 contexts will take longer than in contexts

that have been there already. Performance may therefore suffer a bit for the

initial requests sent to ArangoDB or when there are only few but performance-

critical situations in which new V8 contexts will be created. If this is a

concern, it can easily be fixed by setting `--javascipt.v8-contexts-minimum`

and `--javascript.v8-contexts` to a relatively high value, which will guarantee

that many number of V8 contexts to be created at startup and kept around even

when unused.

Waiting for an unused V8 context will now also abort if no V8 context can be

acquired/created after 120 seconds.

* improved diagnostic messages written to logfiles by supervisor process

* fixed issue #2367

* added "bindVars" to attributes of currently running and slow queries

* added "jsonl" as input file type for arangoimp

* upgraded version of bundled zlib library from 1.2.8 to 1.2.11

* added input file type `auto` for arangoimp so it can automatically detect the

type of the input file from the filename extension

* fixed variables parsing in GraphQL

* added `--translate` option for arangoimp to translate attribute names from

the input files to attriubte names expected by ArangoDB

The `--translate` option can be specified multiple times (once per translation

to be executed). The following example renames the "id" column from the input

file to "_key", and the "from" column to "_from", and the "to" column to "_to":

arangoimp --type csv --file data.csv --translate "id=_key" --translate "from=_from" --translate "to=_to"

`--translate` works for CSV and TSV inputs only.

* changed default value for `--server.max-packet-size` from 128 MB to 256 MB

* fixed issue #2350

* fixed issue #2349

* fixed issue #2346

* fixed issue #2342

* change default string truncation length from 80 characters to 256 characters for

`print`/`printShell` functions in ArangoShell and arangod. This will emit longer

prefixes of string values before truncating them with `...`, which is helpful

for debugging.

* always validate incoming JSON HTTP requests for duplicate attribute names

Incoming JSON data with duplicate attribute names will now be rejected as

invalid. Previous versions of ArangoDB only validated the uniqueness of

attribute names inside incoming JSON for some API endpoints, but not

consistently for all APIs.