Skip to content

Commit

Permalink
feat: simple expressions (molgenis#9238)
Browse files Browse the repository at this point in the history
* chore: add molgenis expressions library

* feat: switch to simple expressions

* fix: update system entity metadata to use simple expressions

* refactor: process review comments

* test: fix expressions in api test data

* doc: update expression documentation

* chore: externalize forms and expressions js

* fix: truthiness of expressions plus mapping of collections

* fix: getEntity may be null

* fix: load umd js before require.js

* doc: update test questionnaire

* fix: put global js inside <head> tags

* fix: fix scheduled job validation regexes

* doc: fix emx expression documentation

* chore: upgrade version to 9.0.0

* doc: replace magma references in docs

* test: test resolve method

* fix(data-explorer): load expressions correctly

* fix: change metadata manager examples to simple expressions

* chore: migration step for the metadata manager localization keys

* doc: fix sample questionnaire

Fixes molgenis#9248

* test: test migration step

* feat: upgrade expressions library

* test: update expressions in test files
  • Loading branch information
fdlk authored Jul 2, 2021
1 parent ae80b27 commit 1b54335
Show file tree
Hide file tree
Showing 133 changed files with 1,000 additions and 706 deletions.
3 changes: 2 additions & 1 deletion docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,13 @@
* [Setup authentication](guide-authentication.md)
* Data management
* [EMX format](guide-emx.md)
* [Using expressions in EMX](guide-expressions.md)
* [Quickly import data](guide-data-quick-import.md)
* [Advanced data import](guide-data-import.md)
* [Modify metadata](guide-data-metadata.md)
* [Questionnaires](guide-questionnaire.md)
* [Downloading data](guide-data-download.md)
* [Using expressions](guide-expressions.md)
* [MagmaScript expressions (mapping service)](guide-magma.md)
* Access control
* [Users](guide-users.md)
* [Groups and roles](guide-groups-roles.md)
Expand Down
Binary file modified docs/data/simple-questionnaire.xlsx
Binary file not shown.
18 changes: 12 additions & 6 deletions docs/guide-data-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,16 +140,22 @@ __Aggregatable__
: When checked, allows MOLGENIS to compute aggregation on the values of this attribute.

__Computed value expression__
: Possible Magma script that allows you to write computed expressions. e.g you have int column A and int column B. For column C you can create
the following expression: (A/B) * 2.
: Computes value for a column based on other columns.

For String type attributes, uses a string
template, e.g.: `{"template": "Hello {{world.label}}"}` which fills in the value of the `label`
attribute of the `world` reference.

For single reference type attributes, creates an instance of the refEntity type, copying
attribute values. E.g.: `{Chromosome: foo, Position: bar}`

__Visible expression__
: Possible Magma script that allows you to write an expression that determines whether the column should be shown or not.
e.g. $('A').value === true will only show column B when the value in A is true. Useful for one line datasets.
: Expression that determines whether the column should be shown or not.
e.g. `{A}` will only show column B when the value in A is truthy. Useful for one line datasets.

__Validation expression__
: Possible magma script that allows you to write an expression that validates the values inside a column.
e.g. $('A').value() > 5 will throw a validation error when you try to add data that is lower then 5
: Expression that validates the values inside a column.
e.g. `{A} > 5` will throw a validation error when you try to add data that is lower than 5

After you have created some attributes, selected an ID attribute, and feel comfortable with your EntityType, you can hit the 'Save all changes' button.
If everything is correct, you will get a message saying save was successful. If something went wrong, you will get a message telling you which fields you forgot.
Expand Down
7 changes: 5 additions & 2 deletions docs/guide-emx.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,8 @@ Used in combination with xref, mref, categorical, categorical_mref or one_to_man

#### nillable
Whether the column may be left empty. Default: true
Can also contain an expression to dynamically decide if the attribute should be shown or not.
See the [Expressions](guide-expressions.md) section for a syntax description.

#### idAttribute
Whether this field is the unique key for the entity. Default: false. Use 'AUTO' for auto generated (string) identifiers.
Expand Down Expand Up @@ -217,10 +219,11 @@ true/false to indicate a readOnly attribute
ability to tag the data referring to the tags sections, described below

#### validationExpression
Magma JavaScript validation expression that must return a bool. Must return true if valid and false if invalid. See the [Expressions](guide-expressions.md) section for a syntax description.
Validation expression that must return a bool. Must return true if valid and false if invalid. See the [Expressions](guide-expressions.md) section for a syntax description.

#### visible
true/false to indicate whether the attribute can be seen by users. Can also contain a Magma JavaScript expression to dynamically decide if the attribute should be shown or not. See the [Expressions](guide-expressions.md) section for a syntax description.
true/false to indicate whether the attribute can be seen by users.
Can also contain an expression to dynamically decide if the attribute should be shown or not. See the [Expressions](guide-expressions.md) section for a syntax description.

#### defaultValue
value that will be filled in in the forms when a new entity instance is created. For mref and categorical_mref, this should be a comma separated list of ids. For categorical and xref this should be the id of the refEntity. For bool should be true or false. For datetime should be a string in the format YYYY-MM-DDTHH:mm:ssZZ. For date should be a string in the format YYYY-MM-DD.
Expand Down
124 changes: 10 additions & 114 deletions docs/guide-expressions.md
Original file line number Diff line number Diff line change
@@ -1,127 +1,23 @@
# Using expressions
# Using simple expressions

## Introduction
For some entity type attributes an expression can be set that determines whether or not a condition is true based on the
attribute values e.g. to determine whether values are valid. The expression format is based on the
[Magma JavaScript API](http://wiki.obiba.org/display/OPALDOC/Magma+Javascript+API).
attribute values to determine whether values are valid, visible or required.

### Examples
The following expression returns true when the value of attribute 'myAttributeName' of an entity only contains
alphanumberic characters:
```js
$('myStringAttributeName').matches(/^[a-z0-9]+$/i).value()
regex('^[a-zA-Z0-9]+$',{myStringAttributeName})
```
In this example:
- '$(...)' is the selector
- 'myStringAttributeName' is an attribute name
- 'matches' is a chaining operation
- 'value' is a terminal operation
- `{myStringAttributeName}` looks up the value of the `myStringAttributeName` variable
- `regex` is a function

Chaining operations allows you to express complex expressions:
```js
$('myIntAttributeName').gt(3).and($('myIntAttributeName').lt(6)).value()
```

## Chaining operations

### Numerical operations
| Operator | Parameters | Example | Description |
|----------|------------|------------------------------------------|---------------|
| plus | Number | `$('height').plus(100).value()` | height + 100 |
| pow | Number | `$('height').pow(100).value()` | height ^ 100 |
| times | Number | `$('height').times(100).value()` | height * 100 |
| div | Number | `$('height').div(100).value()` | height / 100 |
| gt | Number | `$('height').gt(100).value()` | height > 100 |
| lt | Number | `$('height').lt(100).value()` | height < 100 |
| ge | Number | `$('height').ge(100).value()` | height >= 100 |
| le | Number | `$('height').le(100).value()` | height <= 100 |

### Binary operations
| Operator | Parameters | Example | Description |
|----------|------------|------------------------------------------|------------------------|
| eq | Number | `$('height').eq(100).value()` | height === 100 |
| matches | Regex | `$('name').matches(/^[a-z0-9]+$/i).value()`| name is alphanumerical |
| isNull | - | `$('height').isNull().value()` | height === null |
| not | - | `$('hasEars').not().value()` | !hasEars |
| or | Expression | `$('male').or($('female')).value()` | male || female |
| and | Expression | `$('female').and($('pregnant')).value()` | female && pregnant |

### Unit operations
| Operator | Parameters | Example | Description |
|----------|------------|--------------------------------------------|---------------------------------------------------------|
| unit | Unit | `$('height').unit('cm')` | Sets the current value unit to cm |
| toUnit | Unit | `$('height').unit('m').toUnit('cm').value()` | Converts the current value based on the change in units |

### Other
| Operator | Parameters | Example | Description |
|----------|------------|------------------------------------------|-----------------------------------------------------------------------|
| age | - | `$('dateOfBirth').age().value()` | Returns the age based on the date of birth and the current year |
| map | Object | `$('data').map({0:1, 1:2}).value()` | Maps categories to each other |
| group | Array | `$('age').group([18, 35, 50, 75]).value()` | Produces left-inclusive ranges with the given boundaries, i.e. `(-∞, 18), [18, 35), [35, 50), [50, 75), [75, +∞)` and then returns the interval that the attribute value is in, i.el '18-35' |
| attr | String | `$('person').attr('name').value()` | Returns the value of an attribute of a reference

## Terminal operations
| Operator | Parameters | Example | Description |
|----------|------------|---------------------|------------------|
| value | - | `$('height').value()` | JavaScript value |

# Special case: reference types
If an attribute is a reference type (MREF, XREF, CATEGORICAL, CATEGORICAL_MREF) you can use the 'attr' operation, to access the values of different columns in the row being referenced. E.g. `$('cookie').attr('name').value()` gives you the value of the name column inside the table being referenced by the cookie column. See below for a more detailed example.

Imagine __table A__ referencing __table B__.

__Table A__ has 2 columns: id, cookie.
__Table B__ has 3 columns: id, name, tastiness.

The _cookie_ column in __table A__ references __table B__.

__Table A__

| id | cookie |
|----|--------|
| A | 1 |

__Table B__

| id | name | tastiness |
|----|--------|-----------|
| 1 | Chocolate chip | 9/10 |

Expressions allow you to do the following

```js
// Expressions are based on table A

$('cookie').value() // results in '1'
$('cookie').attr('id').value() // results in '1'
$('cookie').attr('name').value() // results in 'Chocolate chip'
$('cookie').attr('tastiness').value() // results in '9/10'
You can combine expressions to perform more complex logic:
```

In the following case, we have Table A which has multiple references to Table B e.g. an MREF

__Table A__

| id | cookie |
|----|--------|
| A | 1,2,3 |

__Table B__

| id | name | tastiness |
|----|--------|-----------|
| 1 | Chocolate chip | 9/10 |
| 2 | Strawberry cookie | 10/10 |
| 3 | Banana cookie | 7/10 |

```js
// Expressions are based on table A

$('cookies').map(function (cookie) {
// Pick one of the following
return cookie.value() // results in ['1', '2', '3']
return cookie.attr('id').value() // results in ['1', '2', '3']
return cookie.attr('name').value() // results in ['Chocolate chip', 'Strawberry cookie', 'Banana cookie']
return cookie.attr('tastiness').value() // results in ['9/10', '10/10', '7/10']
}).value()
{myIntAttributeName} > 3 and {myIntAttributeName} < 6
```
### Reference
See [molgenis/molgenis-expressions](https://github.com/molgenis/molgenis-expressions#readme) for a
complete description of the syntax.
127 changes: 127 additions & 0 deletions docs/guide-magma.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Using MagmaScript expressions

## Introduction
In the mapping service, MagmaScript is used to define the mapping algorithms.
The expression format is based on the
[Magma JavaScript API](http://wiki.obiba.org/display/OPALDOC/Magma+Javascript+API).

### Examples
The following expression returns true when the value of attribute 'myAttributeName' of an entity only contains
alphanumberic characters:
```
regex('^[a-zA-Z0-9]+$',{myStringAttributeName})
```
In this example:
- '$(...)' is the selector
- 'myStringAttributeName' is an attribute name
- 'matches' is a chaining operation
- 'value' is a terminal operation

Chaining operations allows you to express complex expressions:
```js
$('myIntAttributeName').gt(3).and($('myIntAttributeName').lt(6)).value()
```

## Chaining operations

### Numerical operations
| Operator | Parameters | Example | Description |
|----------|------------|------------------------------------------|---------------|
| plus | Number | `$('height').plus(100).value()` | height + 100 |
| pow | Number | `$('height').pow(100).value()` | height ^ 100 |
| times | Number | `$('height').times(100).value()` | height * 100 |
| div | Number | `$('height').div(100).value()` | height / 100 |
| gt | Number | `$('height').gt(100).value()` | height > 100 |
| lt | Number | `$('height').lt(100).value()` | height < 100 |
| ge | Number | `$('height').ge(100).value()` | height >= 100 |
| le | Number | `$('height').le(100).value()` | height <= 100 |

### Binary operations
| Operator | Parameters | Example | Description |
|----------|------------|------------------------------------------|------------------------|
| eq | Number | `$('height').eq(100).value()` | height === 100 |
| matches | Regex | `$('name').matches(/^[a-z0-9]+$/i).value()`| name is alphanumerical |
| isNull | - | `$('height').isNull().value()` | height === null |
| not | - | `$('hasEars').not().value()` | !hasEars |
| or | Expression | `$('male').or($('female')).value()` | male || female |
| and | Expression | `$('female').and($('pregnant')).value()` | female && pregnant |

### Unit operations
| Operator | Parameters | Example | Description |
|----------|------------|--------------------------------------------|---------------------------------------------------------|
| unit | Unit | `$('height').unit('cm')` | Sets the current value unit to cm |
| toUnit | Unit | `$('height').unit('m').toUnit('cm').value()` | Converts the current value based on the change in units |

### Other
| Operator | Parameters | Example | Description |
|----------|------------|------------------------------------------|-----------------------------------------------------------------------|
| age | - | `$('dateOfBirth').age().value()` | Returns the age based on the date of birth and the current year |
| map | Object | `$('data').map({0:1, 1:2}).value()` | Maps categories to each other |
| group | Array | `$('age').group([18, 35, 50, 75]).value()` | Produces left-inclusive ranges with the given boundaries, i.e. `(-∞, 18), [18, 35), [35, 50), [50, 75), [75, +∞)` and then returns the interval that the attribute value is in, i.el '18-35' |
| attr | String | `$('person').attr('name').value()` | Returns the value of an attribute of a reference

## Terminal operations
| Operator | Parameters | Example | Description |
|----------|------------|---------------------|------------------|
| value | - | `$('height').value()` | JavaScript value |

# Special case: reference types
If an attribute is a reference type (MREF, XREF, CATEGORICAL, CATEGORICAL_MREF) you can use the 'attr' operation, to access the values of different columns in the row being referenced. E.g. `$('cookie').attr('name').value()` gives you the value of the name column inside the table being referenced by the cookie column. See below for a more detailed example.

Imagine __table A__ referencing __table B__.

__Table A__ has 2 columns: id, cookie.
__Table B__ has 3 columns: id, name, tastiness.

The _cookie_ column in __table A__ references __table B__.

__Table A__

| id | cookie |
|----|--------|
| A | 1 |

__Table B__

| id | name | tastiness |
|----|--------|-----------|
| 1 | Chocolate chip | 9/10 |

Expressions allow you to do the following

```js
// Expressions are based on table A

$('cookie').value() // results in '1'
$('cookie').attr('id').value() // results in '1'
$('cookie').attr('name').value() // results in 'Chocolate chip'
$('cookie').attr('tastiness').value() // results in '9/10'
```

In the following case, we have Table A which has multiple references to Table B e.g. an MREF

__Table A__

| id | cookie |
|----|--------|
| A | 1,2,3 |

__Table B__

| id | name | tastiness |
|----|--------|-----------|
| 1 | Chocolate chip | 9/10 |
| 2 | Strawberry cookie | 10/10 |
| 3 | Banana cookie | 7/10 |

```js
// Expressions are based on table A

$('cookies').map(function (cookie) {
// Pick one of the following
return cookie.value() // results in ['1', '2', '3']
return cookie.attr('id').value() // results in ['1', '2', '3']
return cookie.attr('name').value() // results in ['Chocolate chip', 'Strawberry cookie', 'Banana cookie']
return cookie.attr('tastiness').value() // results in ['9/10', '10/10', '7/10']
}).value()
```
2 changes: 1 addition & 1 deletion docs/guide-questionnaire.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ questionnaires.

## Visible expressions
The cool feature in the questionnaire module is that you can show certain questions based on the answer given on other questions.
You do this by using the __Javascript Magma syntax__ to create an expression inside the visible column (See driverslicence in the simple questionnaire example).
You do this by creating an expression inside the visible column (See driverslicence in the simple questionnaire example).

## Questionnaire timestamp on submit
MOLGENIS will automatically add a submit date for you when you submit a questionnaire. So keep in mind that you do not have to manually
Expand Down
2 changes: 1 addition & 1 deletion molgenis-amazon-bucket/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<groupId>org.molgenis</groupId>
<artifactId>molgenis</artifactId>
<version>8.8.0-SNAPSHOT</version>
<version>9.0.0-SNAPSHOT</version>
</parent>
<artifactId>molgenis-amazon-bucket</artifactId>
<packaging>jar</packaging>
Expand Down
2 changes: 1 addition & 1 deletion molgenis-api-data/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>molgenis</artifactId>
<groupId>org.molgenis</groupId>
<version>8.8.0-SNAPSHOT</version>
<version>9.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion molgenis-api-fair/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>molgenis</artifactId>
<groupId>org.molgenis</groupId>
<version>8.8.0-SNAPSHOT</version>
<version>9.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
Binary file modified molgenis-api-fair/src/main/resources/FDP.xlsx
Binary file not shown.
2 changes: 1 addition & 1 deletion molgenis-api-files/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>molgenis</artifactId>
<groupId>org.molgenis</groupId>
<version>8.8.0-SNAPSHOT</version>
<version>9.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion molgenis-api-identities/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>molgenis</artifactId>
<groupId>org.molgenis</groupId>
<version>8.8.0-SNAPSHOT</version>
<version>9.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion molgenis-api-metadata/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>molgenis</artifactId>
<groupId>org.molgenis</groupId>
<version>8.8.0-SNAPSHOT</version>
<version>9.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
Loading

0 comments on commit 1b54335

Please sign in to comment.