MedCATTrainer is an interface for building, improving and customising a given Named Entity Recognition and Linking (NER+L) model (MedCAT) for biomedical domain text.

MedCATTrainer was presented at EMNLP/IJCNLP 2019 🎉 here

1. Installation
2. Admin Setup
3. User Guide
   1. Create An Annotation Project
      1. Notes
   2. Annotation Interface
      1. Section 1 - Document Summary List
      2. Section 2 - Clinical Text
         1. Additional Annotations
      3. Section 3 - Action Bar
         1. Concept Navigation Buttons
         2. Concept Status Buttons
         3. Submit Button
      4. Section 4 - Header Toolbar
      5. Section 5 - Concept Summary
   3. Meta-Annotations
      1. Meta Annotation Configuration
   4. Project & Tool Administration
      1. Configuring Concept Picker
      2. Downloading Annotations
      3. Cloning Projects
      4. Resetting Projects
      5. Save Models
      6. Download Models
4. Annotation Guidelines
5. Advanced Usage

MedCATtrainer is a docker-compose packaged Django application.

Quick-install script, downloads the docker-compose file, default env var and runs them. 1.

$ git clone https://github.com/CogStack/MedCATtrainer
$ cd MedCATtrainer
$ docker-compose up

If you have MedCAT v0.x models, and want to use the trainer please use the following docker-compose file: This refences the latest built image for the trainer that is still compatible with MedCAT v0.x. and under.

$ docker-compose -f docker-compose-mc0x.yml up

If the build fails with an error code 137, the virtual machine running the docker daemon does not have enough memory. Increase the allocated memory to containers in the docker daemon settings CLI or associated docker GUI.

On MAC: https://docs.docker.com/docker-for-mac/#memory

On Windows: https://docs.docker.com/docker-for-windows/#resources

2. MedCATTrainer is now running:

• The main app is available at http://localhost:8001/
• The administrator (admin) app is available at http://localhost:8001/admin/

Upon initial install, an example administrator user, example dataset, concept database, vocab and project are setup. The next section provides further details.

3. (Optional), Environment variables are used to configure the app:

Set these a re-run the install script with:

$ curl https://cdn.githubraw.com/CogStack/MedCATtrainer/f9f753a8/install.sh | bash -E

You'll need to docker stop / rm the running containers if you have already run the install.

1. The container runs a vanilla django app, that upon initially loaded will create a defaulted administrator user with details:

username: admin
password: admin

2. We strongly recommend creating a new admin user before using the trainer in 'production' and storing sensitive clinical documents on the trainer. To add a new user navigate to select http://localhost:8001/admin/ and select 'Users'.

3. Select 'Add User' and complete the form with a new username / password.

4. Once created, select the new user, and tick the 'Staff Status' or 'Superuser Status' to allow the user to access the admin app.

5. Remove the default admin user by navigating to step 2, select the user and the action

For demonstration purposes and general testing of a current model a stripped back version of the annotator is provided via the 'Demo' tab of the main screen.

This presents a similar looking annotation screen as a real project, described in detail here, but does not force usage of a particular dataset, setup of filters, and other project settings. This view does not allow for 'annotating' identified concepts (or adding new concepts) but allows for users to get a feel for what an existing MedCAT model is capable of annotating in via an interactive model

1. A form to:

• Select the appropriate project model to view concept annotations for.
• Clinical text to annotate and display in 2.
• CUI and TUI filters can be used to only show concepts of interest in 2. For example for a UMLS CDB this could be T047 for "Disease or Syndrome". CUI and TUI filters are combined if entries are included in both form inputs.

2. Example clinical text is displayed here, with text spans highlighted in blue. Click any annotation to show linked concept DB details in 3.

3. Linked concept details from selected concepts from the 2.

Using the admin page, a configured superuser can create, edit and delete annotation projects.

Annotation projects are used to inspect, validate and improve concepts recognised by MedCAT.

1. Navigate to http://localhost:8001/admin/ and select 'Project annotate entities'.

2. 'Add Project Annotate Entities'

3. Complete the new annotation project form. The table below provides details the purpose of each field:

Datasets can be uploaded in CSV or XLSX format. Example:

The name column is optional, and will be auto-generated for each document if not supplied in the upload. Example datasets are supplied under docs/example_data/*.csv

4. Click 'Save' to store the new project.

5. Navigate to the home screen (http://localhost:8001/admin/), login with your username and password setup previously.

6. select your new project to begin annotating documents

NB. Example Concept and Vocab databses are freely available on MedCAT github. Note. UMLS and SNOMED-CT are licensed products so only these smaller trained concept / vocab databases are made available currently.

More documentation on the creation of UMLS / SNOMED-CT CDBs from respective source data will be released soon.

NNB. Tasks allow for the creation of meta-annotations and their associated set of values an annotator can use. An example 'meta-annotation' could be 'Temporality'. Values could then be 'Past', 'Present', 'Future'.

NB Please NOTE Firefox and IE are currently not supported. Please use Chrome or Safari.

The annotation interface can be split initially into 5 sections.

A list of documents to be completed in this project. Currently selected documents are highlighted in blue left border. Submitted documents are marked with a .

The selected documents text, highlighted with each concept recognised by the configured MedCAT model. Highlighted spans of text indicate status of the annotation:

• Grey: A User has not reviewed this span that has been recognised and linked by MedCAT to a CDB concept.
• Blue: A User has reviewed the span and marked it as correct in terms of its linked MedCAT concept.
• Red: A User has reviewed the span and marked it as incorrect in terms of its linked MedCAT concept.
• Dark Red: A User has reviewed the span and marked it to terminate, meaning the text span should never again link to this text span, this informs MedCAT that
• Turqoise: A User has reviewed the span and marked it as an alternative linked concept. The user has used the 'Concept Picker' to choose the correct concept that should be linked.

MedCAT may miss text spans that are acronyms, abbreviations or misspellings of concepts. Missing annotations can be added to the text by directly highlighting the text span, right clicking, selecting 'Add Annotation', searching for concept (via ID, or name), and selecting Add Synonym:

-> ->

Select:

• Add Synonym: to add this anno§tation to the text span and link the selected concept
• Cancel: (Shortcut esc): to cancel adding the annotation to the text.

Navigating between the list of concepts as they appear in the document:

• Action buttons, left and right
• Left and right arrow keys on keyboard
• Directly clicking on the concept within the text.

A concept can be marked with only one status. Status is recorded but only sent to MedCAT for training on submit of the document and if the projects configured with "Train Model On Submit" is ticked.

Submit is disabled until all concepts have been reviewed and marked with a status. Clicking submit will produce
a submission confirmation dialog with an annotation summary. Confirming submission will send all new annotations to MedCATTrainer middle tier, and re-train the MedCAT model. The following document will be selected and annotated by the newly trained MedCAT model

Lists the current name of the document under review and the number of remaining documents to annotate in this project action buttons for:

• : Summary of current annotations. f A similar view is shown before confirmation of submission of the annotations
• : Help dialog, showing shortcuts for document & concept navigation, concept annotation and submission.
• : Reset document. If an annotation is incorrectly added, or incorrectly submitted resetting the document will clear all previous annotations and their status.

Lists the current selected concepts details.

MedCAT is also able to learn project & context specific annotations that overlay on top of the base layer of concept annotations.

Example use cases of these annotations could be to train models to predict if:

• all disease concepts were experienced by the patient, a relative, or N/A.
• all symptom concepts are temporally reference present day, or are historical.
• all drug concepts are mentions of patients consuming drugs rather than hypothetical mentions.
• a complaint for a patient is primary or secondary.

MedCATTrainer is configurable (via the administrator app), to allow for the collection of these meta annotations. We currently have not integrated the active learning components of the concept recognition.

To create a new Meta Annotation Task and attach to an existing project:

1. Enter your project configuration settings via the admin page (http://localhost:8001/admin/)

2. At the bottom of the form, select the + icon to bring up the new Meta Annotation Task Form.

3. Complete the form and add additional meta task values if required for your task via the '+' icon and the 'values' input. Values are enumerated options for your specific task. These can be re-used across projects or be project specific. Ensure the default is one of the corresponding values available. Descriptions appear alongside the tasks in interface and in full in the help dialog.

4. Select desired Meta Annotation tasks for the project by holding down (ctrl / cmd) and clicking the meta tasks, then select 'Save' to save the project changes.

5. Meta Annotations now appear in the interface for that project under the concept summary. Meta-annotations only appear for concepts that are correct.

The concept picker is used to:

• Pick alternative concepts for an existing recognised span
• Pick a concept during the 'Add Annotation' process.

The available list of concepts is populated via a MedCAT CDB and indexed to enable fast type-ahead style search.

SNOMED-CT / UMLS built databases can contain thousands if not millions of concepts so this process is executed in asynchronous task to ensure the admin page and app are still available for use.

This process should only be done once for each concept universe (i.e. SNOMED-CT, UMLS are 2 distinct concept universes) per deployment or if the underlying MedCAT CDB changes Concepts will be indexed by there CUI, so importing different CDB instances that reference the same concept universe will only import the concepts that are in the set difference.

To make these concepts available to a (or any project):

1. Open the admin app. (http://localhost:8001/admin/)

2. Select 'Concept Dbs'

3. Select the Concept DB entry, and choose the action 'Import Concept', then press the 'Go' button.

Project annotations can be downloaded with or without the source text, especially important if the source text is particularly sensitive and should be not be shared.

1. Open the admin app. (http://localhost:8001/admin/)

2. Select 'Project annotate entities',

3. Select the project(s) to download the annotations for and select the appropriate action for w/ or w/o source text, then press the 'Go' button. This will download all annotations, the meta-annotations (if any) for all projects selected. Annotations

4. An example jupyter notebook is provided under docs/Processing_Annotations.ipynb.

Cloning Projects is a easy & fast method to create copies of configured projects. This includes the dataset, CDB / vocab reference, meta annotation settings etc. Annotations themselves will not be copied across.

1. Open the admin app. (http://localhost:8001/admin/), and select 'Project annotate entities' (same as above for downloading)

2. Select the project(s) to clone, select the 'Clone projects', then press the 'Go' button.

NB: Cloning projects will use the same CDB instance. If you're double annotating datasets to then calculate agreement scores (IIA, Cohen's Kappa etc.) then uncheck "Train Model On Submit" for each of the projects to ensure the model is not trained by each annotator. If you do want 'online training' of the model, use separate instances of the same model. You can directly upload multiple instances of the same CDB file appropriately named to achieve this.

Use with caution. Resetting projects deletes all annotations and resets a project to its state upon initial creation.

1. Open the admin app. (http://localhost:8001/admin/), and select 'Project annotate entities' (same as above for downloading)

2. Select the project(s) to reset, then press the 'Go' button.

We strongly suggest models are not saved within MedCATtrainer then directly used. Instead, we suggest you use the collected annotations from projects to train and test a new MedCAT model.

However, to save the current state of the model you can use:

An API call - <deployment>:<port>/save-models/ that can be used to save the current state of a model. This will overwrite the current CDB file.

Alternatively, login with an 'admin', (i.e. staff or superuser) account and hit the save model button associated with the project.

1. Open the admin app. (http://localhost:8001/admin/), and select 'Concept dbs'.

2. Click the CDB item you would like to download.

4. Click the CDB file, you will be prompted to save down the new CDB file. This file will be of the same format you have used previously, i.e. you've called medcat.cdb.save_dict('').

The saved MedCAT models can be used in any instance a regular MedCAT model may be used. I.e. in a jupyter notebook, part of a web service, or further fine-tuning in another MedCATTrainer instance.

The Trainer currently does not support inspection / training / storage of the meta annotation models. These will be integrated in a forthcoming release.

5. To load the new dictionary use medcat.cdb.load_dict('')

Annotation guidelines can assist guiding annotators when annotating texts for a MedCATTrainer project.

Once an initial guideline has been defined, a pilot project in MedCATTrainer can be used to further refine the guideline.

• ReST API Usage for bulk dataset / project creation: available in: docs/API_Examples.ipynb