Skip to content

Commit

Permalink
CATTRAINER-44: Use type_ids, over tuis
Browse files Browse the repository at this point in the history
  • Loading branch information
@tomolopolis
tomolopolis committed Oct 22, 2021
1 parent 6930f71 commit e16dc87
Show file tree
Hide file tree
Showing 14 changed files with 134 additions and 65 deletions.
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,8 @@ capable of annotating in via an interactive model
1\. A form to:
- Select the appropriate project model to view concept annotations for.
- Clinical text to annotate and display in 2.
- CUI and TUI filters can be used to only show concepts of interest in 2. For example for a UMLS CDB this could be
T047 for "Disease or Syndrome". CUI and TUI filters are combined if entries are included in both form inputs.
- CUI and TypeID filters can be used to only show concepts of interest in 2. For example for a UMLS CDB this could be
T047 for "Disease or Syndrome". CUI and TypeID filters are combined if entries are included in both form inputs.

2\. Example clinical text is displayed here, with text spans highlighted in blue. Click any annotation to show linked
concept DB details in 3.
Expand Down Expand Up @@ -277,7 +277,7 @@ Lists the current selected concepts details.
|--------------| ------------|
|Annotated Text| The text span linked to the concept|
|Name | The linked concept name from within the MedCAT CDB|
|Term ID | The higher level group of concepts that this concept sits under. This may be 'N/A' depending if you CDB is complete with TUIs.|
|Type ID | The higher level group of concepts that this concept sits under. This may be 'N/A' depending if your CDB has Type IDs or not.|
|Concept ID | The unique identifier for this linked concept from the MedCAT CDB.|
|Accuracy | The MedCAT found accuracy of the linked concept for this span. Text spans will have an accuracy 1.0, if they are uniquely identified by that name in the CDB|
|Description | The MedCAT associated description of the concept. SNOMED-CT does not provide descriptions of concepts, only alternative names whereas UMLS does provide descriptions|
Expand Down
119 changes: 85 additions & 34 deletions docs/API_Examples.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
{
"data": {
"text/plain": [
"{'Authorization': 'Token 46b67a3292f3b2d1253b0008ae5c64d966082b0d'}"
"{'Authorization': 'Token 0330dc5c1ec5eb6512b9824fd99650b834900082'}"
]
},
"execution_count": 4,
Expand All @@ -95,29 +95,32 @@
},
{
"cell_type": "code",
"execution_count": 62,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'users': 'http://localhost/api/users/',\n",
" 'concepts': 'http://localhost/api/concepts/',\n",
" 'entities': 'http://localhost/api/entities/',\n",
" 'project-annotate-entities': 'http://localhost/api/project-annotate-entities/',\n",
" 'documents': 'http://localhost/api/documents/',\n",
" 'annotated-entities': 'http://localhost/api/annotated-entities/',\n",
" 'meta-annotations': 'http://localhost/api/meta-annotations/',\n",
" 'meta-tasks': 'http://localhost/api/meta-tasks/',\n",
" 'meta-task-values': 'http://localhost/api/meta-task-values/',\n",
" 'concept-dbs': 'http://localhost/api/concept-dbs/',\n",
" 'vocabs': 'http://localhost/api/vocabs/',\n",
" 'datasets': 'http://localhost/api/datasets/',\n",
" 'icd-codes': 'http://localhost/api/icd-codes/',\n",
" 'opcs-codes': 'http://localhost/api/opcs-codes/'}"
"{'users': 'http://localhost:8001/api/users/',\n",
" 'concepts': 'http://localhost:8001/api/concepts/',\n",
" 'entities': 'http://localhost:8001/api/entities/',\n",
" 'project-annotate-entities': 'http://localhost:8001/api/project-annotate-entities/',\n",
" 'documents': 'http://localhost:8001/api/documents/',\n",
" 'annotated-entities': 'http://localhost:8001/api/annotated-entities/',\n",
" 'meta-annotations': 'http://localhost:8001/api/meta-annotations/',\n",
" 'meta-tasks': 'http://localhost:8001/api/meta-tasks/',\n",
" 'meta-task-values': 'http://localhost:8001/api/meta-task-values/',\n",
" 'relations': 'http://localhost:8001/api/relations/',\n",
" 'entity-relations': 'http://localhost:8001/api/entity-relations/',\n",
" 'concept-dbs': 'http://localhost:8001/api/concept-dbs/',\n",
" 'vocabs': 'http://localhost:8001/api/vocabs/',\n",
" 'datasets': 'http://localhost:8001/api/datasets/',\n",
" 'icd-codes': 'http://localhost:8001/api/icd-codes/',\n",
" 'opcs-codes': 'http://localhost:8001/api/opcs-codes/',\n",
" 'upload-deployment': 'http://localhost:8001/api/upload-deployment/'}"
]
},
"execution_count": 62,
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -141,9 +144,66 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 6,
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Subject 7</td>\n",
" <td>EXAM:,MRI LEFT KNEE WITHOUT CONTRAST,CLINICAL:...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Subject 7</td>\n",
" <td>REASON FOR CONSULTATION: , Left hip fracture.,...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Subject 7</td>\n",
" <td>REASON FOR CONSULTATION: , Left hip fracture.,...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name text\n",
"0 Subject 7 EXAM:,MRI LEFT KNEE WITHOUT CONTRAST,CLINICAL:...\n",
"1 Subject 7 REASON FOR CONSULTATION: , Left hip fracture.,...\n",
"2 Subject 7 REASON FOR CONSULTATION: , Left hip fracture.,..."
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Add a name column to the other datasets\n",
"ortho_notes['name'] = ortho_notes.subject_id.apply(lambda l: f'Subject {l}')\n",
Expand All @@ -154,7 +214,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -173,8 +233,8 @@
"for name, d_s in datasets:\n",
" payload = {\n",
" 'dataset_name': name, # Name that appears in each\n",
" 'dataset': dataset.loc[:, ['name', 'text']].to_dict(), # Dictionary representation of only \n",
" 'description': f'{d_name} first 20 notes from each category' # Description that appears in the trainer\n",
" 'dataset': d_s.loc[:, ['name', 'text']].to_dict(), # Dictionary representation of only \n",
" 'description': f'{name} first 20 notes from each category' # Description that appears in the trainer\n",
" }\n",
" resp = requests.post(f'{URL}/api/create-dataset/', json=payload, headers=headers)\n",
" dataset_ids.append(json.loads(resp.text)['dataset_id']) \n",
Expand Down Expand Up @@ -313,7 +373,7 @@
"|name|# Name of the project that appears on the landing page|\n",
"|description| Example projects', # Description as it appears on the landing page|\n",
"|cuis | Comma separated list if needed |\n",
"|tuis | A comma separated list of TUIs. TUIs are logical groupings of CUIs such as 'disease', or 'symptom'|\n",
"|type_ids | A comma separated list of Type IDs. Type IDs are logical groupings of CUIs such as 'disease', or 'symptom'|\n",
"|dataset | The set of documents to be annotated|\n",
"|concept_db | Previously retrieved CDB ID |\n",
"|cdb_search_filter|**list** of CDB IDs that are used to lookup concepts during addition of annotations to a document| \n",
Expand Down Expand Up @@ -377,18 +437,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.1"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}
6 changes: 3 additions & 3 deletions webapp/api/api/admin.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -667,9 +667,9 @@ def _import_concepts(id):
concept = Concept()
concept.pretty_name = cdb.cui2preferred_name.get(cui, cui)
concept.cui = cui
concept.tui = ','.join(list(cdb.cui2type_ids.get(cui, '')))
concept.semantic_type = ','.join([cdb.addl_info['type_id2name'].get(tui, '')
for tui in list(cdb.cui2type_ids.get(cui, ''))])
concept.type_ids = ','.join(list(cdb.cui2type_ids.get(cui, '')))
concept.semantic_type = ','.join([cdb.addl_info['type_id2name'].get(type_id, '')
for type_id in list(cdb.cui2type_ids.get(cui, ''))])
concept.desc = cdb.addl_info['cui2description'].get(cui, '')
concept.synonyms = ", ".join(cdb.addl_info['cui2original_names'].get(cui, []))
concept.cdb = concept_db
Expand Down
18 changes: 18 additions & 0 deletions webapp/api/api/migrations/0060_auto_20211022_0940.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Generated by Django 2.2.24 on 2021-10-22 09:40

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
('api', '0059_auto_20211008_2038'),
]

operations = [
migrations.RenameField(
model_name='concept',
old_name='tui',
new_name='type_ids',
),
]
2 changes: 1 addition & 1 deletion webapp/api/api/models.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ class Concept(models.Model):
pretty_name = models.CharField(max_length=300, db_index=True)
cui = models.CharField(max_length=100, db_index=True, unique=True)
desc = models.TextField(default="", blank=True)
tui = models.CharField(max_length=20)
type_ids = models.CharField(max_length=20)
semantic_type = models.CharField(max_length=200, blank=True, null=True)
synonyms = models.TextField(default='', blank=True)
icd10 = models.ManyToManyField(ICDCode, default=None, blank=True, related_name='concept')
Expand Down
2 changes: 1 addition & 1 deletion webapp/api/api/serializers.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ class Meta:
class ConceptSerializer(serializers.ModelSerializer):
class Meta:
model = Concept
fields = ['pretty_name', 'cui', 'desc', 'tui', 'synonyms', 'semantic_type',
fields = ['pretty_name', 'cui', 'desc', 'type_ids', 'synonyms', 'semantic_type',
'icd10', 'opcs4', 'id', 'cdb']

def to_representation(self, instance):
Expand Down
6 changes: 3 additions & 3 deletions webapp/api/api/utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def check_ents(ent):

for ent in ents:
label = ent._.cui
tuis = list(cat.cdb.cui2type_ids.get(label, ''))
type_ids = list(cat.cdb.cui2type_ids.get(label, ''))

# Add the concept info to the Concept table if it doesn't exist
cnt = Concept.objects.filter(cui=label).count()
Expand All @@ -71,8 +71,8 @@ def check_ents(ent):
concept = Concept()
concept.pretty_name = pretty_name
concept.cui = label
concept.tui = ','.join(tuis)
concept.semantic_type = ','.join([cat.cdb.addl_info['type_id2name'].get(tui, '') for tui in tuis])
concept.type_ids = ','.join(type_ids)
concept.semantic_type = ','.join([cat.cdb.addl_info['type_id2name'].get(type_id, '') for type_id in type_ids])
concept.desc = cat.cdb.addl_info['cui2description'].get(label, '')
concept.synonyms = ",".join(cat.cdb.addl_info['cui2original_names'].get(label, []))
concept.cdb = project.concept_db
Expand Down
8 changes: 4 additions & 4 deletions webapp/api/api/views.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,14 +127,14 @@ class DocumentViewSet(viewsets.ModelViewSet):


class ConceptFilter(drf.FilterSet):
tui__in = TextInFilter(field_name='tui', lookup_expr='in')
type_ids__in = TextInFilter(field_name='type_ids', lookup_expr='in')
cui__in = TextInFilter(field_name='cui', lookup_expr='in')
cdb__in = NumInFilter(field_name='cdb', lookup_expr='in')
id__in = NumInFilter(field_name='id', lookup_expr='in')

class Meta:
model = Concept
fields = ['tui', 'cui', 'cdb']
fields = ['type_ids', 'cui', 'cdb']


class ConceptViewSet(viewsets.ModelViewSet):
Expand Down Expand Up @@ -388,7 +388,7 @@ def add_concept(request):
# TODO These aren't used, but no API in current MedCAT add_name func
# Add these fields to the add_name func of MedCAT add_name
desc = request.data['desc']
tui = request.data['tui']
type_ids = request.data['type_ids']
s_type = request.data['type']
synonyms = request.data['synonyms']

Expand Down Expand Up @@ -426,7 +426,7 @@ def add_concept(request):
c.cui = cui
c.pretty_name = name
c.desc = desc
c.tui = tui
c.type_ids = type_ids
c.synonyms = synonyms
c.semantic_type = s_type
c.cdb = project.concept_db
Expand Down
4 changes: 2 additions & 2 deletions webapp/frontend/src/components/anns/AddAnnotation.vue
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@
<td>Name</td>
<td class="cui-mappings">{{selectedCUI.name.split(':')[0] || 'n/a'}}</td>
</tr>
<tr v-if="selectedCUI.tui !== 'unk' ">
<tr v-if="selectedCUI.type_ids !== 'unk' ">
<td>Type ID</td>
<td>{{selectedCUI.tui || 'n/a'}}</td>
<td>{{selectedCUI.type_ids || 'n/a'}}</td>
</tr>
<tr v-if="selectedCUI.semantic_type">
<td>Semantic Type</td>
Expand Down
6 changes: 3 additions & 3 deletions webapp/frontend/src/components/common/AddNewConcept.vue
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@
<td><input v-model="concept.type" type="text" class="form-control"></td>
</tr>
<tr v-if="showAdd">
<td>Term ID</td>
<td><input v-model="concept.tui" type="text" class="form-control"></td>
<td>Type IDs</td>
<td><input v-model="concept.type_ids" type="text" class="form-control"></td>
</tr>
<tr v-if="showAdd">
<td>Synonyms</td>
Expand Down Expand Up @@ -63,7 +63,7 @@ export default {
name: '',
desc: '',
type: '',
tui: '',
type_ids: '',
cui: '',
synonyms: ''
},
Expand Down
4 changes: 2 additions & 2 deletions webapp/frontend/src/components/common/ConceptPicker.vue
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ export default {
return {
name: isDupName ? `${r.pretty_name} : ${r.cui}` : r.pretty_name,
cui: r.cui,
tui: r.tui,
type_ids: r.type_ids,
desc: r.desc,
icd10: r.icd10,
opcs4: r.opcs4,
Expand All @@ -80,7 +80,7 @@ export default {
name: `${cuiInfo.code} | ${cuiInfo.desc}\n${concept.cui} | ${concept.pretty_name}`,
orderKey: cuiInfo.code,
cui: concept.cui,
tui: concept.tui,
type_ids: concept.type_ids,
desc: concept.desc,
icd10: concept.icd10,
opcs4: concept.opcs4,
Expand Down
10 changes: 5 additions & 5 deletions webapp/frontend/src/components/common/ConceptSummary.vue
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
</span>
</td>
</tr>
<tr v-if="conceptSummary['Term ID'] !== 'unk' ">
<td>Term ID</td>
<td>{{conceptSummary['Term ID'] || 'n/a'}}</td>
<tr v-if="conceptSummary['Type IDs'] !== 'unk' ">
<td>Type IDs</td>
<td>{{conceptSummary['Type IDs'] || 'n/a'}}</td>
</tr>
<tr v-if="conceptSummary['Semantic Type']">
<td>Semantic Type</td>
Expand Down Expand Up @@ -81,7 +81,7 @@ const HIDDEN_PROPS = [
const PROP_MAP = {
'acc': 'Accuracy',
'desc': 'Description',
'tui': 'Term ID',
'type_ids': 'Type IDs',
'semantic_type': 'Type',
'cui': 'Concept ID',
'icd10': 'ICD-10',
Expand All @@ -90,7 +90,7 @@ const PROP_MAP = {
}
const CONST_PROPS_ORDER = [
'Name', 'Description', 'Type', 'Term ID', 'Concept ID', 'ICD-10', 'OPCS-4', 'Accuracy'
'Name', 'Description', 'Type', 'Type IDs', 'Concept ID', 'ICD-10', 'OPCS-4', 'Accuracy'
]
export default {
Expand Down
2 changes: 1 addition & 1 deletion webapp/frontend/src/mixins/ConceptDetailService.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ export default {
this.$http.get(`/api/concepts/?cui=${selectedEnt.cui}`).then(resp => {
if (selectedEnt && resp.data.results.length > 0) {
selectedEnt.desc = resp.data.results[0].desc
selectedEnt.tui = resp.data.results[0].tui
selectedEnt.type_ids = resp.data.results[0].type_ids
selectedEnt.pretty_name = resp.data.results[0].pretty_name
selectedEnt.semantic_type = resp.data.results[0].semantic_type
if (resp.data.results[0].icd10.length > 0) {
Expand Down
Loading

0 comments on commit e16dc87

Please sign in to comment.