Updated: 2024-01-12

An attempt at organizing some spaCy workflows, including some functions for disentangling spaCy output as data frames.

• Some spaCy & scispacy wrapper functions
  • Conda environment
  • Reticulate
  • News article corpus
  • Libraries
  • Scispacy components
  • Spacy annotate
  • Extraction functions
    • Standard annotation
    • Entities & linking
    • Abbreviations
    • Noun phrases
    • Hyponyms
    • Negation
    • Sentences
  • References

conda create -n scispacy051 python==3.9

conda activate scispacy051

conda update --all
conda install nmslib pandas numpy
pip install dframcy

pip install scispacy==0.5.1

conda install spacy -c conda-forge
 
pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.1/en_core_sci_sm-0.5.1.tar.gz
pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.1/en_core_sci_md-0.5.1.tar.gz

## <R-console>
library(dplyr)
Sys.setenv(RETICULATE_PYTHON = "/home/jtimm/miniconda3/envs/scispacy051/bin/python")
library(reticulate)
#reticulate::use_python("/home/jtimm/anaconda3/envs/m3demo/bin/python")
reticulate::use_condaenv(condaenv = "scispacy051",
                         conda = "/home/jtimm/miniconda3/bin/conda")

dd.df <- textpress::web_scrape_urls(x = 'Alzheimer Disease', cores = 10) |>
  filter(!is.na(text))
df <- reticulate::r_to_py(dd.df)

import sys
sys.path.append('/home/jtimm/pCloudDrive/GitHub/git-projects/spacy-nlp')
import spacyHelp

import pandas as pd
import os
import spacy
import scispacy
#nlp = spacy.load("en_core_sci_sm")
#nlp = spacy.load("en_core_web_sm")

• en_ner_bc5cdr_md, A spaCy NER model trained on the BC5CDR corpus. For disease and chemical NER. Details for additional models available here.

Another option is to use the generic scispacy “mention detector”, and then link to UMLS, eg.

• An abbreviation detector.

• An entity linker – here umls, but mesh, rxnorm, go, and hpo are also available knowledge bases that can be linked to.

• A hyponym detector.

nlp = spacy.load("en_ner_bc5cdr_md")
nlp.add_pipe("sentencizer", first=True)
nlp.add_pipe("merge_entities")

from scispacy.abbreviation import AbbreviationDetector
nlp.add_pipe("abbreviation_detector")

from scispacy.linking import EntityLinker
nlp.add_pipe(
  "scispacy_linker", 
  config={"resolve_abbreviations": True, 
  "linker_name": "mesh"})
  
linker = nlp.get_pipe("scispacy_linker")

from scispacy.hyponym_detector import HyponymDetector
nlp.add_pipe(
  "hyponym_detector", 
  last = True, 
  config={"extended": True})

print(nlp.pipe_names)

## ['sentencizer', 'tok2vec', 'tagger', 'attribute_ruler', 'lemmatizer', 'parser', 'ner', 'merge_entities', 'abbreviation_detector', 'scispacy_linker', 'hyponym_detector']

texts = list(r.df['text'])
doc = list(nlp.pipe(texts))

sp_df = spacyHelp.spacy_extract_df(doc)

reticulate::py$sp_df |>
  slice(1:5) |> knitr::kable()

sp_entities = spacyHelp.spacy_extract_entities(
  doc, 
  linker = linker)

reticulate::py$sp_entities |>
  sample_n(15) |> knitr::kable()

sp_abbrevs = spacyHelp.spacy_extract_abbrevs(doc)

reticulate::py$sp_abbrevs |>
  distinct(abrv, .keep_all = T) |>
  sample_n(15) |> knitr::kable()

sp_noun_phrases = spacyHelp.spacy_extract_nps(doc)

set.seed(9)
reticulate::py$sp_noun_phrases |>
  sample_n(10) |> knitr::kable()

Works better with nlp.add_pipe(“merge_entities”)

sp_hearst = spacyHelp.spacy_extract_hyponyms(doc)

reticulate::py$sp_hearst |>
  select(doc_id, sbj, pred, obj) |>
  sample_n(15) |> knitr::kable()

reticulate::py$sp_hearst |>
  count(pred) |>
  knitr::kable()

sp_sentences = spacyHelp.spacy_extract_sentences(doc)

reticulate::py$sp_sentences |>
  sample_n(5) |> knitr::kable()

Eyre, A.B. Chapman, K.S. Peterson, J. Shi, P.R. Alba, M.M. Jones, T.L. Box, S.L. DuVall, O. V Patterson, Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python, AMIA Annu. Symp. Proc. 2021 (in Press. (n.d.). http://arxiv.org/abs/2106.07799.

Kormilitzin, A., Vaci, N., Liu, Q., & Nevado-Holgado, A. (2021). Med7: A transferable clinical natural language processing model for electronic health records. Artificial Intelligence in Medicine, 118, 102086.

Neumann, M., King, D., Beltagy, I., & Ammar, W. (2019). ScispaCy: fast and robust models for biomedical natural language processing. arXiv preprint arXiv:1902.07669.