BERT-based approach for relation extraction using NLP - RelEx-BERT

Our BERT-based approach explores using BERT contextualized embeddings into a simple feed-forward neural network. We first extract the sentence containing the relation and pass it through a pre-trained BERT model. The output is then fed into a dropout layer and then into a fully-connected dense layer for classification. As with our deep learning-based approaches, we treat the RE as a binary classification task building a separate model for each drug-entity type. Figure shows the architecture of our BERT-based approach. Our approach contains the following BERT-based language models:

• BERT (-cased and -uncased) - Original BERT models trained on a large corpus of English data: BookCorpus (800M words) and Wikipedia (2,500M words) in a self-supervised manner (without human annotation).
• BioBERT - Model is initialized with the general BERT and further trained over a corpus of biomedical research articles from PubMed¹ abstracts and PubMed Central² article full texts.
• Clinical BERT - Model is initialized with BioBERT and further fine-tuned over the Medical Information Mart for Intensive Care-III (MIMIC-III) clinical note corpus.