Center for treatment of Rheumatic and Musculoskeletal Diseases (REMEDY)
Diakonhjemmet Hospital, Oslo, Norway
Joseph Sexton MSc PhD, statistician
Amirhossein Khazemi MSc, statistician
Eirik Klami Kristianslund MD PhD, medical researcher
Siri Lillegraven MD MPH PhD, associate professor
Summary
Reporting of adverse events in clinical studies are typically coded according to the Medical Dictionary for Regulatory Activities (MedDRA) classification to enable analysis. This document describes the development and validation of a system to automatically code the free-text adverse events (AE) entries in clinical studies according to the MedDRA system. The automatic coding will be developed from a training dataset of AEs already manually coded in the MedDRA system, using natural language processing. The automatic coding is validated by assessing the system’s performance in already coded datasets. In addition, manual assessment of several subsamples of the resulting coded data can be performed.
Aim of study
The aim of this study is to develop methods to automatically code AE reports from clinical studies in the MedDRA classification exploiting automated processes based on previously coded registrations, automated language translation tools and natural language processing models.
Background
Adverse events
An AE is any untoward medical occurrence in a patient under treatment which does not necessarily have a causal relationship with the treatment. An AE can therefore be any unfavorable and unintended sign, symptom, or disease while the patient is using a medication, whether or not related to the medication. Recording and reporting adverse events is an essential part of clinical studies.
MedDRA classification
MedDRA is a standardised medical terminology commonly used for the classification of adverse events in research studies. MedDRA has five levels in the hierarchy. The lowest is the Lowest Level Term (LLT) with more than 80 000 terms that reflect how observations are communicated by patients. The next level is the «Preferred Term» (PT), which is a single medical concept, and includes more than 20 000 terms. Each LLT is linked to only one PT, but each PT may have several LLT. Terms are further grouped into «High Level Term» (HLT), «High Level Group Term» (HLGT) and «System Organ Class» (SOC).
Understanding MedDRA (meddra.org)
Help to Shape the MedDRA Terminology | MedDRA
Motivation for the project
To enable analyses of AE-data, each event must be labeled with MedDRA classes. This is, however, not always done in the clinic. Many events are registered only with a comment from the clinical investigator. These comments might be in Norwegian, English or a mixture of the two with possible spelling errors. Traditionally, it has been the job of the researchers to go through all events and code them according to the correct MedDRA classes. However, this is a time consuming and cumbersome task. This project is therefore initiated to automate the classification of adverse events.
Previous automated coding of adverse events
Italian study on automatic coding of adverse events: From narrative descriptions to MedDRA: automagically encoding adverse drug reactions - PubMed (nih.gov)
Master thesis exploring automatic coding of adverse events using previously coded data and the pre-trained language model BERT: Mapping medical expressions to MedDRA using Natural Language Processing (diva-portal.org)
Blog post describing different approaches to automatic coding of AE entries: Blog : MedDRA and Natural Language Processing (NLP) | MedDRA Coding Feature (terminologix.com)
Congress abstracts describing automatic coding of AE entries in study of Alzheimer disease using machine learning: (PDF) Machine Learning in Automated Classification of Adverse Events in Clinical Studies of Alzheimer's Disease
https://alz-journals.onlinelibrary.wiley.com/doi/full/10.1016/j.jalz.2018.06.1369
Methods
The data used for this project comes from the NOR-DMARD study. The results can potentially be validated in other clinical studies, including randomized controlled trials.
NOR-DMARD
The Norwegian Antirheumatic Drug Register (NOR-DMARD) is a register-based longitudinal observational study of which the main objectives are to study the effectiveness of treatment of inflammatory joint diseases with biological disease modifying anti-rheumatic drugs (DMARDs) in clinical practice, by measuring disease activity, health related quality of life, function and joint damage, and to study the long-term safety of such treatment. Other objectives include the assessment of cost-effectiveness of treatment, to identify and validate clinical, genetic and immunological predictors of efficacy and adverse events, to assess the impact of treatment with biological DMARDs on work participation and work productivity, to investigate different strategies for use of biological DMARDs, to assess the performance of different outcome measures, and to ensure a systematic and timely follow-up of patients treated with biological DMARDs.
This study has been conducted in 5 Norwegian rheumatology departments since the year 2000. The study includes a structured follow-up with study visits at baseline (start of biological drug) and after 3, 6, 12 months and every 12 months thereafter. Each study visit includes clinical assessment, patient-reported outcomes and measurement of acute-phase reactants (markers of inflammation in blood samples). The data collection also includes blood samples for biobank at baseline and 3 months. Serious adverse events data will be systematically recorded, but other adverse events can also be retrieved by linkage to other registers.
This is a cohort study of about 10 000 treatment courses with biological anti-rheumatic drugs for inflammatory joint disease and about 7000 treatment courses of conventional anti-rheumatic drugs.
• NOR-DMARD 2000-2012: 32 000 adverse event entries in 10 000 treatment courses, both original entry by study physician or nurse and the MEDRA-coded LLT entry. Includes 3000 different LLT terms, corresponding to 1800 unique PT terms.
• NOR-DMARD 2012-2021: 12 000 adverse event entries in 4600 treatment courses, only original entry by study physician or nurse.
A potential challenge is sparsity of some of the labels. NOR-DMARD 2000-2012 have 3046 unique LLT, of which 477 appear at least 10 times in the dataset. 1404 appear only once, and 449 appear only twice. There are 1804 unique PT, of which 394 appear at least 10 times. 707 appear once and 249 appear twice.
Previous work on the project
The aim is correctly coded MedDRA PT in the NOR-DMARD 2012-2021 data. The methods tried so far have used the following on a training set from the NOR-DMARD 2000-2012 data as well as the MedDRA code book.
1. String distance
2. SVM using the distribution of characters
3. Automated language translation (Google Translate)
A random decision forest selects the best prediction. Expected performance is correct coding of 70 % of LLT and 80 % of PT. The method may select a different LLT than the training dataset, but still select one that results in the same PT. The method should return a measure of uncertainty. Predictions with a high uncertainty are disregarded and the entries are classified manually.
Validation
Manual validation of random sample of coded entries
1. Random sample of coded data
2. Random sample of rare entries, both rare in the training dataset and in the dataset to be coded (e.g. entries coded only once)
3. Random sample of the entries with the highest uncertainty
Performance in subset of already coded data not included in the training dataset (validation set).
• Validation subset of NOR-DMARD 2000-2012, not included in training dataset
• Other clinical trials, such as the randomized controlled trials ARCTIC, ARCTIC-REWIND, NOR-SWITCH and NOR-DRUM.
Impact and dissemination
Automatic coding of adverse events from free-text entries to the MedDRA system is relevant for all clinical studies reporting adverse events. Previous approaches has been published in scientific journals. The method can potentially be expanded to databases of adverse events outside clinical studies (see previous approach in master thesis from University of Uppsala).