Artificial intelligence in pharmacovigilance: using AI to detect ICSRs in scientific literature

The question many Market Authorisation Holders (MAH) are asking themselves at the moment is how they can use artificial intelligence in pharmacovigilance literature review. The reason? Many MAHs face a growing challenge to reviewing scientific literature for Individual Case Safety Reports (ICSRs) caused by a rising volume of literature that needs to be assessed. To put this into context, every day around 6,000 peer-reviewed articles are published in roughly 10,000 journals; in 2020 alone the volume of literature for review grew by 10%. And to make matters worse, even with precision search only 5% of literature search results contain a valid ICSR once they’ve been manually reviewed. They are also under pressure to cut costs while maintaining full compliance.

This is similar to the situation that our customer GlaxoSmithKline faces. GSK, a global healthcare company, needs to review thousands of abstracts daily to identify ICSRs as part of its MAH obligations. To do this, it searches the literature using our Dialog platform. The results of Dialog are then output into our Drug Safety Triager where they are manually reviewed to determine if they contain the four criteria (a suspect drug, identifiable patient, an associated adverse event and the reporter) to be classed as containing a valid ICSR.

We’d previously spoken to GSK about our relevancy ranking engine DialogML, which is available with our new generation of Drug Safety Triager. DialogML uses Natural Language Processing (NLP) to enhance the efficiency of pharmacovigilance literature monitoring workflows. So with this in mind, GSK asked us to show exactly how DialogML could be used with Dialog and Drug Safety Triager to make literature reviewing for ICSRs more efficient than today.

 

Demonstrating artificial intelligence in pharmacovigilance for GSK

To prove exactly how DialogML could help them, we decided with GSK to run a proof of concept of DialogML.

For context, DialogML uses NLP to identify the abstracts that are most and least likely to contain a valid ICSR. It then gives each article it has looked at a relevancy ranking score and lists them based on this score. This means the abstracts most likely to contain an ICSR are at the top of the list and those least likely are at the bottom. This allows ICSR reviewers to focus their time and effort on the articles that are most relevant, helping them more quickly find relevant ICSRs and eliminate the non-relevant ones. DialogML also permits highlighting of the key drug safety terms that make a literature article relevant, further enhancing the speed of the literature review process.

To demonstrate DialogML’s benefits GSK provided us with test data (from  previously manually annotated scientific literature abstracts). This data was used to both train the NLP model in DialogML and then test the efficacy of this model.

 

What were the results?

Currently, the way articles are reviewed are in the order of them being received from alerts, which means that you can find your last ICSR at the end of the batch of the references received.

But when DialogML analysed the test data and ranked abstracts based on relevancy, 99% of the ICSRs were found in the top 33% of the list of abstracts. And all 100% of the ICSRs were in the top 65% of the total abstracts (See graph below). There were no ICSRs on the bottom 35% of the batch, making these articles candidates for a quick bulk assessment. Or to explain it another way, a literature reviewer could find 85% of valid ICSRs within the first hour, compared to 12.5% of ICSRs in the first hour without using DialogML.

 

 

The test also showed that DialogML can reduce the time each reference spends in the review queue. Ordinarily, each reference spends an average of 4.5 hours in the queue before it is seen by a literature reviewer. With DialogML, the test showed the time would be cut to 1.2 hours on average for each reference.

The highlighting of key drug safety terms like patient, adverse event, drug etc was also seen as useful for prompting the reviewer’s attention to the most relevant passages in the text.

 

What does this mean for drug safety literature monitoring?

Our test with GSK has demonstrated a number of key benefits for drug safety literature review teams using AI in pharmacovigilance.

The first is increased literature review efficiency. The relevancy ranking means that literature reviewers can focus first on the abstracts that have the highest chance of containing a valid ICSR.

The second advantage is that DialogML can reduce the time needed to move potential ICSRs from review to case processing, who are also under time pressure to submit the ICSRs to regulatory agencies. As well as improving efficiency this also has significant implications for supporting regulatory compliance. Another benefit is that the reviewer can finish the review faster with the help of highlighting.

And once DialogML is up and running in a literature review workflow, we’re able to gather evidence of how DialogML performs compared with human review. This means that there’s the potential in the future to target references at the bottom of the batch for automatic elimination and references on the top can be targeted for automatic submission to case processing.

 

Next steps

The artificial intelligence in pharmacovigilance test we ran with GSK proved the benefits of using AI and NLP within the literature review process. Partly due to this Proof of Concept (PoC) GSK has signed up to use DialogML as part of its medical literature monitoring process by upgrading to the latest version of Drug Safety Triager. With DialogML and Drug Safety Triager, GSK will benefit from the opportunity to further improve the efficiency of its literature review process.

GSK presented the results of this test at ICPE 2021 (23-25 August, 2021). You’ll be able to access the poster presenting the results of this PoC  in Pharmacoepidemiology and Drug Safety (PDS), the official journal of the International Society for Pharmacoepidemiology. The poster is titled “Automating Individual Case Safety Report identification within scientific literature using Natural Language Processing”.

We’re very excited to have demonstrated the real world impact DialogML, Drug Safety Triager and Dialog can have on the efficiency and effectiveness of the drug safety screening process. If you’d like to discuss your literature screening workflow and how Dialog Solutions can help, get in touch with us.