August 24, 2020
If you work in pharmacovigilance (PV), are the robots coming for your job? Medical literature monitoring is part of the PV process with significant scope for automation. On the face of it, this seems an obvious insight. Literature monitoring ticks all the boxes for needing to be automated. It can be repetitive, time-consuming work that ties up experts who could be spending more time on patient safety work; and it deals with information types and terminology that can have a great deal of commonality. Now that natural-language AIs are coming close to passing the Turing test, why wouldn’t modern AI and machine learning (ML) technology be able to read items of medical literature and decide which were relevant?
There are several AI and machine learning systems for medical literature monitoring already in the market and in development by both vendors and pharma companies. One uses AI to pre-assess references, reading the title, abstract and index of an article to determine whether it discusses a patient safety issue. However, even though these elements of a reference can contain rich information, an AI may miss an ICSR if it relies on them alone.
In the example below, the reference data contains no author abstract or indication that patient safety issues exist: yet the full text contains a reference to a serious ICSR relating to a diabetic individual sharing a friend’s Metformin prescription in place of his insulin. While an AI may have categorised this reference as irrelevant, an expert reviewer’s “sixth sense” based on years of experience may have caused them to consider it.
Then there’s the problem of bias in machine learning: the algorithm is only ever as good as the data it’s trained on. AIs have been seen to reflect race or gender bias, for example: fundamentally, to work effectively, they depend on data that has integrity and accurately reflects the sets of references being searched and monitored.
So even with state-of-the-art technology, it can be risky to depend on AI, machine learning or automation alone for medical literature monitoring. Replace a monitoring team with an AI that misses ICSRs and the stakes are, of course, very high: missing a relevant reference can have unthinkable knock-on consequences for pharma companies. Plus there’s the time and effort of going back manually through references, and then switching off and retraining the algorithm.
Dialog Solutions’ approach uses a blend of ML and human intervention to reduce the effort in literature monitoring while minimising the risk of missing references. We use ML to determine the relevance of a reference, giving a ranking of between 1 and 0 to each article. Applying this score before references are presented for assessment enables reviewers to identify and prioritise references with patient safety issues more quickly. References with low relevance scores can be quickly eliminated manually on examination of titles, as in the example below, which is clearly not related to a patient safety issue:
In literature monitoring, we believe the role of ML is to augment rather than replace human expertise: it helps reviewers to work faster and deliver higher-quality results. A 2018 study suggested that technology-enabled literature monitoring could reduce the amount of time that pharmacovigilance reviewers spend on literature monitoring by 89.5 percent.
The key, as always, is to achieve efficiency while not missing relevant references. At the moment, automation technologies cannot provide both. In our view, a hybrid expert/machine solution is the best way to reduce the administrative burden of medical literature monitoring without compromising patient safety.
One of the most important weapons in the fight against the global threat of coronavirus (COVID-19)...
As part of our ongoing support of the research efforts for COVID-19 treatments, vaccines, and...
As we continue to monitor the spread of the novel coronavirus (COVID-19) and its global impact,...