AI Abstract Screening Challenges: Risks and Validation

AI abstract screening challenges arise when teams treat a model’s output as a reliable eligibility decision rather than as review-specific evidence to evaluate. AI can prioritize records for human review, classify records against criteria, or support a hybrid workflow, but these uses carry different risks: prioritization changes screening order, while automated exclusion removes records without human assessment and therefore needs stronger validation. [2]

The central trade-off is between finding eligible studies and reducing human workload. No single accuracy score resolves that trade-off for every review. Teams need to define the AI’s role in advance, test it against clear eligibility criteria, and preserve human oversight and an audit trail. [1][2]

What does AI abstract screening actually do?

AI abstract screening can rank records, suggest inclusion decisions, or exclude records automatically, depending on the workflow. [2] These functions should not be treated as interchangeable: ranking helps reviewers decide what to inspect first, whereas automated exclusion can hide potentially eligible records from human assessment.

A tool may also be used as decision support, with reviewers checking its recommendations, or as part of a hybrid process in which human and automated decisions are combined. The appropriate role depends on the review’s eligibility criteria, the consequences of missed studies, and the strength of review-specific validation. A tool’s availability does not establish that its particular screening function is suitable for a particular review.

How should you measure screening performance?

Screening performance should be assessed using recall and precision, with specificity and workload reported at the selected decision threshold. [1] Recall is the proportion of eligible records the workflow identifies; low recall means eligible records may be missed. Precision is the proportion of records sent forward that are truly eligible; lower precision means more records require human review.

A single accuracy figure can conceal these trade-offs. In a medical-literature screening meta-analysis, pooled recall was 0.928 and pooled precision was 0.200 when models were tuned for maximum recall; when tuned for maximum precision, pooled recall fell to 0.708. [1] These results describe the studies included in that analysis, not a performance guarantee for a new review.

Report the metrics together with the threshold or operating point used and the resulting workload. That makes it possible to understand what was gained or lost—for example, whether fewer records reached reviewers at the cost of potentially missing eligible studies. Performance should be interpreted in the context of the review’s own data and criteria, not inferred from a published average alone. [1]

Why can faster screening still miss important studies?

Faster screening can still miss eligible records because reducing the volume sent to reviewers may also reduce recall. The risk is clearest when a system automatically excludes records: unlike prioritization, exclusion removes records from human assessment, so it requires stronger validation. [2]

Large language model results also vary across models, prompts, settings, and datasets. An exploratory biomedical study reported sensitivities from 81.93% to 97.58% and specificities from 19.12% to 75.19% across four open models and ten datasets. [3] The range illustrates why results from one setup should not be assumed to transfer to another review.

Active-learning workflows add a stopping challenge. A 95% recall target, including WSS@95 reporting, is a benchmark, not proof that a live screen is safe. Relevant records may stop appearing in the queue even though the true recall is unknown; teams therefore need a prespecified stopping method rather than relying on an empty run of relevant results. [1][7]

How do you validate an AI workflow for your review?

Validate the workflow against the review’s own eligibility criteria and a clear reference set before relying on its recommendations. Retrospective evaluation on completed reviews does not establish performance on a new review; a 2026 scoping review reported that 63% of included evaluations used retrospective comparisons with human review and found substantial variation in metrics and definitions. [4]

A practical validation plan should specify the AI’s role, the records reviewers will assess, how disagreements will be resolved, and how false exclusions will be examined. Agreement between an AI system and reviewers is not, by itself, evidence that the workflow is valid: the assessment also needs to establish whether eligible records were missed. [8]

Use caution when transferring results between review topics, datasets, prompts, or model versions. LLM screening performance varies across those conditions, and retrospective evidence may not reflect how the system behaves in a new workflow. [3][4] Keep a human review pathway for decisions that could exclude eligible studies, and do not treat a benchmark recall target as a universal safety threshold. [1][2]

What should you document and report?

Document the AI tool and version, prompts or settings, eligibility and stopping rules, reviewer roles, and how disagreements were handled. Report whether any records were excluded solely by automation, since that makes the consequences of the workflow visible to readers. [2][3]

Also document how performance was evaluated and which measures were used. Cohen’s kappa can summarize chance-adjusted agreement between two reviewers, but it should be interpreted alongside raw counts or percentage agreement, particularly when inclusion is uncommon. [9] Agreement statistics describe consistency; they do not replace checking whether eligible records were missed. [8]

Account for the full workflow effort when reporting time savings. This includes setup, calibration, false-positive review, and ongoing verification—not only the time spent making screening decisions. One living-review team reported 30–40 researcher-hours refining and testing prompts, while a separate study reported reduced human-review volume in its particular dataset and called for validation elsewhere. [5] Those findings should not be treated as general estimates of effort or savings.

Key takeaways

• Decide whether AI will rank, recommend, or exclude records; automated exclusion needs stronger validation than prioritization. [2]
• Report recall, precision, specificity, and workload at the chosen threshold rather than relying on one accuracy score. [1]
• Treat published screening results as context, not as guaranteed performance for a new review. [1][3]
• Define stopping rules in advance; a benchmark recall target does not prove that a live screen is safe. [1][7]
• Evaluate false exclusions as well as reviewer agreement, and document the tool, settings, reviewer roles, and automation decisions. [2][3][8]

FAQ

How does AI improve abstract screening in systematic reviews?

AI can prioritize records, classify them against eligibility criteria, or support a hybrid screening workflow. [2] Its value should be assessed against the review’s actual workload and the risk of missing eligible studies, rather than assumed from automation alone.

What are the benefits of using AI for abstract screening?

AI may reduce the number of records requiring human review in a particular workflow, but the evidence does not establish a universal level of time savings. [5] A realistic assessment includes setup, testing, verification, and the effort needed to review records the system sends forward.

How accurate is AI in abstract screening for systematic reviews?

There is no universal accuracy figure for AI abstract screening because performance depends on the model, prompt, settings, dataset, and threshold. [1][3] Report review-specific recall and precision, alongside specificity and workload, and examine false exclusions.

How is inter-rater agreement measured in systematic reviews?

Cohen’s kappa is one way to summarize chance-adjusted agreement between two reviewers. [9] Interpret it alongside raw counts or percentage agreement, especially when included records are uncommon; agreement alone does not show that the screening process found all eligible studies. [8][9]