Skip to content

LUMINA

Automating the manual screening of medical reviews

first author · NUS research · 2024–25

Why it exists

A medical systematic review takes a team of five about 15 months (Borah et al., 2017 (opens in new tab)). The first step is the grind: reading thousands of abstracts by hand to find the 3 in 100 that belong. Miss one and the evidence is wrong.

What I built

I built LUMINA to do that screening. Four agents work in two passes: a generous first cut, a strict check against the review’s criteria, and a reasoning model that challenges every call before it stands. As first author, I designed it, built it and ran the evaluation.

Impact

  • Missed 8 of 378 included studies across 15 published reviews (98.2% sensitivity).
  • Cut what a team reads in full to under 9% of the candidates.
  • 35× fewer misses than a reimplementation of Li et al. (2024), and every study found on the four reviews Tran et al. reported.
  • Under a cent per citation, across about 150,000 abstracts screened.

73,523 abstracts, from 15 published reviewsEach band is one review, to scale

6,469 left for people to read in full, 8.8%

Review 1 of 15, largest first

Abstracts
12,961
Left to read in full
1,46511.3%
Included studies kept
28 of 29

The studies that belonged

370 kept8 missed

Underneath. Every decision, review, revision and its cost is written to a trace, so any call can be checked afterwards.

How it narrows

  1. Classifier gpt-4o-mini Asks whether a study could be relevant, and keeps anything uncertain.

  2. Reviewer o3-mini A separate reasoning model checks every call and its reason.

  3. Improver gpt-4o-mini Revises a call the reviewer disputes, for up to three reviews.

  4. Detailed screener gpt-4o-mini Checks what is left against the review’s PICOS criteria, with the same reviewer and improver.

LUMINA on 15 published reviews. The bands and dots are the evaluation’s own numbers, review by review: the data.
LUMINA · screening, review, and decision traces

Scroll across to follow the full system

Citation titles and abstracts pass through a lenient classifier. Relevant or uncertain candidates advance to PICOS screening with the review's objectives and methods. Each tier uses an o3-mini reviewer and a gpt-4o-mini improver, with at most three reviews per tier. At the limit, the last improver output is used. Included citations proceed to human full-text review. Completed candidates' decisions, feedback, revisions, and costs are saved as JSONL.Candidate citationsCSV / RIS · title + abstractReview protocolTitle / abstract · objectives + methodsINPUT01TriageTitle + abstractrevise, then recheckdisagreeClassifiergpt-4o-miniReviewero3-miniImprovergpt-4o-miniExcludeLikely irrelevant02PICOS screeningCriteria + methodsrevise, then recheckdisagreeDetailed screenergpt-4o-miniReviewero3-miniImprovergpt-4o-miniExcludeDecision recordedrelevant / uncertainFull-text reviewHuman eligibility checksJSONL traceDecisions + justificationsReview feedback + revisionsToken cost per completed citationPython library + CLI · versioned promptsAt most 3 reviews per tier. On the iteration limit, the last improver output is used.
Follow one citation through screening
Two-tier screeningCitation lifecycle
Title & abstractCitation A

Illustrated inclusion path

trace.jsonlDecision · feedback · revisions · cost

Begin with the title and abstract

A citation enters alongside the systematic review’s scope. The first tier asks whether it could be relevant, keeping uncertain studies in play.

01 / 08Read citation
Screening pipeline source

Method and results

Two screening tiers

The classifier sees the review and candidate titles and abstracts. It keeps potentially relevant and uncertain citations for the detailed screener, which also receives the review’s objectives and methods. The second tier works through Population, Intervention, Comparison, Outcome, and Study design (PICOS) before returning an inclusion or exclusion. Read the prompts (opens in new tab).

Review, revision, and termination

The classifier, screener, and improver use gpt-4o-mini; the reviewer uses o3-mini. At each tier, the reviewer checks the decision and its justification. Disagreement sends the original task and reviewer feedback to the improver, then back for review. The implementation caps each tier at three reviewer calls; if agreement has not been reached, it uses the last improver output. Follow the control flow (opens in new tab).

Decisions you can inspect

Each completed citation produces a typed AgentTrace: classifier and screener outputs, review feedback, revisions, final decision, and cost. The CLI accepts CSV or RIS candidates and writes one JSONL record per completed citation. Prompts are versioned alongside the Python code. Decision markers are parsed and validated, with bounded retries for invalid output. LUMINA decides what people read in full; it does not do that reading for them.

Evaluation

Screening 15 published systematic reviews after the fact, against the studies their authors included, LUMINA achieved 98.2% mean sensitivity and 87.9% mean specificity, missing 8 of those 378 studies. Metrics are calculated per review, then averaged. Against a reimplementation of Li et al. (2024) on the same reviews, its miss rate was 35× lower (0.018 against 0.630). In all, the pipeline has screened about 150,000 candidate abstracts, at under a cent each. Sensitivity comes first: the goal is to retain relevant studies while reducing the material that needs full-text review. Inspect the results and evaluation data (opens in new tab).

The baseline and the design change

Tran et al., Annals of Internal Medicine (2024) (opens in new tab) tested GPT-3.5 Turbo with separate eligibility prompts, combining their answers under a balanced rule or a rule favoring sensitivity. LUMINA adds a lenient first pass, contextual PICOS screening, and explicit review and revision at both tiers. On the four reviews Tran reported, LUMINA found every included study; Tran’s pipeline found 81% to 97%.

Python · gpt-4o-mini · o3-mini · PICOS · JSONL

zanwenfu/agentic-reviewers-for-SRMA