This pilot study aims to evaluate the accuracy of an artificial intelligence (AI) tool when extracting data for a scoping review. Use of AI tools to support literature reviews is expanding, with potential to accelerate complicated and labor-intensive processes while maintaining quality results. In an ongoing faculty-student scoping review with 437 articles, the research team piloted the Elicit AI tool on a subset of 50 studies. For each article, data was extracted manually by one human reviewer using a standardized template in Covidence and extracted in parallel by Elicit using aligned standard and custom fields. Two additional team members independently compared the human and AI extracted data, using a 0 to 3 rating scale reflecting absent, poor, acceptable, and optimal matches, consulting full texts as needed. 18 categories were evaluated. Disagreements will be resolved by consensus. Percent agreement between raters will be calculated, and mean ratings computed across categories. Using the final rating results, sums and means for human versus Elicit output will also be compared. Findings will indicate the suitability of Elicit for data extraction and identify which categories are usable or unsuitable. Results have the potential to inform review methodology and expedite timelines, particularly for large literature reviews.
