Human observer net
- Background: Current software applications for human observer studies of images lack flexibility in study design, platform independence, multicenter use, and assessment methods and are not open source, limiting accessibility and expandability. Purpose: To develop a user-friendly software platform that enables efficient human observer studies in medical imaging with flexibility of study design. Materials and Methods: Software for human observer imaging studies was designed as an open-source web application to facilitate access, platform-independent usability, and multicenter studies. Different interfaces for study creation, participation, and management of results were implemented. The software was evaluated in human observer experiments between May 2019 and March 2021, in which duration of observer responses was tracked. Fourteen radiologists evaluated and graded software usability using the 100-point system usability scale. The application was tested in Chrome, Firefox, Safari, and Edge browsers. Results: SoftwareBackground: Current software applications for human observer studies of images lack flexibility in study design, platform independence, multicenter use, and assessment methods and are not open source, limiting accessibility and expandability. Purpose: To develop a user-friendly software platform that enables efficient human observer studies in medical imaging with flexibility of study design. Materials and Methods: Software for human observer imaging studies was designed as an open-source web application to facilitate access, platform-independent usability, and multicenter studies. Different interfaces for study creation, participation, and management of results were implemented. The software was evaluated in human observer experiments between May 2019 and March 2021, in which duration of observer responses was tracked. Fourteen radiologists evaluated and graded software usability using the 100-point system usability scale. The application was tested in Chrome, Firefox, Safari, and Edge browsers. Results: Software function was designed to allow visual grading analysis (VGA), multiple-alternative forced-choice (m-AFC), receiver operating characteristic (ROC), localization ROC, free-response ROC, and customized designs. The mean duration of reader responses per image or per image set was 6.2 seconds 6 4.8 (standard deviation), 5.8 seconds 6 4.7, 8.7 seconds 6 5.7, and 6.0 seconds 6 4.5 in four-AFC with 160 image quartets per reader, four-AFC with 640 image quartets per reader, localization ROC, and experimental studies, respectively. The mean system usability scale score was 83 6 11 (out of 100). The documented code and a demonstration of the application are available online (https://github.com/genskeu/HON, https://hondemo.pythonanywhere.com/). Conclusion: A user-friendly and efficient open-source application was developed for human reader experiments that enables study design versatility, as well as platform-independent and multicenter usability.…
Author details: | Ulrich GenskeORCiD, Paul JahnkeORCiD |
---|---|
DOI: | https://doi.org/10.1148/radiol.211832 |
ISSN: | 0033-8419 |
ISSN: | 1527-1315 |
Pubmed ID: | https://pubmed.ncbi.nlm.nih.gov/35258375 |
Title of parent work (English): | Radiology |
Subtitle (English): | a platform tool for human observer studies of image data |
Publisher: | Radiological Society of North America |
Place of publishing: | Oak Brook, Ill. |
Publication type: | Article |
Language: | English |
Date of first publication: | 2022/03/08 |
Publication year: | 2022 |
Release date: | 2024/05/17 |
Volume: | 303 |
Issue: | 3 |
Number of pages: | 7 |
First page: | 524 |
Last Page: | 530 |
Funding institution: | Bundesministerium fur Wirtschaft und Energie [03EFHBE093]; Berlin; Institute of Health; BIH-Charite Clinician Scientist Program -; Charite-Universitatsmedizin Berlin |
Organizational units: | Digital Engineering Fakultät / Hasso-Plattner-Institut für Digital Engineering GmbH |
DDC classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 000 Informatik, Informationswissenschaft, allgemeine Werke |
6 Technik, Medizin, angewandte Wissenschaften / 61 Medizin und Gesundheit / 610 Medizin und Gesundheit | |
Peer review: | Referiert |