Review of ROC analysis

Some of the material was modified from the review of the application of ROC analysis, "Measuring the Observer Performance of Digtial Systems" by Philip F Judy, Cornella M Schaefer, Jorg-Wilhem Oestmann and Reginald E Greene. In:Computed Digital Radiolgraphy in Clical Practice. Editors RE Greene and JW Ostmann, Thieme Medical Publishers, New York, 1992:59-69. (Reprint available from first author.)

Maximum likelihood fitting procedure
Experimental images are divided into two classes of stimuli such as A (control images) and B (images with pulmonary nodules). In an ROC experiment, observers make an ordinal rating to indicate their confidence that the stimulus belongs to alternative A rather than alternative B. The rating is assumed to be related to a continuous decision variable that is monotonically related to the observer's estimate of the likelihood ratio that the stimuli is alternative A rather than alternative B. The decision variable has a separate probability density distribution for each alternative. Each probability distribution can be used to determine the fraction of stimuli A or B that the observer will assign to each ordinal rating. It is the area under the probability density distribution between two values of the decision variable, called criteria.

Primarily for the convenience of fitting the rating data, gaussian distributions of the decision variable are usually assumed. The parameters of distribution and quantities derived from them are useful as summary statistics of the observers' performance. A computer program originally developed by Dorfman and Alf¹ and widely distributed by Charles Metz of the Department of Radiology at the University of Chicago is used to analyze rating data. We have extended the procedure to treat more the two alternatives.²

Area under the ROC curve
Receiver operating characteristic (ROC) analysis is the standard approach to evaluate the sensitivity and specificity of diagnostic procedures.³ ROC analysis estimates a curve, which describes the inherent tradeoff between sensitivity and specificity of a diagnostic test. Each point on the ROC curve is associated with a specific diagnostic criterion. This point will vary among observers because their diagnostic criteria will vary even when their ROC curves are the same. The area under the ROC curve (A-z) has become a particularly important metric for evaluating diagnostic procedures because it is the average sensitivity over all possible specificities.^4,5,6

Notes
¹ Dorfmann DD, Alf E. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals - Rating-method data. J Math Psychol 1969;6:487-496.
² Kijewski MF, Swensson RG, Judy PF. Analyisis of rating data from multiple-alternative tasks. J MAth Psychol 1989;33(4):428-451.
³ Swets JA, Pickett RM. Evaluation of diagnostic systems: Methods from signal detection theory. Academic Press, New York, 1992.
⁴ Swets JA. ROC anaylis applied to the evaluation of medical imaging techniques. Invest Radiol 1979;14:109-121.
⁵ Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic curve. Radiology 1982;143:29-36.
⁶ Metz CE. ROC methodology in radiologic imaging. Invest Radiolo 1986;21:720-733.

[Format Density] | [Introduction]

Influence of CT Image Size and Format on Accuracy of Lung Nodule Detection
SE Seltzer, PF Judy, U Feldman, L Scarff, FL Jacobson. Radiology 1998;206:617-622.

Is this a mirrored page?
The official URL is http://brighamrad.harvard.edu/research/topics/vispercep/ROC.html