SIGNAL DETECTION THEORY

Week 7 BCS 208 Spring 2012

Signal Detection Theory

Baggage screening example

Who has better hearing, Julie or Regina?

Problem of criterion
Perceptual decisions are influenced by many factors:
-! -! Strength of signal relative to background level of noise Persons sensitivity to the signal Relative frequency of occurrence of the signal Persons level of motivation Costs associated with hits and false alarms Personal biases (individual differences)

SOLUTION: Measure sensitivity independent of criterion = Signal detection theory

Signal Detection Theory (SDT)"

! Set of assumptions and procedures for measuring perceptual performance (decision making) under conditions of uncertainty ! The goal being to distinguish performance changes/differences attributable to sensitivity vs criterion. ! Originates from World War II: aircraft detection on radar signals ! Today: widely used in psychophysics, medicine, radiology and machine learning
- Detecting faint sounds (e.g., murmur heard through stethoscope) - Interpreting blips on a radar screen (plane vs flock of birds - Stock market transactions (buy or sell?) - Jury decision (innocent vs guilty) - Deciding whether or not two lines are identical in length - Diagnosing X-ray images (e.g., is it a tumor?)

Signal Detection Theory (SDT)"

! Discrete observations occasioned by presence of noise only or signal plus noise ! Noise is anything that complicates detection of signal by introducing uncertainty about whether or not the signal is present
- Misleading evidence in a legal trial - Flock of birds on a radar screen - Scar tissue on a mammogram - Background noise in a hearing task

Signal Detection Theory (SDT)

Discrete observations occasioned by presence of noise only or signal plus noise

signal + noise

S response

S response

S response

S response

S response

S response

signal + noise

S response

S response

S response

S response

S response

S response

Signal Detection Theory (SDT) "

Four possible outcomes:" "- Signal was present and person says yes - HIT" "- Signal was present and person says no - MISS" "- Signal was not present and person says no - CORRECT REJECTION" "- Signal was not present and person says yes - FALSE ALARM"

Medical Diagnosis example:

Patient#s condition no tumor Diagnosis normal correct tumor incorrect

abnormal

incorrect

correct

Hit correct diagnosis of disease Miss physician tells patient that no disease exists when one does False alarm unnecessary operation or treatment Correct rejection patient properly diagnosed with no disease

Patient#s condition no tumor Diagnosis normal correct tumor incorrect

abnormal

incorrect

correct

Hit correct diagnosis of disease Miss physician tells patient that no disease exists when one does False alarm unnecessary operation or treatment Correct rejection patient properly diagnosed with no disease Which error is more costly?!

Patient#s condition no tumor Diagnosis normal

Correct rejection

tumor Miss

abnormal

False alarm

Hit

Which error is more costly?!

Accused is" Accused is" innocent guilty Jury decision innocent correct verdict mistake mistake correct verdict

guilty

P?

R PR R R R R R R R R

noise

signal + noise

Magnitude of sensory response

Weak response -- signal highly unlikely

Strong response -- signal highly likely

noise

signal + noise

Magnitude of sensory response

moderate response -- signal or noise?

criterion criterion criterion

Magnitude of sensory response

criterion

Magnitude of sensory response

noise signal + noise

Magnitude of sensory response

Signal present
criterion miss hit

no

yes

Magnitude of sensory response

noise signal + noise

Magnitude of sensory response

Signal absent
criterion correct rejection false alarm

no

yes

Magnitude of sensory response

noise signal + noise

Magnitude of sensory response

Magnitude of sensory response

noise signal + noise

Magnitude of sensory response

criterion criterion criterion

Magnitude of sensory response

noise signal + noise

Magnitude of sensory response

Balanced: false alarm and miss rates are equal Liberal: the observer says yes whenever there may be a signal Conservative: decision is yes only when it is almost certain that there is a signal

liberal

criterion

balanced
criterion

criterion

conservative

Magnitude of sensory response

noise signal + noise

Magnitude of sensory response

Discriminability: how well the observer can separate the presence of signal from its absence - overlap between the two distributions - Measured by d (discriminability index, also called sensitivity) - computed as: d" = z(Hit) z(FA)

criterion

Magnitude of sensory response

noise signal + noise

Magnitude of sensory response

Practical example: medical diagnosis

Liberal criterion

Tumor absent Probability

Tumor present

CR M

FA

Internal response strength (arbitrary unit)

Practical example: medical diagnosis

Conservative criterion

Tumor absent Probability

Tumor present

M CR CR M FA H

Internal response strength (arbitrary unit)

Receiver operating characteristic (ROC):"

The ROC curve is traced out by plotting Hits against False Alarms as the criterion moves."
100%

hit rate

false alarm rate

100%

criterion

Magnitude of sensory response

criterion

Magnitude of sensory response

criterion

Magnitude of sensory response

Receiver Operating Characteristic: ROC

criterion

Magnitude of sensory response

Receiver Operating Characteristic: ROC

criterion

Magnitude of sensory response

Receiver Operating Characteristic: ROC

criterion

Magnitude of sensory response

Receiver Operating Characteristic: ROC

criterion

Magnitude of sensory response

Receiver Operating Characteristic: ROC d

criterion

Magnitude of sensory response

ROC Curve Questions"

! Why do ROC curves start at (0,0)? Why do they go to (100,100)?" ! If d-prime is zero, what is the shape of the ROC curve?" ! If d-prime is large (e.g. 4 or larger) what is the shape of the ROC curve?"

ROC Curve"
100" Hit rate (%)" 80" 60" 40" 20" 0" 0" 20" 40" 60" 80" 100" False alarm rate (%)"

d# = 3" d# = 1" d# = 0"

ROC Curve"
100" Hit rate (%)" 80" 60" D! 40" 20" E! A! C! B!
! Which observer did best?" ! Who was just there completing the subject pool requirement?"

0" 0" 20" 40" 60" 80" 100" False alarm rate (%)"

Signal Detection Theory (SDT)- Online demos

http://www-psych.stanford.edu/~lera/psych115s/notes/signal/ http://psych.hanover.edu/JavaTest/Media/Chapter02.html Interactive Model 2.x: Signal Detection Theory Interactive Model 2.x: Signal Detection Illustration Interactive Model 2.x: Decisions In SDT Interactive Model 2.x: Receiver Operating Characteristic

Signal Detection Theory (SDT)

! Explains yes-no decisions ! Detecting signal in noise (S + N) ! Noise (N) is what makes detection hard (internal + external noise) ! Sensitivity (discriminability) vs. criterion (bias) ! Sensitivity (d) depends on ! signal strength ! noise strength ! observer sensitivity ! Criterion depends on ! Personal bias ! Cost/benefit factors (risk factors) ! Signal frequency ! ROC curve ! used to visualize SDT concepts & results

Signal Detection Theory (SDT) - experiment 1.2

! Sensitivity (d) depends on ! signal strength ! noise strength ! observer sensitivity ! Criterion depends on ! Personal bias ! Cost/benefit factors (risk factors) ! Signal frequency

Signal Detection Theory (SDT) - experiment 1.2

! Sensitivity (d) depends on ! signal strength ! noise strength ! Criterion depends on ! Cost/benefit factors (risk factors) ! Signal frequency

! Version #2 - Payoffs ! The other half will do one of the experiments manipulating sensitivity ! Version #4 Manipulation of the intensity of noise ! Cover general SDT issues in intro ! Compute C and d (detailed instruction in SG: pages 1.11 - 1.12) ! Hand plot YOUR data on the empty ROC curve ! use empty symbols for criterion exp (2X, label each, connect by line) ! use full symbols for sensitivity exp (2X, label each, connect by line) ! Group data will be emailed

! Half of the class will do one of the experiments manipulating the criterion