Professional Documents
Culture Documents
Provides a method for characterizing human performance in detecting, discriminating and estimating signals. For noisy signals, provides a method for identifying the optimal detector (the ideal observer) and for expressing human performance relative to this. Origins in radar detection theory Developed through the 1950s and on by Peterson, Birdsall, Fox, Tanner, Green & Swets
J. Elder
Example 1
3
The observer sits in a dark room On every trial, a dim light will be flashed with 50% probability. The observer indicates whether she believes the light was flashed or not. This is a yes-no detection task.
J. Elder
Noise
4
In this example, the information useful for the task is the light energy of the stimulus. By the time the stimulus information is received by decision centres in the brain, it will be corrupted by many sources of noise:
Many of these noise sources are Poisson in nature: the dispersion increases with the mean.
PSYC 6256 Principles of Neural Coding J. Elder
It is often possible to approximate this noise as Gaussian-distributed, with the same variance for both stimulus conditions. Then the noise is independent of the signal state.
J. Elder
Discriminability d
6
p x | S = sH =
x H exp 2 2 2 1 x L exp 2 2 2 1
) )
p x | S = sL =
H L
J. Elder
Criterion Threshold
7
The internal response is often approximated as a continuous variable, called the decision variable. But to yield an actual decision, this has to be converted to a binary variable (yes/no). A reasonable way to do this is to define a criterion threshold z:
x z ' yes' x < z 'no'
x z
J. Elder
J. Elder
p x | S = sH =
x H exp 2 2 2 1 x L exp 2 2 2 1
) )
p x | S = sL =
H L
J. Elder
Suppose the observer wants to maximize the expected number of times they are right. Then the optimal decision rule is to always select the state s with higher probability for the observed internal response x:
This is the maximum likelihood detector. For the equal-variance case, this means that the criterion is the average of the two signal levels:
1 z = L + H 2
J. Elder
Optimal Performance
11
The performance of the maximum likelihood observer for this yes/no task is given by
d p(correct) = p(HIT) = p(CORRECT REJECT) = erfc 2 2
J. Elder
Bias
12
For this optimal decision rule, the different types of errors are balanced: p(FA) = p(MISS) For observers that use a different criterion, the different types of errors will be unbalanced. Such observers have lower p(correct) and are said to be biased.
J. Elder
ROC Curves
13
Suppose the experiment is repeated many times under different instructions. The first time, the observer is instructed to be extremely stringent in their criterion, only reporting yes when they are 100% sure the light was flashed. On subsequent repetitions, the observer is instructed to gradually relax their criterion.
J. Elder
ROC Curves
14
As the criterion threshold is swept from right to left, p(HIT) increases, but p(FA) also increases. The resulting plot of p(HIT) vs p(FA) is called a receiveroperating characteristic (ROC).
Increasing d
d = 0
J. Elder
ROC Curves
15
Note that d remains fixed as the criterion is varied! Thus d is criterion-invariant, and is thus a pure reflection of the signal-to-noise ratio.
J. Elder
Britten et al (1992) Random dot kinematogram Signal dots are either all moving up or all moving down Noise dots are moving in random directions
J. Elder
100% Coherence
17
J. Elder
30% Coherence
18
J. Elder
5% Coherence
19
J. Elder
0% Coherence
20
J. Elder
www.thebrain.mcgill.ca
J. Elder
Experimental Details
22
Signal direction always in preferred or antipreferred direction for cell. What kind of task is this? Note that now there is external noise as well as internal noise. To calculate neural discrimination performance, assumed neuron paired with identical neuron, tuned to opposite direction of motion.
J. Elder
Behaviour
Anti-Preferred Direction
Neuron
Preferred Direction
Hit Rate
Priors
25
Note that if the probabilities of the two signal states are not equal, the maximum likelihood observer will be suboptimal. In this case we must make use of the posterior ratio.
J. Elder
MAP Inference
26
( ) = p ( x | s ) p (s ) p (s | x ) p ( x | s ) p (s )
H H L L
J. Elder
Maximizing p(correct) is not always the best thing to do. How would you adjust your criterion if you were
venture capitalist trying to detect the next Google? A pilot looking for obstacles on a runway?
J. Elder
Loss Function
28
In general, different types of correct decision or action will yield different payoffs, and different types of errors will yield different costs. These differences can be accounted for through a loss function:
Let a( x) represent the action of the observer, given internal response x.
Then L s, a( x) represents the cost of taking action a, given world state s.
J. Elder
The Ideal Observer uses the decision rule that minimizes the Expected Loss, aka the Risk R(a|x):
R(a | x) = L s, a( x) p(s, x) = L s, a( x) p( x | s )p(s )
s s
J. Elder
J. Elder