) 5A = JE?

K JE A@E= ) = OIEI )FFH =?D 7JE E E C = 4ACE 6DAI=KHKI = @ 5)

-L=CCA I 5FOH K /E HC I 6 E=I 2DEL I O =I = @ ;= EI )LHEJDEI 1 =CA 8E@A = @ K JE A@E= => H=J HO 5?D B - A?JHE?= = @ - A?JH E?I - CE AAHE C =JE = 6A?D E?= 7 ELAHIEJO B )JDA I {AIFOH K CJ E=I B O =I E=LH}(E =CA JK= CH

)>IJH=?J
6DEI F=FAH FHAIA JI = =FFH =?D DECD ALA BA= JKHA @AJA?JE MEJDE LE@A @ ?K A JI KIE C = 4ACE 6DAI=KHKI = @ =JA J 5A = JE? ) = OIEI ) LE@A ID J EI HAFHAIA JA@ >O = IE C A AOBH= A 2-/ % BA= JKHAI =HA ANJH=?JA@ BH ? =HIA HACE I B EJ ) ? KIJAH E C = C HEJD EI =FF EA@ = ANJH=?JA@ HACE I = @ = HACE JDAI=KHKI EI ? IJHK?JA@ 1JI KIA EI J =I IEIJ J JDA =FFE C B M J DECD ALA BA=JKHAI >O = @A LA?J H HAFHAIA J=JE =JA J 5A = JE? ) = OIEI EI JDA =FF EA@ JDA @A LA?J HI J ANF EJ JDA =JA J HA =JE I = C HACE JOFAI =E E C J E FH LA @AJA? JE FAHB H = ?A 6DA FH F IA@ =FFH =?D EI JD H KCD O AN= E A@ KIE C 64-+81, % @ALA F A J @=J=

 1 JH @K?JE
6DA ? JE K KI O CH ME C L K A B K JE A@E= ? JA J D=I A@ = O HAIA=H?D A HJI J DECD ALA ? ?AFJ @AJA?JE IE ?A JDA IA = JE?I JD=J = @ ?K A J ? J=E I FH LE@A = A A?JELA = @ @AIEH=> A = J=JE B EJI ? JA J 0 MALAH @AJA?JE C JDA =?JK= IA = JE?I MEJDE E =CA = @ LE@A @ ?K A JI HA =E I IJE = ?D= A CE C OAJ K I LA@ FH > A 1JI JM =E = @ IJ E JAHAIJE C =IFA?JI =HA JDA IA A?JE B JDA M ALA BA= JKHAI J >A ANJH=?JA@ = @ = I JDA =FF EA@ AJD @ B H =IIEC E C JDA J DECD ALA ? ?AFJI = FH > A ? O HABAHHA@ J =I JDA 5A = JE? /=F = O @A I?HEFJ HI D=LA >AA FH F IA@ JD=J ?=FJKHA JDA M ALA BA=JKHAI B K JE A@E= @ ?K A JI = @ = O A=H E C JA?D EGKAI IK?D =I AKH= AJM H I D=LA >AA IK??AII BK O =FF EA@ J =F JDA E J IA = JE? ? ?AFJI 6DA E@A= B KIE C = @E?JE =HO J @AI?HE>A = @A ? F IA@ E =CA @AHELA@ =BJAH AEJDAH ? KIJAHE C H IAC A J=JE H AOF E J ANJH=?JE D=I >AA ANF EJA@ E
 6DEI M H M=I F=HJE= O IKFF HJA@ >O JDA -+ K @AH ? JH=?JI .2$  % $ 5F=?A = @ .2$  %$&# -50

= O HAIA=H?D A HJI 1  = ANE? @HELA =FFH =?D EI E JH @K?A@ ) HACE >=IA@ =FFH =?D E ? JA J HA JHEAL= KIE C =JA J 5A = JE? ) = OIEI EI FHAIA JA@ E  MDAHA=I E " E =CAI =HA F=HJEJE A@ E HACE I MDE?D =HA JDA ? KIJAHA@ J >J=E = ? @A> B HA CE JOFAI = @ = >=C B HACE I =FFH =?D EI =FF EA@ B H I?A A HAFHAIA J=JE 1 LEIK= ?=JAC HE =JE EI =?DEALA@ KIE C = >=C B AOF E JI =FFH =?D .E = O E & IAF=H=JA ID=FA @AJA?J HI =HA JH=E A@ KIE C = ID=FA = FD=>AJ MDE?D EI = @E?JE =HO B ?KHLA BH=C A JI 0 MALAH JDEI CH MJD B =K@E LEIK= ? JA J M=I J =?? F= EA@ >O = IE E =H CH MJD B = J=JE I 8AHO BAM =HA JDA @=J=>=IAI JD=J FH LE@A = = J=JE FAH E =CA HACE IK?D =I =>A A JDA JDAH D= @ = J=JE C = E =CA C >= O =FFA=HI = K?D A=IEAH J=I 5K?D = AN= F A EI JDA A B 5+ M H ID F % MDAHA = DKCA K >AH B ID JI B AMI >K AJE I =HA C >= O = J=JA@ B H = =HCA K >AH B ? ?AFJI 9EJDE 64-+ ? BAHA ?A IAHEAI 64-+81, AL= K=JE ' =JJH=?JI = O HAIA=H?DAHI >O ? F=HE C JDAEH = C HEJD I E L=HE KI J=I I IK?D =I DECD ALA BA=JKHA @A JA?JE MEJDE LE@A @ ?K A JI MDAHA JDA C = EI J C >= O = J=JA LE@A ID JI B H ?AHJ=E ? ?AFJI 6DEI M H B= I MEJDE JDA I? FA B 64-+81, = @ FH LE@AI = = C HEJD J J=? A ' ? ?AFJI MEJDE JDA % @ALA F A J @=J= 6DAIA ? ?AFJI ?= J >A @AI?HE>A@ =I > A?JI >KJ H=JDAH =I =JAHE= I H I?A AI 6DKI ? H = @ JANJKHA BA=JKHAI =HA JDA O =FF E?=> A M ALA BA=JKHAI . H A=?D ? ?AFJ = IAF =H=JA AKH= AJM H >=IA@ @AJA?J H EI JH=E A@ >=IA@ BA=JKHAI ANJH=?JA@ BH AOBH= A HACE I MDE A AOBH= AI =HA = J=JA@ C >= O 6DEI F=FAH EI HC= E A@ =I B MI IA?JE FHAIA JI JDA AJD @ KIA@ B H ANJH=?JE C ? H = @ JANJKHA BA= JKHAI B = CELA AOBH= A 6DA ? IJHK?JE B JDA HACE JDAI=KHKI EI FHAIA JA@ E IA?JE ! B MA@ >O JDA B H K =JE B JDA @A LA?J HI KIA@ J @AI?HE>A
DJJF =>A A ?I=E EJ A@K

= AOBH= A E IA?JE " 6DA =JA J 5A = JE? ) = O IEI JA?D EGKA EI FHAIA JA@ E IA?JE # -NJA IELA AN FAHE A J= HAIK JI @=J= =HA FHAIA JA@ E IA?JE $ = @ ? ? KIE I =HA @H=M E IA?JE %

A=?D HACE JOFA ME 6DA JD A A A J B = @A LA? J H E @AI?HE>E C AOBH= A E EI @AFE?JA@ E AG 

E ( ) = min H 4 ( E )

@ B (M ) B (H)



M ALA .A=JKHA -NJH=?JE

)J = FHAFH ?AIIE C IJAF = CELA LE@A @ ?K A J EI HIJ IAC A JA@ E J ID JI .H A=?D ID J = HAFHA IA J=JELA BH= A  AOBH= A EI ANJH=?JA@ AJ E @A JA JDA AOBH= A B H ID J IE 5 MDAHA = @ 5 @A JA JDA IAJI B = AOBH= AI = @ ID JI HAIFA?JELA O -=?D AOBH= A EI HIJ IAC A JA@ E J HACE I KIE C = ? H 4556 IAC A J=JE = C HEJD JK A@ J FH @K?A ? =HIA HACE I AJ HE 4 @A JA A=?D B JDA HAIK JE C HACE I MDAHA 4 EI JDA IAJ B = HACE I = @ 4( E ) 4 EI JDA IAJ B = HACE I B JDA AOBH= A E . H A=?D HACE H 4 IEN 2-/ % @AI?HEFJ HI $ =HA ANJH=?JA@ , E = J + H + H 5JHK?JKHA + H =O KJ 5?= => A + H 0 CA A KI 6ANJKHA = @ -@CA 0EIJ CH= J ?=FJKHA ? H = @ JANJKHA BA= JKHAI . H = HACE H JDAIA @AI?HEFJ HI =HA AHCA@ E J = IE C A LA?J H MDE?D ME >A HABAHHA@ J =I BA=JKHA LA?J H @A JA@ >O B (H)

MDAHA E = 1 =1 6 @() @A JAI JDA -K ? E@A= @EIJ= ?A BK ?JE B (M ) = @ B (H) @A JA JDA BA=JKHA LA?J HI B = HACE JOFA M = @ = HACE H HA IFA?JELA O

=JA J 5A = JE? ) = OIEI

! 4ACE

6DAI=KHKI

)BJAH JDA ANJH=?JE B JDA BA=JKHA LA?J HI B = AOBH= A HACE I JDA =FFH =?D B  EI B MA@ E H@AH J FH LE@A = DECDAH @AI?HEFJE JD=J ME >A KIA@ B H JDA DECD ALA ? ?AFJ @AJA?JE HA IFA?E ?= O = A= I ? KIJAHE C = C HEJD EI KIA@ J ? KIJAH = HACE I @AHELA@ BH JDA AOBH= AI B JDA JH=E E C IAJ 6DA K >AH B ? KIJAHI 6 EI IA A?JA@ ANFAHE A J= O =BJAH = JHE= = @ AHH H FH ?AII 6DEI ? KIJAHE C EI =FF EA@ JDA BA=JKHA LA?J HI KI E C JDA -K? E@A= @EIJ= ?A =I JDA ? KIJAHI IE E =HEJO A=IKHA 4ACE I JD=J EA ? IAIJ J JDA ?A JH E@I B JDA HAIK JE C ? KIJAHI =HA IA A?JA@ J B H JDA HACE JDAI=KHKI 6DAIA HACE I ME E = 1 6 ME >A HA BAHHA@ J =I HACE JOFAI 1J ID K @ >A =@A ? A=H JD=J A=?D HACE JOFA @ AI J ? J=E = O DECD ALA IA = JE? E B H =JE 0 MALAH EJ EI = DECDAH @AI?HEF JE E ? F=HEI J = M ALA @AI?HEFJ H

6DA ANJ IJAF B JDA FHAIA JA@ DECD ALA ? ?AFJ @AJA?JE =FFH =?D EI JDA KIA B JDA MA M = JA J 5A = JE? ) = OIEI JA?D EGKA  5) ! E EJE= O E JH @K?A@ E JDA A @ B =JKH= = CK=CA FH ?AIIE C 5) =E I J ANF EJ JDA =JA J HA =JE I = C = IAJ B @ ?K A JI = @ JDA JAH I JDAO ? J=E 1 JDEI M H = AOBH= A ? HHAIF @I J = @ ?K A J = @ EJI IAC A JA@ HACE I ? HHAIF @ J JDA JAH I 6DA C = EI J E LAI JEC=JA D M JDAIA DE@@A HA =JE I = C HACE JOFAI =O >A ANF EJA@ J E FH LA JDA IA = JE? = = OIEI )BJAH JDA B H K =JE B JDA @A LA?J HI E = JDAEH L= KAI =HA H = E A@ I JD=J JDAO B= MEJDE [0 1] MEJD 1 @AFE?JE C JDA =NE K ? @A ?A B = HA CE JOFA J = AOBH= A 6DA H = E A@ @A LA?J HI ME >A @A JA@ =I E 6DEI M=O JDA ? ??KHHA ?A =JHEN M B AG EI B H A@ @AI?HE>E C JDA HA =JE I B HACE JOFAI J AOBH= AI
M=
1 (1)

(1) ( 6)

 

1( 6 )

HA IFA?E ?= O A=?D E A B M GE6 = ( 1 (E) (E)) @AI?HE>AI JDA HA =JE IDEF B HACE JOFA ME MEJD A=?D AOBH= A JAH LA?J H 6 ) I A=?D ? K BM =( (1) ( 6 )) ? HHAIF @I J = IFA?E ? AOBH= A @AI?HE>E C EJI HA =JE MEJD ALAHO HACE JOFA @ ?K A J LA?J H 6DKI JDA ? ??KHHA ?A =JHEN M =O >A @AI?HE>A@ KIE C JDA ANJH=?JA@  H = E A@ @A LA?J HI E =I
M=[
6
1

!

"

@A 8A?J HI

7IE C JDA HACE JDAI=KHKI JDA @EIJ= ?AI >AJMAA A=?D HACE B JDA E =CA = @ = HACE JOFAI =HA ?= ?K =JA@ 6DEI M=O = @A LA?J H JD=J IA = JE?= O @AI?HE>AI JDA LEIK= ? JA J B = E =CA EI B H A@ >O AAFE C JDA I = AIJ @EIJ= ?A B = E =CA HACE I H J

AJ GE = @ GF @A JA JM JAH LA?J HI 6DA JDAEH E AH FH @K?J GE6 GF @A JAI JDAEH ? HHA =JE 6DKI EJ =O A=IE O >IAHLA@ JD=J MM6 =?JK= O ? IEIJI B = JD IA E AH FH @K?JI HA LAH M6 M ? IEIJI B = E AH FH @K?JI >AJMAA JDA @ ?K A J LA?J HI 6 F E @AI?HE>E C JDAEH ? HHA =JE LAH JDA JAH I ) @A? F IEJE B M EI @AI?HE>A@ >O AG "
M=7

86

"

9DA 7 = @ 8 =HA HJD H = =JHE?AI = @ EI = @E=C = =JHEN 6DEI EI JDA 5E CK =H 8= KA ,A? F IEJE 58, @AFE?JA@ E AG #
M=

K

0
6

L

# 1 JDEI AG E =HA JDA IE CK =H L= KAI = @ KE LE =HA JDA IE CK =H LA?J HI B 7 = @ 8 HAIFA?JELA O *O AAFE C JDA =HCAH IE CK =H L= KAI E = C MEJD JDA ? HHAIF @E C ? K I B 7 = @ H MI B 8 = AIJE =JA B M @AI?HE>A@ E AG $ ??KHI
=M M =7

86

$

Figure 1. AP vs  for

KJ@ H,

?A and 4 =@.

6DEI M=O JDA LA?J HI B HACE JOFAI = @ AOBH= AI =HA JH= IB H A@ J JDA ? ?AFJ IF=?A ) @A LA?J H E EI JH= IB H A@ J JDA ? ?AFJ IF=?A KIE C = @ 7 =I @AFE?JA@ E AG %
E =
1

76

%

6DEI M=O = @A LA?J HI ANJH=?JA@ BH AOBH= AI B JDA JH=E E C IAJ =HA JH= IB H A@ J JDA ? ?AFJ IF=?A = @ =HA JDA KIA@ J JH=E IALAH= DECD ALA ? ?AFJ @AJA?J HI =I @AI?HE>A@ E IA?JE $

$ -NFAHE A J= 4AIK JI
6DEI IA?JE FHAIA JI JDA HAIK JI B JDA =B HA A JE A@ = C HEJD 64-+81, % ,ALA F A J ,=J= = IAJ ? IEIJE C B  LE@A I IAC A JA@ E J ID JI ) AOBH= A EI ANJH=?JA@ BH A=?D ID J HAIK JE C J = IAJ ? IEIJE C B &! AOBH= AI 6DA = J=JE KIA@ @AHELAI BH = E J A HJ  6=> A  IK = HE AI JDA ? ?AFJI JD=J =HA @AJA?JA@ = @ JDA K >AH B JDAEH F IEJELA AN= F AI ) IAF=H=JA JH=E E C = @ JAIJ E C IAJ D=I >AA CA AH=JA@ B H A=?D ? ?AFJ % B JDA F IEJELA AN= F AI M=I H= @ O IA A?JA@ B H JDA JH=E E C IAJ B A=?D ? ?AFJ = @ JDA HA =E E C ! B H JDA JAIJE C IAJ AC=JELA AN= F AI MAHA IA A?JA@ H= @ O BH JDA HA =E E C AOBH= AI )BJAH IAC A JE C ALAHO AOBH= A B JDA JH=E E C IAJ J ? =HIA HACE I = HACE JDAI=KHKI B  HACE JOFAI EI ? IJHK?JA@ >O = A= I ? KIJAHE C )BJAH ANJH=?J E C @A LA?J HI BH = E =CAI B JDA JH=E E C IAJ 5) EI =FF EA@ 6DA K >AH B JDA =HCAIJ IE CK =H L= KAI J AAF EI IAJ J 70 6DA = AKH= AJM H >=IA@ @AJA?J H EI JH=E A@ B H A=?D ? ?AFJ 1JI E FKJ EI AEJDAH = @A LA?J H E H JDA KJFKJ B JDA 5) = C HEJD E 1JI KJFKJ @A JAI JDA ? @A ?A JD=J JDA IFA?E ? ? ?AFJ ANEIJI
Figure 2. AP vs  for 8ACAJ=JE and 5

O.

.ECKHAI  = @ ! ID M JDA )LAH=CA 2HA?EIE )2 # LI JDA H=JE  B AC=JELA J F IEJELA AN= F AI 6DA K >AH B F IEJELA AN= F AI EI AFJ NA@ MDE A JDA K >AH B AC=JELA E ?HA=IAI 1J =O >A >IAHLA@ JD=J MDA  D=I = HA =JELA O I = L= KA E A  = 4 =I E JDA ?=IA B JOFE?= JAIJ IAJI JDA FAHB H = ?A B JDA ? =IIE AHI HA =E I F=HJE?K =H O DECD 9DA  E ?HA=IAI JDA JDA FAHB H = ?A B= I HA LAH MA =O >IAHLA JD=J JDA KIA B 5) @ AI J = M=OI E FH LA JDA HA IK JI . H ?AHJ=E ? ?AFJI IK?D =I KJ@ H ?A = @ 4 =@ 5) E FH LAI JDA HAIK JI MDE A  E ?HA=IAI =I @AFE?JA@ E .EC  6DEI A= I JD=J F IEJELA AN= F AI =HA @AJA?JA@ E = MAH = @ HA ? HHA?J H= 6DA ? FH FAHJO B JDAIA ? ?AFJI EI JD=J JDAO ?= J >A @AI?HE>A@ E = IJH=ECDJB HM=H@ M=O IK?D =I 8ACAJ=JE = @ 5 O @AFE?JA@ E .EC 6D=J >A? AI >LE KI MDA AN= E E C JDA 64-+81, @=J= .E = O B H JDA ? ?AFJI @AFE?JA@ E .EC ! MDAHA JDA =L=E => A K >AH B F IEJELA AN= F AI EI F=HJE?K =H O I = KIE C 5) E FH LAI O JDA IA = JE? ? ?AFJ 5 M

? ?AFJ ?E

8ACAJ=JE 4 =@ -NF IE .EHA 5 O 5 M ?A ,AIAHJ KJ@ H K J=E

K >AH B F IEJELAI '!' ' ! ' "$  "' # #&# '%

 $"!  '#  '  #%  %%%  ""$  !!! " #  """

*AB HA 5)  !  "$  %%%  !"  "  #%  !  #"  !%

 "

 "$  &  &  "!$  "$  !&  &%  !$  "

)2

$ $  "    ##'  &&  "$  #  !!  

)BJAH 5)    #    %  $"  "%  $&%  $!"  !#

 !'#    "&  !% # '  &#  "$  !&  %

)2

!  "#    #&  !  %  !  $&!  !

*AB HA 5)  !!  "%    !"  "  #%  !!  #  !%'

=N  $&  !$    &&  !  '#    $  !

 !  "!    "  &  %  $"  ##  !%

)2

)BJAH 5)   #    %  $#  "&  "!&  $"$  %

 %'  ""    &"      $! #  

)2

Table 1. Experimental results for all high-level concepts.

2=precision, 4=recall, )2=average precision.

IA = JE? = = OIEI KH = B JDA 5 ?EAJO B H 1 B H =JE 5?EA ?A "$ !' "% '' , / = F = @ 5 ) I O 5?A A ? =IIE ?=JE KI E C >=C B HACE I HAFHAIA J=JE I 1 1--- + BAHA ?A + FKJAH 8EIE = @ 2=JJAH 4A? C E JE % EIDE@= 2H FAHJO B =LAH=CA FHA?EIE = @ EJI CA AH= E =JE = AN= E =JE B AL= K=JE E @E?=J H B H E B H =JE HAJHEAL= 11 6A?D E?= 4AF HJI 11 # "- # * = K =JD D 8 8=IK@AL= = @ ) ;= =@= + H = @ JANJKHA @AI?HEFJ HI 1--- JH= I +EH?KEJI = @ 5OIJA I B H 8E@A 6A?D CO $ %! %#  4 =FD=@A A A@O 4 A @AH 5 . +D= C 4 5 EJD 2 LAH = @ ) 0=KFJ = ) ECDJ 5?= A + ?AFJ J CO B H K JE A@E= K @AHIJ= @E C B H JHA?LE@ # 1* 4AIA=H?D 6A?D E?= 4AF HJ # ) FA J ) 2E = @ ) EIIAH = 1 ?HA A J= A=H E C B > A?J @AJA?J HI KIE C = LEIK= ID=FA = FD=>AJ 1 1--- + FKJAH 5 ?EAJO + BAHA ?A + FKJAH 8EIE = @ 2=JJAH 4A? C EJE $ ) . 5 A=J = @ 2 LAH = @ 9 H==E -L= K =JE ?= F=EC I = @ 64-+8E@ 1 14 $ 2H ?AA@E CI B JDA &JD )+ 1 JAH =JE = 9 H ID F K JE A@E= 1 B H =JE 4AJHEAL= $ + 5 A 9 HHE C , A = = @ ) 5 AK @AHI ) A=H A@ ANE? @HELA F=H=@EC B H E JAH =?JELA LE@A HAJHEAL= 1--- 6H= I K JE A@E= '  & ' % . 5 KL= =L C * AHE= @ = @ * 0KAJ 4ACE >=IA@ LE@A ? JA J E @ANE C = @ HAJHEAL= 1 "JD 1 JAH =JE = 9 H ID F + JA J *=IA@ K JE A@E= 1 @ANE C # - 5FOH K = @ ; )LHEJDEI ) HACE JDAI=KHKI =F FH =?D B H DECD ALA ? ?AFJ @AJA?JE E JDA =J KH= @EI=IJAH @ =E 1 @ 1 JAH =JE = + BAH A ?A 5A = JE?I ) @ @ECEJ= A@E= 6A?D CEAI 5) 6 %

"

Figure 3. AP vs and K J=E .

 for ,AIAHJ, -NF IE

.EHA, 5 M

% + ? KIE I = @ .KJKHA 9 H
1 JDEI F=FAH MA FHAIA JA@ KH FH F I= J M=H@I AB ?EA J IA = JE? K JE A@E= = = OIEI >=IA@ = HACE JDAI=KHKI = @ 5) 9A KIA@ = ? KIJAHA@ IAJ B HACE JOFAI J FH @K?A = @A LA?J H = @ JDKI =F M ALA BA=JKHAI J DECD ALA ? ?AFJI 5) M=I JDA =FF EA@ JDA =JJAH KIE C JDA 64-+81, % ,ALA F A J ,=J= 9EJDE KH BKJKHA C = I =HA JDA KJE E =JE B ? JANJK= E B H =JE E JDA FH ?AII = @ JDA ANJA IE B JDA FH F IA@ AJD @ CO J IKFF HJ @AJA?JE B = =HCA K >AH B ? ?AFJI

&

'



4ABAHA ?AI
 5 )O=?DA = @ / 3KA J 64-+81, % ? => H=JELA = J=JE KIE C =?JELA A=H E C 1 64-+81, % 9 H ID F /=EJDAHI>KHC + ,= ?A 9E = MI E .= + *H=O = @ / +IKH = 8EIK= ?=JAC HE =JE MEJD >=CI B AO F E JI 1 -++8 1 JAH =JE = 9 H ID F 5J=JEIJE?= A=H E C E + FKJAH 8EIE " ! 5 ,AAHMAIJAH 5 ,K =EI / 9 .KH =I 6 = @=KAH = @ 4 0=HID = 1 @ANE C >O =JA J