Introduction to ILP

ILP = Inductive Logic Programming = machine learning logic programming = learning with logic
Introduced by Muggleton in 1992

(Machine) Learning
he proce!! by which relatively permanent change! occur in behavioral potential a! a re!ult o" e#perience$ (%nder!on) Learning i! con!tructing or modi"ying repre!entation! o" what i! being e#perienced$ (Michal!&i) % computer program i! !aid to learn "rom e#perience E with re!pect to !ome cla!! o" ta!&! T and per"ormance mea!ure P' i" it! per"ormance at ta!&! in T' a! mea!ured by P' improve! with e#perience E$ (Mitchell)

Machine Learning echni(ue!

)eci!ion tree learning *onceptual clu!tering *a!e+ba!ed learning ,ein"orcement learning -eural networ&! .enetic algorithm! and/ Inductive Logic Programming

0hy ILP 1 + 2tructured data

2eed e#ample o" 3a!t+0e!t train! (Michal!&i)

What makes a train to go eastward ?

0hy ILP 1 4 2tructured data

Mutagenicity o" chemical molecule! (5ing' 2riniva!an' Muggleton' 2ternberg' 1996)

0hat ma&e! a molecule to be mutagenic 1

0hy ILP 1 4 multiple relation!

hi! i! related to !tructured data
ha!8car
rain t1 t1 t1 t1 t2 / *ar c11 c12 c17 c16 c21 /

car8propertie!
*ar c11 c12 c17 c16 c21 / Length !hort long !hort long !hort / 2hape %#e! ,oo" none none pea&ed none "lat / / / / / / / / rectangle 2 rectangle 7 rectangle 2 rectangle 2 rectangle 2 / /

0hy ILP 1 4 multiple relation!

.enealogy e#ample9 .iven &nown relation!/
father(Old,Young) and mother(Old,Young) male(Somebody) and female(Somebody)

/learn new relation!

parent(X,Y) :- father(X,Y). parent(X,Y) :- mother(X,Y). brother(X,Y) :male(X),father(Z,X),father(Z,Y).

Mo!t ML techni(ue! can:t u!e more than 1 relation e$g$9 deci!ion tree!' neural networ&!' /

0hy ILP 1 4 logical "oundation

Prolog = Programming with Logic i! u!ed to repre!ent9
4 ;ac&ground &nowledge (o" the domain)9 facts 4 3#ample! (o" the relation to be learned)9 facts 4 heorie! (a! a re!ult o" learning)9 rules

2upport! 2 "orm! o" logical rea!oning

4 )eduction 4 Induction

Prolog + de"inition!
<ariable!9 X, Y, Something, Somebody erm!9 arthur, 1, 1,!,"# Predicate!9 father$!, female$1 =act!9
father(%hri&topher,'i%toria). female('i%toria).

,ule!9
parent(X,Y) :- father(X,Y).

Logical rea!oning9 deduction

=rom rule! to "act!/ ; >+ 3

mother(penelope'victoria)$ mother(penelope'arthur)$ "ather(chri!topher'victoria)$ "ather(chri!topher'arthur)$

parent(penelope'victoria)$ parent(penelope'arthur)$ parent(chri!topher'victoria)$ parent(chri!topher'arthur)$

parent(?'@) 9+ "ather(?'@)$ parent(?'@) 9+ mother(?'@)$

Logical rea!oning9 induction

=rom "act! to rule!/ ; 3 >+

mother(penelope'victoria)$ mother(penelope'arthur)$ "ather(chri!topher'victoria)$ "ather(chri!topher'arthur)$

parent(penelope'victoria)$ parent(penelope'arthur)$ parent(chri!topher'victoria)$ parent(chri!topher'arthur)$ parent(?'@) 9+ "ather(?'@)$ parent(?'@) 9+ mother(?'@)$

Induction o" a cla!!i"ier or *oncept Learning

Most studied task in Machine Learning Given9
4 bac&ground &nowledge B 4 a !et o" training e#ample! E 4 a cla!!i"ication c C "or each e#ample e

Find9 a theory T (or hypothesis) !uch that ; >+ c(e)' "or all e 3

Induction o" a cla!!i"ier9 e#ample

3#ample o" 3a!t+0e!t train! B9 relation! ha&(%ar and %ar(propertie&
(length, roof, &hape, etc$) e#$9 ha&(%ar(t1,%11), &hape(%11,bu%)et) E9 the train! t1 to t1* C9 ea&t, +e&t

0hy ILP 1 + 2tructured data

2eed e#ample o" 3a!t+0e!t train! (Michal!&i)

What makes a train to go eastward ?

Induction o" a cla!!i"ier9 e#ample

3#ample o" 3a!t+0e!t train! B9 relation! ha&(%ar and %ar(propertie&
(length, roof, &hape, etc$) e#$9 ha&(%ar(t1,%11) E9 the train! t1 to t1* C9 ea&t, +e&t Po!!ible T9
ea&t(,) :ha&(%ar(,,-), length(-,&hort), roof(-,().

Induction o" a cla!!i"ier9 e#ample

3#ample o" mutagenicity
B9 relation! atom and bond e#$9 atom(mol!",atom1,%,1./).
bond(mol!",atom1,atom",0).

E9 27A molecule! with &nown cla!!i"ication C9 a%ti'e and nona%ti'e w$r$t$ mutagenicity Po!!ible T9
a%ti'e(1ol) :atom(1ol,2,%,!!), atom(1ol,3,%,1*), bond(1ol,2,3,1).

%!!

%1*

Learning a! !earch
Given9
4 4 4 4 4 ;ac&ground &nowledge ; heory )e!cription Language T Po!itive! e#ample! P (cla!! B) -egative e#ample! - (cla!! +) % covering relation covers(B,T,e)

Find9 a theory that cover!

4 all po!itive e#ample! (completene!!) 4 no negative e#ample! (con!i!tency)

Learning a! !earch
*overing relation in ILP cover!(;' 'e) ; >+ e % theory i! a !et o" rule! 3ach rule i! !earched !eparately (e""iciency) % rule mu!t be con!i!tent (cover no negative!)' but not nece!!ary complete Separate-and-conquer !trategy
4 ,emove "rom P the e#ample! already covered

2pace e#ploration
2trategy1 ,andom wal&
4 ,edundancy' incompletene!! o" the !earch

2y!tematic according to !ome ordering

4 ;etter control =C no redundancy' completene!! 4 he ordering may be u!ed to guide the !earch toward! better rule!

What kind o ordering?

.enerality ordering
,ule 1 i! more general than rule 2 =C ,ule 1 cover! more e#ample! than rule 2
4 I" a rule i! con!i!tent (cover! no negative!) then every !peciali!ation o" it i! con!i!tent too 4 I" a rule i! complete (cover! all po!itive!) then every generali!ation o" it i! complete too

Mean! to prune the !earch !pace 2 &ind! o" move!9 !peciali!ation and generali!ation *ommon ILP ordering9 D+!ub!umption

.enerality ordering
parent(X,Y):-

parent(X,Y):- female(X)

parent(X,Y) :- father(X,Y)

parent(X,Y) :- female(X), father(X,Y) parent(X,Y) :- female(X), mother(X,Y)

&pe%iali&ation

%on&i&tent rule

2earch bia!e!
!"ias re ers to any criterion or choosing one genera#i$ation over another other than strict consistency with the o%served training instances&' (Mitche##)

,e!trict the !earch !pace (e""iciency) .uide the !earch (given domain &nowledge) )i""erent &ind! o" bia!
4 Language bia! 4 2earch bia! 4 2trategy bia!

Language bia!
*hoice o" predicate!9
roof(-,flat) 4 roof(-) 4 flat(-) 4

ype! o" predicate! 9

ea&t(,) :- roof(,), roof(-,")

Mode! o" predicate! 9

ea&t(,) :- roof(-,flat) ea&t(,) :- ha&(%ar(,,-), roof(-,flat)

)i!cretiEation o" numerical value!

2earch bia!
he move! direction in the !earch !pace op+down
4 !tart9 the empty rule (c(?) 9+ $) 4 move!9 !peciali!ation!

;ottom+up
4 !tart9 the bottom clau!e (F c(?) 9+ ;$) 4 move!9 generali!ation!

;i+directional

2trategy bia!
Geuri!tic !earch "or a be!t rule Gill+climbing9
4 5eep only one rule 4 e""icient but can mi!! global ma#imum

;eam !earch9
4 al!o &eep k rule! "or bac&+trac&ing 4 le!! greedy

;e!t+"ir!t !earch9
4 &eep all rule! 4 more co!tly but complete !earch

% generic ILP algorithm

procedure 567(Examples) 5nitiali8e(Rules, Examples) repeat R 9 Sele%t(Rules, Examples) Rs 9 :efine(R, Examples) Rules 9 :edu%e(Rules;Rs, Examples) until Stopping-riterion(Rules, Examples) return(Rules)

% generic ILP algorithm

< 5nitiali8e(Rules,Examples): < < <

initialiEe a !et o" theorie! a! the !earch !tarting point! Sele%t(Rules,Examples): !elect the mo!t promi!ing candidate rule R :efine(R,Examples): return! the neighbour! o" R (u!ing !peciali!ation or generali!ation) :edu%e(Rules,Examples): di!card unpromi!ing theorie! (all but one in hill+climbing' none in be!t+ "ir!t !earch)

ILPnet2 4 www$c!$bri!$ac$u&HFILPnet2H
-etwor& o" 3#cellence in ILP in 3urope 7I univer!itie! and re!earch in!titute! 3ducational material! Publication! 3vent! (con"erence!' !ummer !chool!' /) )e!cription o" ILP !y!tem! %pplication!

ILP !y!tem!
=JIL (Kuinlan and *ameron+Lone! 1997)9 top+down hill+climbing !earch Progol (Muggleton' 199M)9 top+down be!t+"ir!t !earch with bottom clau!e .olem (Muggleton and =eng 1992)9 bottom+up hill+ climbing !earch LI-N2 (Lavrac and )Eero!&i 1996)9 propo!itionali!ation %leph (FProgol)' ilde (relational deci!ion tree!)' /

ILP application!
Li"e !cience!
4 mutagenecity' predicting to#icology 4 protein !tructureH"olding

-atural language proce!!ing

4 engli!h verb pa!t ten!e 4 document analy!i! and cla!!i"ication

3ngineering
4 "inite element me!h de!ign

3nvironmental !cience!
4 biodegradability o" chemical compound!

he end
% "ew boo&! on ILP/ L$ Lloyd$ Logic "or learning9 learning comprehen!ible theorie! "rom !tructured data$ 2AA7$ 2$ )Eero!&i and -$ Lavrac' editor!$ ,elational )ata Mining$ 2eptember 2AA1$ L$ )e ,aedt' editor$ %dvance! in Inductive Logic Programming$ 199O$ -$ Lavrac and 2$ )Eero!&i$ Inductive Logic Programming9 echni(ue! and %pplication!$ 1996$