01 Intro

Introduction
Inverted index
Processing Boolean queries
Query optimization
Course overview
Introduction to Information Retrieval

http://informationretrieval.org IIR 1: Boolean Retrieval
Hinrich Schtze u
Institute for Natural Language Processing, University of Stuttgart
2011-10-18
Schtze: Boolean Retrieval u
1 / 62
Introduction
Inverted index
Query optimization
Course overview
Take-away
2 / 62
Introduction
Inverted index
Query optimization
Course overview
Take-away
Boolean Retrieval: Design and data structures of a simple information retrieval system
2 / 62
Introduction
Inverted index
Query optimization
Course overview
Take-away
Boolean Retrieval: Design and data structures of a simple information retrieval system What topics will be covered in this class?
2 / 62
Introduction
Inverted index
Query optimization
Course overview
Outline
Introduction Inverted index Processing Boolean queries Query optimization Course overview
3 / 62
Introduction
Inverted index
Query optimization
Course overview
Denition of information retrieval
Information retrieval (IR) is nding material (usually documents) of an unstructured nature (usually text) that satises an information need from within large collections (usually stored on computers).
4 / 62
Introduction
Inverted index
Query optimization
Course overview
4 / 62
Introduction
Inverted index
Query optimization
Course overview
4 / 62
Introduction
Inverted index
Query optimization
Course overview
4 / 62
Introduction
Inverted index
Query optimization
Course overview
4 / 62
Introduction
Inverted index
Query optimization
Course overview
4 / 62
Introduction
Inverted index
Query optimization
Course overview
4 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base an information retrieval system on.
7 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base an information retrieval system on. Queries are Boolean expressions, e.g., Caesar and Brutus
7 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base an information retrieval system on. Queries are Boolean expressions, e.g., Caesar and Brutus The seach engine returns all documents that satisfy the Boolean expression.
7 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean retrieval
The Boolean model is arguably the simplest model to base an information retrieval system on. Queries are Boolean expressions, e.g., Caesar and Brutus The seach engine returns all documents that satisfy the Boolean expression.
Does Google use the Boolean model?
7 / 62
Introduction
Inverted index
Query optimization
Course overview
8 / 62
Introduction
Inverted index
Query optimization
Course overview

On Google, the default interpretation of a query [w1 w2 . . . wn ] is w1 AND w2 AND . . . AND wn
8 / 62
Introduction
Inverted index
Query optimization
Course overview

On Google, the default interpretation of a query [w1 w2 . . . wn ] is w1 AND w2 AND . . . AND wn Cases where you get hits that do not contain one of the wi :
8 / 62
Introduction
Inverted index
Query optimization
Course overview

anchor text
8 / 62
Introduction
Inverted index
Query optimization
Course overview

anchor text page contains variant of wi (morphology, spelling correction, synonym)
8 / 62
Introduction
Inverted index
Query optimization
Course overview

anchor text page contains variant of wi (morphology, spelling correction, synonym) long queries (n large)
8 / 62
Introduction
Inverted index
Query optimization
Course overview

anchor text page contains variant of wi (morphology, spelling correction, synonym) long queries (n large) boolean expression generates very few hits
8 / 62
Introduction
Inverted index
Query optimization
Course overview

Simple Boolean vs. Ranking of result set
8 / 62
Introduction
Inverted index
Query optimization
Course overview


Simple Boolean retrieval returns matching documents in no particular order.
8 / 62
Introduction
Inverted index
Query optimization
Course overview


Simple Boolean retrieval returns matching documents in no particular order. Google (and most well designed Boolean engines) rank the result set they rank good hits (according to some estimator of relevance) higher than bad hits.
Schtze: Boolean Retrieval u 8 / 62
Introduction
Inverted index
Query optimization
Course overview
Outline
9 / 62
Introduction
Inverted index
Query optimization
Course overview
Unstructured data in 1650: Shakespeare
10 / 62
Introduction
Inverted index
Query optimization
Course overview
Unstructured data in 1650
Which plays of Shakespeare contain the words Brutus and Caesar, but not Calpurnia?
11 / 62
Introduction
Inverted index
Query optimization
Course overview
Which plays of Shakespeare contain the words Brutus and Caesar, but not Calpurnia? One could grep all of Shakespeares plays for Brutus and Caesar, then strip out lines containing Calpurnia.
11 / 62
Introduction
Inverted index
Query optimization
Course overview
Which plays of Shakespeare contain the words Brutus and Caesar, but not Calpurnia? One could grep all of Shakespeares plays for Brutus and Caesar, then strip out lines containing Calpurnia. Why is grep not the solution?
11 / 62
Introduction
Inverted index
Query optimization
Course overview
Slow (for large collections)
11 / 62
Introduction
Inverted index
Query optimization
Course overview
Slow (for large collections) grep is line-oriented, IR is document-oriented
11 / 62
Introduction
Inverted index
Query optimization
Course overview
Slow (for large collections) grep is line-oriented, IR is document-oriented not Calpurnia is non-trivial
11 / 62
Introduction
Inverted index
Query optimization
Course overview
Slow (for large collections) grep is line-oriented, IR is document-oriented not Calpurnia is non-trivial Other operations (e.g., nd the word Romans near countryman) not feasible
11 / 62
Introduction
Inverted index
Query optimization
Course overview
Term-document incidence matrix

Anthony and Cleopatra 1 1 1 0 1 1 1 Julius Caesar The Tempest Hamlet Othello Macbeth ...
Anthony 1 0 0 0 1 Brutus 1 0 1 0 0 Caesar 1 0 1 1 1 Calpurnia 1 0 0 0 0 Cleopatra 0 0 0 0 0 mercy 0 1 1 1 1 worser 0 1 1 1 0 ... Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar. Entry is 0 if term doesnt occur. Example: Calpurnia doesnt occur in The tempest.
12 / 62
Introduction
Inverted index
Query optimization
Course overview

Anthony 1 0 0 0 1 Brutus 1 0 1 0 0 Caesar 1 0 1 1 1 1 0 0 0 0 Calpurnia Cleopatra 0 0 0 0 0 mercy 0 1 1 1 1 worser 0 1 1 1 0 ... Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar. Entry is 0 if term doesnt occur. Example: Calpurnia doesnt occur in The tempest.
12 / 62
Introduction
Inverted index
Query optimization
Course overview

Anthony 1 0 0 0 1 Brutus 1 0 1 0 0 Caesar 1 0 1 1 1 0 0 0 0 Calpurnia 1 Cleopatra 0 0 0 0 0 mercy 0 1 1 1 1 worser 0 1 1 1 0 ... Entry is 1 if term occurs. Example: Calpurnia occurs in Julius Caesar. Entry is 0 if term doesnt occur. Example: Calpurnia doesnt occur in The tempest.
12 / 62
Introduction
Inverted index
Query optimization
Course overview
Incidence vectors
So we have a 0/1 vector for each term. To answer the query Brutus and Caesar and not Calpurnia:
13 / 62
Introduction
Inverted index
Query optimization
Course overview
Incidence vectors
So we have a 0/1 vector for each term. To answer the query Brutus and Caesar and not Calpurnia:
Take the vectors for Brutus, Caesar, and Calpurnia Complement the vector of Calpurnia Do a (bitwise) and on the three vectors 110100 and 110111 and 101111 = 100100
13 / 62
Introduction
Inverted index
Query optimization
Course overview
0/1 vector for Brutus

Anthony and Cleopatra 1 1 1 0 1 1 1 1 Julius Caesar 1 1 1 1 0 0 0 0 The Tempest 0 0 0 0 0 1 1 0 Hamlet Othello Macbeth ...
Anthony Brutus Caesar Calpurnia Cleopatra mercy worser ... result:
0 1 1 0 0 1 1 1
0 0 1 0 0 1 1 0
1 0 1 0 0 1 0 0
14 / 62
Introduction
Inverted index
Query optimization
Course overview
Answers to query
Anthony and Cleopatra, Act III, Scene ii Agrippa [Aside to Domitius Enobarbus]: Why, Enobarbus, When Antony found Julius Caesar dead, He cried almost to roaring; and he wept When at Philippi he found Brutus slain. Hamlet, Act III, Scene ii Lord Polonius: I did enact Julius Caesar: I was killed i the Capitol; Brutus killed me.
15 / 62
Introduction
Inverted index
Query optimization
Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens
16 / 62
Introduction
Inverted index
Query optimization
Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens total of 109 tokens
16 / 62
Introduction
Inverted index
Query optimization
Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens total of 109 tokens On average 6 bytes per token, including spaces and punctuation size of document collection is about 6 109 = 6 GB
16 / 62
Introduction
Inverted index
Query optimization
Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens total of 109 tokens On average 6 bytes per token, including spaces and punctuation size of document collection is about 6 109 = 6 GB Assume there are M = 500,000 distinct terms in the collection
16 / 62
Introduction
Inverted index
Query optimization
Course overview
Bigger collections
Consider N = 106 documents, each with about 1000 tokens total of 109 tokens On average 6 bytes per token, including spaces and punctuation size of document collection is about 6 109 = 6 GB Assume there are M = 500,000 distinct terms in the collection (Notice that we are making a term/token distinction.)
16 / 62
Introduction
Inverted index
Query optimization
Course overview
Cant build the incidence matrix
M = 500,000 106 = half a trillion 0s and 1s.
17 / 62
Introduction
Inverted index
Query optimization
Course overview
M = 500,000 106 = half a trillion 0s and 1s. But the matrix has no more than one billion 1s.
17 / 62
Introduction
Inverted index
Query optimization
Course overview
Matrix is extremely sparse.
17 / 62
Introduction
Inverted index
Query optimization
Course overview
What is a better representations?
17 / 62
Introduction
Inverted index
Query optimization
Course overview
What is a better representations?

We only record the 1s.
17 / 62
Introduction
Inverted index
Query optimization
Course overview
Inverted Index
For each term t, we store a list of all documents that contain t. Brutus Caesar Calpurnia . . . dictionary postings 1 1 2 2 2 31 4 4 54 11 5 101 31 6 45 16 173 57 174 132 ...
18 / 62
Introduction
Inverted index
Query optimization
Course overview
Inverted Index
18 / 62
Introduction
Inverted index
Query optimization
Course overview
Inverted Index
18 / 62
Introduction
Inverted index
Query optimization
Course overview
Inverted index construction
Collect the documents to be indexed: Friends, Romans, countrymen. So let it be with Caesar . . . Tokenize the text, turning each document into a list of tokens: Friends Romans countrymen So . . . Do linguistic preprocessing, producing a list of normalized tokens, which are the indexing terms: friend roman countryman so . . . Index the documents that each term occurs in by creating an inverted index, consisting of a dictionary and postings.
19 / 62
Introduction
Inverted index
Query optimization
Course overview
Tokenization and preprocessing

Doc 1. I did enact Julius Caesar: I was killed i the Capitol; Brutus killed me. Doc 2. So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious:
Doc 1. i did enact julius caesar i was killed i the capitol brutus killed me Doc 2. so let it be with caesar the noble brutus hath told you caesar was ambitious
20 / 62
Introduction
Inverted index
Query optimization
Course overview
Generate postings
term docID i 1 did 1 enact 1 julius 1 caesar 1 i 1 was 1 killed 1 i 1 the 1 capitol 1 brutus 1 killed 1 me 1 so 2 let 2 it 2 be 2 with 2 caesar 2 the 2 noble 2 brutus 2 hath 2 told 2 you 2 caesar 2 was 2 ambitious 2
Doc 1. i did enact julius caesar i was killed i the capitol brutus killed me Doc 2. so let it be with caesar the noble brutus hath told you caesar was ambitious
21 / 62
Introduction
Inverted index
Query optimization
Course overview
Sort postings
term docID i 1 did 1 enact 1 julius 1 caesar 1 i 1 was 1 killed 1 i 1 the 1 capitol 1 brutus 1 killed 1 me 1 so 2 let 2 it 2 be 2 with 2 caesar 2 the 2 noble 2 brutus 2 hath 2 told 2 you 2 caesar 2 was 2 ambitious 2 term docID ambitious 2 be 2 brutus 1 brutus 2 capitol 1 caesar 1 caesar 2 caesar 2 did 1 enact 1 hath 1 i 1 i 1 i 1 it 2 julius 1 killed 1 killed 1 let 2 me 1 noble 2 so 2 the 1 the 2 told 2 you 2 was 1 was 2 with 2
22 / 62
Introduction
Inverted index
Query optimization
Course overview
Create postings lists, determine document frequency

term docID ambitious 2 be 2 brutus 1 brutus 2 capitol 1 caesar 1 caesar 2 caesar 2 did 1 enact 1 hath 1 i 1 i 1 i 1 it 2 julius 1 killed 1 killed 1 let 2 me 1 noble 2 so 2 the 1 the 2 told 2 you 2 was 1 was 2 with 2 term doc. freq. ambitious 1 be 1 brutus 2 capitol 1 caesar 2 did 1 enact 1 hath 1 i 1 i 1 it 1 julius 1 killed 1 let 1 me 1 noble 1 so 1 the 2 told 1 you 1 was 2 with 1 postings lists 2 2 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 2 2 1 2 2 2 1 2 2
23 / 62
Introduction
Inverted index
Query optimization
Course overview
Split the result into dictionary and postings le
Brutus Caesar Calpurnia . . . dictionary
1 1 2
2 2 31
4 4 54
11 5 101
31 6
45 16
173 57
174 132 ...
postings le
24 / 62
Introduction
Inverted index
Query optimization
Course overview
Later in this course
Index construction: how can we create inverted indexes for large collections?
25 / 62
Introduction
Inverted index
Query optimization
Course overview
Index construction: how can we create inverted indexes for large collections? How much space do we need for dictionary and index?
25 / 62
Introduction
Inverted index
Query optimization
Course overview
Index construction: how can we create inverted indexes for large collections? How much space do we need for dictionary and index? Index compression: how can we eciently store and process indexes for large collections?
25 / 62
Introduction
Inverted index
Query optimization
Course overview
Index construction: how can we create inverted indexes for large collections? How much space do we need for dictionary and index? Index compression: how can we eciently store and process indexes for large collections? Ranked retrieval: what does the inverted index look like when we want the best answer?
25 / 62
Introduction
Inverted index
Query optimization
Course overview
Outline
26 / 62
Introduction
Inverted index
Query optimization
Course overview
Simple conjunctive query (two terms)
Consider the query: Brutus AND Calpurnia
27 / 62
Introduction
Inverted index
Query optimization
Course overview
Consider the query: Brutus AND Calpurnia To nd all matching documents using inverted index:
27 / 62
Introduction
Inverted index
Query optimization
Course overview
1
Locate Brutus in the dictionary
27 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2
Locate Brutus in the dictionary Retrieve its postings list from the postings le
27 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 3
Locate Brutus in the dictionary Retrieve its postings list from the postings le Locate Calpurnia in the dictionary
27 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 3 4
Locate Brutus in the dictionary Retrieve its postings list from the postings le Locate Calpurnia in the dictionary Retrieve its postings list from the postings le
27 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 3 4 5
Locate Brutus in the dictionary Retrieve its postings list from the postings le Locate Calpurnia in the dictionary Retrieve its postings list from the postings le Intersect the two postings lists
27 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 3 4 5 6
Locate Brutus in the dictionary Retrieve its postings list from the postings le Locate Calpurnia in the dictionary Retrieve its postings list from the postings le Intersect the two postings lists Return intersection to user
27 / 62
Introduction
Inverted index
Query optimization
Course overview
Intersecting two postings lists
Brutus Calpurnia Intersection
1 2 4 11 31 45 173 174 2 31 54 101
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2 31
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2 31
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2 31
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2 31
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2 31
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2 31
This is linear in the length of the postings lists.
28 / 62
Introduction
Inverted index
Query optimization
Course overview
1 2 4 11 31 45 173 174 2 31 54 101 2 31
This is linear in the length of the postings lists. Note: This only works if postings lists are sorted.
28 / 62
Introduction
Inverted index
Query optimization
Course overview

Intersect(p1 , p2 ) 1 answer 2 while p1 = nil and p2 = nil 3 do if docID(p1 ) = docID(p2 ) 4 then Add(answer , docID(p1 )) 5 p1 next(p1 ) 6 p2 next(p2 ) 7 else if docID(p1 ) < docID(p2 ) 8 then p1 next(p1 ) 9 else p2 next(p2 ) 10 return answer
29 / 62
Introduction
Inverted index
Query optimization
Course overview
Query processing: Exercise

france paris lear 1 2 3 4 5 7 8 9 11 12 13 14 15 2 6 10 12 14 12 15
Compute hit list for ((paris AND NOT france) OR lear)
30 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
The Boolean retrieval model can answer any query that is a Boolean expression.
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
Boolean queries are queries that use and, or and not to join query terms.
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
Boolean queries are queries that use and, or and not to join query terms. Views each document as a set of terms.
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
Boolean queries are queries that use and, or and not to join query terms. Views each document as a set of terms. Is precise: Document matches condition or not.
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
Primary commercial retrieval tool for 3 decades
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
Primary commercial retrieval tool for 3 decades Many professional searchers (e.g., lawyers) still like Boolean queries.
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
You know exactly what you are getting.
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Boolean queries
You know exactly what you are getting.
Many search systems you use are also Boolean: spotlight, email, intranet etc.
31 / 62
Introduction
Inverted index
Query optimization
Course overview
Commercially successful Boolean retrieval: Westlaw
Largest commercial legal search service in terms of the number of paying subscribers
32 / 62
Introduction
Inverted index
Query optimization
Course overview
Largest commercial legal search service in terms of the number of paying subscribers Over half a million subscribers performing millions of searches a day over tens of terabytes of text data
32 / 62
Introduction
Inverted index
Query optimization
Course overview
Largest commercial legal search service in terms of the number of paying subscribers Over half a million subscribers performing millions of searches a day over tens of terabytes of text data The service was started in 1975.
32 / 62
Introduction
Inverted index
Query optimization
Course overview
Largest commercial legal search service in terms of the number of paying subscribers Over half a million subscribers performing millions of searches a day over tens of terabytes of text data The service was started in 1975. In 2005, Boolean search (called Terms and Connectors by Westlaw) was still the default, and used by a large percentage of users . . .
32 / 62
Introduction
Inverted index
Query optimization
Course overview
Largest commercial legal search service in terms of the number of paying subscribers Over half a million subscribers performing millions of searches a day over tens of terabytes of text data The service was started in 1975. In 2005, Boolean search (called Terms and Connectors by Westlaw) was still the default, and used by a large percentage of users . . . . . . although ranked retrieval has been available since 1992.
32 / 62
Introduction
Inverted index
Query optimization
Course overview
Westlaw: Example queries

Information need: Information on the legal theories involved in preventing the disclosure of trade secrets by employees formerly employed by a competing company Query: trade secret /s disclos! /s prevent /s employe!
33 / 62
Introduction
Inverted index
Query optimization
Course overview
Information need: Requirements for disabled people to be able to access a workplace Query: disab! /p access! /s work-site work-place (employment /3 place)
33 / 62
Introduction
Inverted index
Query optimization
Course overview
Information need: Cases about a hosts responsibility for drunk guests Query: host! /p (responsib! liab!) /p (intoxicat! drunk!) /p guest
33 / 62
Introduction
Inverted index
Query optimization
Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within a sentence, /p = within a paragraph
34 / 62
Introduction
Inverted index
Query optimization
Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within a sentence, /p = within a paragraph Space is disjunction, not conjunction! (This was the default in search pre-Google.)
34 / 62
Introduction
Inverted index
Query optimization
Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within a sentence, /p = within a paragraph Space is disjunction, not conjunction! (This was the default in search pre-Google.) Long, precise queries: incrementally developed, not like web search
34 / 62
Introduction
Inverted index
Query optimization
Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within a sentence, /p = within a paragraph Space is disjunction, not conjunction! (This was the default in search pre-Google.) Long, precise queries: incrementally developed, not like web search Why professional searchers often like Boolean search: precision, transparency, control
34 / 62
Introduction
Inverted index
Query optimization
Course overview
Westlaw: Comments
Proximity operators: /3 = within 3 words, /s = within a sentence, /p = within a paragraph Space is disjunction, not conjunction! (This was the default in search pre-Google.) Long, precise queries: incrementally developed, not like web search Why professional searchers often like Boolean search: precision, transparency, control When are Boolean queries the best way of searching? Depends on: information need, searcher, document collection, . . .
34 / 62
Introduction
Inverted index
Query optimization
Course overview
Outline
35 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Consider a query that is an and of n terms, n > 2
36 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Consider a query that is an and of n terms, n > 2 For each of the terms, get its postings list, then and them together
36 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Consider a query that is an and of n terms, n > 2 For each of the terms, get its postings list, then and them together Example query: Brutus AND Calpurnia AND Caesar
36 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Consider a query that is an and of n terms, n > 2 For each of the terms, get its postings list, then and them together Example query: Brutus AND Calpurnia AND Caesar What is the best order for processing this query?
36 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar
37 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar Simple and eective optimization: Process in order of increasing frequency
37 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar Simple and eective optimization: Process in order of increasing frequency Start with the shortest postings list, then keep cutting further
37 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar Simple and eective optimization: Process in order of increasing frequency Start with the shortest postings list, then keep cutting further
Brutus Calpurnia Caesar
1 2 4 11 31 45 173 174 2 31 54 101 5 31
37 / 62
Introduction
Inverted index
Query optimization
Course overview
Query optimization
Example query: Brutus AND Calpurnia AND Caesar Simple and eective optimization: Process in order of increasing frequency Start with the shortest postings list, then keep cutting further In this example, rst Caesar, then Calpurnia, then Brutus Brutus Calpurnia Caesar 1 2 4 11 31 45 173 174 2 31 54 101 5 31
37 / 62
Introduction
Inverted index
Query optimization
Course overview
Optimized intersection algorithm for conjunctive queries
Intersect( t1 , . . . , tn ) 1 terms SortByIncreasingFrequency( t1 , . . . , tn ) 2 result postings(rst(terms)) 3 terms rest(terms) 4 while terms = nil and result = nil 5 do result Intersect(result, postings(rst(terms))) 6 terms rest(terms) 7 return result
38 / 62
Introduction
Inverted index
Query optimization
Course overview
More general optimization
Example query: (madding or crowd) and (ignoble or strife)
39 / 62
Introduction
Inverted index
Query optimization
Course overview
Example query: (madding or crowd) and (ignoble or strife) Get frequencies for all terms
39 / 62
Introduction
Inverted index
Query optimization
Course overview
Example query: (madding or crowd) and (ignoble or strife) Get frequencies for all terms Estimate the size of each or by the sum of its frequencies (conservative)
39 / 62
Introduction
Inverted index
Query optimization
Course overview
Example query: (madding or crowd) and (ignoble or strife) Get frequencies for all terms Estimate the size of each or by the sum of its frequencies (conservative) Process in increasing order of or sizes
39 / 62
Introduction
Inverted index
Query optimization
Course overview
Outline
40 / 62
Introduction
Inverted index
Query optimization
Course overview
Course overview
We are done with Chapter 1 of IIR (IIR 01).
41 / 62
Introduction
Inverted index
Query optimization
Course overview
Course overview
We are done with Chapter 1 of IIR (IIR 01). Plan for the rest of the semester: 1820 of the 21 chapters of IIR
41 / 62
Introduction
Inverted index
Query optimization
Course overview
Course overview
We are done with Chapter 1 of IIR (IIR 01). Plan for the rest of the semester: 1820 of the 21 chapters of IIR In what follows: teasers for most chapters to give you a sense of what will be covered.
41 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 02: The term vocabulary and postings lists
Phrase queries: Stanford University Proximity queries: Gates near Microsoft We need an index that captures position information for phrase queries and proximity queries.
42 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 03: Dictionaries and tolerant retrieval
bo or rd
- aboard
- about
- boardroom - border
- border
lord
- morbid
- sordid
- aboard
- ardent
- boardroom - border
43 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 04: Index construction
splits
assign
master
assign
postings a-f g-p q-z
parser parser
a-f g-p q-z a-f g-p q-z
inve rter inve rter inve rter
parser map phase
a-f g-p q-z segment files
reduce phase
44 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 05: Index compression

7 log10 cf 0 0 1 2 3 4 5 6
log10 rank
Zipfs law
45 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 06: Scoring, term weighting and the vector space model
Ranking search results
Boolean queries only give inclusion or exclusion of documents. For ranked retrieval, we measure the proximity between the query and each document. One formalism for doing this: the vector space model
Key challenge in ranked retrieval: evidence accumulation for a term in a document

1 vs. 0 occurence of a query term in the document 3 vs. 2 occurences of a query term in the document Usually: more is better But by how much? Need a scoring function that translates frequency into score or weight
46 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 07: Scoring in a complete search system
Parsing Linguistics Documents Indexers
user query Free text query parser Results page
Document cache
Spell correction
Scoring and ranking
Metadata in zone and field indexes
Inexact Tiered inverted top K positional index retrieval Indexes
k-gram
Scoring parameters MLR
training set
47 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 08: Evaluation and dynamic summaries
48 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 09: Relevance feedback & query expansion
49 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 12: Language models

w STOP the a frog P(w |q1 ) 0.2 0.2 0.1 0.01 w toad said likes that ... P(w |q1 ) 0.01 0.03 0.02 0.04 ...
q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 .
50 / 62
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops.
50 / 62
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog
50 / 62
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog P(string) = 0.01
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said P(string) = 0.01
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said P(string) = 0.01 0.03
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that P(string) = 0.01 0.03
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that P(string) = 0.01 0.03 0.04
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad P(string) = 0.01 0.03 0.04
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad P(string) = 0.01 0.03 0.04 0.01
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad likes P(string) = 0.01 0.03 0.04 0.01
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad likes P(string) = 0.01 0.03 0.04 0.01 0.02
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad likes frog P(string) = 0.01 0.03 0.04 0.01 0.02
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad likes frog P(string) = 0.01 0.03 0.04 0.01 0.02 0.01
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad likes frog STOP P(string) = 0.01 0.03 0.04 0.01 0.02 0.01
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad likes frog STOP P(string) = 0.01 0.03 0.04 0.01 0.02 0.01 0.2
Introduction
Inverted index
Query optimization
Course overview

q1
This is a one-state probabilistic nite-state automaton a unigram language model and the state emission distribution for its one state q1 . STOP is not a word, but a special symbol indicating that the automaton stops. frog said that toad likes frog STOP P(string) = 0.01 0.03 0.04 0.01 0.02 0.01 0.2 = 0.0000000000048
Introduction
Inverted index
Query optimization
Course overview
IIR 13: Text classication & Naive Bayes
Text classication = assigning documents automatically to predened classes Examples:

Language (English vs. French) Adult content Region
51 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 11: Probabilistic information retrieval
52 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 11: Probabilistic information retrieval
Term present Term absent
relevant (R = 1) pt 1 pt pt O(R|q, x) = O(R|q) ut

t:xt =qt =1
document xt = 1 xt = 0
t:xt =0,qt =1
nonrelevant (R = 0) ut 1 ut 1 pt (1) 1 ut
52 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 14: Vector classication

X X X X X
X X X X
X X
53 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 15: Support vector machines
54 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 16: Flat clustering
55 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 17: Hierarchical clustering
http://news.google.com
56 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 18: Latent Semantic Indexing
57 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 19: The web and its challenges
Unusual and diverse documents Unusual and diverse users and information needs Beyond terms and text: exploit link analysis, user data How do web search engines work? How can we make them better?
58 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 20: Crawling

Doc FPs URL set
- DNS 6 ?
www
6 ? Parse Content Seen?

To other nodes URL Filter

-
-

666
6 ?
-Fetch
Host - Dup splitter - URL - Elim From other nodes
6
URL Frontier
59 / 62
Introduction
Inverted index
Query optimization
Course overview
IIR 21: Link analysis / PageRank
60 / 62
Introduction
Inverted index
Query optimization
Course overview
Take-away
Boolean Retrieval: Design and data structures of a simple information retrieval system What topics will be covered in this class?
61 / 62
Introduction
Inverted index
Query optimization
Course overview
Resources
Chapter 1 of IIR http://ifnlp.org/ir

course schedule information retrieval links Shakespeare search engine
62 / 62

01 Intro

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 Intro

Uploaded by

Copyright:

Available Formats

Introduction

Processing Boolean queries

Introduction to Information Retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Schtze: Boolean Retrieval u

Processing Boolean queries

Schtze: Boolean Retrieval u

Processing Boolean queries

Schtze: Boolean Retrieval u

Processing Boolean queries

Schtze: Boolean Retrieval u

Processing Boolean queries

Denition of information retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Denition of information retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Denition of information retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Denition of information retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Denition of information retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Denition of information retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Denition of information retrieval

Schtze: Boolean Retrieval u

Processing Boolean queries

Schtze: Boolean Retrieval u

Processing Boolean queries

Schtze: Boolean Retrieval u

Processing Boolean queries

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u

Processing Boolean queries

Does Google use the Boolean model?

Schtze: Boolean Retrieval u