Review of Last Lecture

CS419/519 • Tour of an research information

retrieval system
Information Filtering and – Traditional IR search engine
Retrieval – Collaborative filtering
Jon Herlocker recommendations
Dept. of Computer Science – Crawling, pre-processing, and
Oregon State University indexing
Tu/Th Day 2 – Personalization
– User study

Project Idea
Today
Presentations
• Project idea presentations
• Introduction to IR
• Assignments for next week

What is Information Data Retrieval vs.

Retrieval? Information Retrieval
• How does it differ from
Data retrieval Information retrieval
databases (data retrieval)? Content Data Information
Data object Table Document, image, other
Matching Exact match Partial match, best match
Items wanted Matching Relevant
Query language SQL(artificial) Natural
Query specification Complete Incomplete
Model Deterministic Probabilistic
Highly structured Less structured

Table by Xin Xao, Drexel University

1
Data Retrieval vs.
Information Retrieval
• Information retrieval solutions
may incorporate data retrieval
– Data retrieval as a subset of
information retrieval
• For this class, data retrieval
alone is not interesting
Traditional
Information Need Model
1. User has an information need
2. Users forms a query
3. IR system makes best match with
documents
4. User evaluates ranked documents
5. Is the need met?
6a. Yes -> done
6b. No -> reformulate query
Relevance?
• What does relevance mean?
• A document might be relevant for
many reasons
– Answers a question with a fact
– Gives part of an answer
– Gives link to the answer
– Gives related information
• Relevance is subjective!
– We’ll return to this discussion later
Next
Models of information retrieval
Some Assumptions of
Traditional Model
• It is possible for the user to specify
their exact needs
• Document texts are functionally
equivalent to information needs
– Essentially document retrieval
• The information need remains
constant throughout search process
• The user will always recognize
relevant documents
(Belkin, Oddy, & Brooks)
Other Models
• Anomalous States of Knowledge
(ASKs)
– (Belkin, Oddy, and Brooks)
– A recognized anomaly in a user’s state
of knowledge that the user is not able to
specify specifically
• Berry-Picking Model
– (Bates 90)
– Interesting information scattered like
berries on bushes
– The query is continually shifting
2
Takeaway Messages
• Modeling information need and
user activity is complex
• Be aware of simplistic
assumptions that most IR work
makes
MIR – Chap. 2 - Models
• Goal is to retrieve documents
that are relevant to a user’s
single information need
– Based on those traditional
assumptions discussed earlier
Textbook Roadmap
Models for Retrieval
• What documents best match the need
described by a query?
• Modeling
– Information need
• From a query
– Information content
• From a document
– Closeness or similarity
• Between need and content
• Based on content analysis
– Measurable attributes of queries and documents
3
Generic Document
Retrieval Model Best Matching Models Covered
Information Need Documents
Ranking In this class:
Query Language Algorithm
• Boolean Model
Representation Representation of • Vector Space Model
of Information Need Document Content
Potentially more if time permits
and interest.
Prior Knowledge &
Assumptions
Documents

Full-text Indexing
Index Terms
Models
• Roughly a word or phrase describing the • Boolean model
content of a document
• Manual Indexing • Vector space model
– A human reads or scans a document and
assigns it index terms • Probabilistic model
– (i.e. Library of Congress subject terms)
• Automatic Indexing
– Full-text indexing
– Every word in the document becomes an index
term

Generic Full-Text
Boolean Model
Indexing Definitions
• K = {k1, ..., kt} • Compute vectors for each
– Set of all index terms document (dj)
• wi, j > 0 • If a keyword ki appears in dj,
– Weight of term ki in document dj then wi,j is 1, otherwise 0
– wi,j = 0 if ki is not in dj
• Each document is describe as a
vector
– dj = (w1,j, w2,j, ..., wt,j)
Modern Information Retrieval, Baeza-Yates & Ribeiro-Neto

4
Boolean Query Example Boolean
Language Queries
• Terms • House
– Words • House AND Corvallis
– Phrases • House OR Corvallis
• Operators • (House OR Condo) AND Corvallis
– AND • House AND Oregon AND NOT
– OR Eugene
– NOT • (House OR Condo) AND Corvallis
and NOT Eugene

Informal Pseudo
Rules for Boolean Logic
Boolean Notation
• DeMorgan’s Law • Evolved in web search engines
– NOT (A AND B) = (NOT A) OR (NOT B) • +house +Corvallis
– NOT(A OR B) = (NOT A) AND (NOT B) – house AND Corvallis
• Search for “Boolean Logic” if you want • +house +Oregon –Eugene
to know more – House and Oregon and NOT Eugene
• House +Corvallis
–?

Pseudo Boolean Ordering of Retrieved

Notation
• House +Corvallis
– (Corvallis AND House) OR Corvallis
• +House +Condo Corvallis Salem
– (House and Condo and Corvallis) OR
(House and Condo and Salem) OR
(House and Condo)
Documents
• Pure Boolean has no order
– All returned documents are equally
relevant
• In reality, different approaches can
be taken
– Chronologically
– Order by number of times a specified
term occurs
– Other approaches – get further and
further away from Boolean.

5
 Who Uses Boolean
Boolean Searching
Searches
• Upsides • Everybody until about ten-fifteen
– Easy to implement
years ago
– Simple queries are easy
– Query language gives significant control • Even now, many commercial
over results systems (library catalogs,
• Downsides abstracting services, etc)
– Binary relevance decision
• No ordering criteria
• Usually too much or too little
– Syntax can be complex

Boolean model – Data

Retrieval or Information Proximity operators
Retrieval?
• NEAR
• Very close – hard to distinguish
• Differences
• WITHIN
– Enormous number of attributes
– A document only has values for a few of
those attributes
– Inefficient to store and search using
traditional data retrieval methods
– Ordering may still be important