! !"#$%!"&'(!$)*+,$#-'%.'/%01(#*+'2'*0*+3$!3',/$*!/*,' $,&"0"4"5'/"01(,' $!.%+0"#$%!'+*#+$*)"&'6/,'789:' $!.%+0"#$%!'+*#+$*)"&'"!5'#*;#'0$!$!3'6/,'<;;:' ,1+$!3'=>8?' ",,$3!0*!#'@'8' !

Due Date: February 19, 2014 (11:55 pm) Submission Instructions: 1. Submit ONE Microsoft Word file containing answers (including the source code) for all the problems given in this assignment via slate. 2. The name of the file should be formatted as mentioned below: <ROLLNO>.docx For example, i090055.docx, etc. Plagiarism: Plagiarism is not allowed. If found plagiarized you will be awarded zero marks in the assignment.

Problem 1 (50 points)

Download the following files from slate: (1) input.tar.gz, and (2) qrel.txt

Run Files: By uncompressing input.tar.gz, you will get 129 run files. Each run file was generated by a search engine. The run file contains ranked lists corresponding to 50 different information needs (the queries). In particular, each line in the run file contains following fields in the order given below: 1. 2. 3. 4. 5. 6. Query ID Dummy value (ignore this field) Document ID Rank Score (the score tells us how well the document matches the query) System name (ignore this field).

QREL File: Qrel.txt is the file that contains the gold set, i.e., the relevance label for selected query-document pair. In particular, each line in the QREL contains following fields in the order given below: 1. 2. 3. 4. Query ID Dummy value (ignore this field) Document ID Relevance Label (0 means non-relevant, 1 means relevant)

IR Evaluation: Your task is to evaluate the search engines (the run) and arrange them in decreasing order of MAP values. In order to achieve this objective, you will need to compute the runs average precision for each query and then compute the mean average precision. Present your result using a plot (run vs. MAP value). Write the program that computes the MAP values in Perl. Submit (a) the source code, (b) a text file containing the runs name and its MAP value ordered from best to worst and (c) the plot. !

Problem 2 (25 points) Courtesy James Allan

A) The following list of R's and N's represents relevant (R) and non-relevant (N) documents in a ranked list of 50 documents. The top of the ranked list is on the left of the list, so that represents the most highly weighted document, the one that the system believes is most likely to be relevant. The list runs across the page to the right. This list shows 10 relevant documents. Assume that there are only 10 relevant documents for this query. RRNRNRNNNN RNNNRNNNNR RNNNNNNNRN NNNNNNNNNN RNNNNNNNNN Based on that list, calculate the following measures: 1. Average precision 2. Precision at cutoff 10 3. Precision at cutoff 20 4. R-precision B) Now, Imagine another system retrieves the following ranked list for the same query. RNNRNNNRNN NNNRNNNNNN NRNNNNNNRN NNRNNNNRNN NNNNRNNNNR Repeat parts (A.1), (A.2), (A.3), and (A.4) for the above ranked list. Compare the two ranked lists on the basis of these four metrics that you have computed--i.e., if you were given only these four numbers (Mean Average Precision, Precision at cutoff 10, Precision at cutoff 20, and R-precision) what can you determine about the relative performance of the two systems in general. C) Plot a recall/precision graph for the above two systems. What do the graphs tell you about the system in A and the one in B? !

Problem 3 (25 points)

Let 'perfect-retrieval' be defined as a list of ranked documents where - All the relevant documents are retrieved and - Every relevant document is ranked higher than any non-relevant one. A) Prove that at a list demonstrates perfect-retrieval if and only if there exists a cutoff=c such that the list has PREC(c)=1 and RECALL(c)=1 B) Consider a particular cutoff c=10 B1. Give example of a list that has PREC(10)=1 and RECALL(10)<1 B2. Give example of a list that has PREC(10)<1 and RECALL(10)=1 C) Prove that a list demonstrates perfect-retrieval if and only if R_PREC=1 D) Prove that a list demonstrates perfect-retrieval if and only if AveragePrecision=1 !